Compare commits
115
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
77f8170153 | ||
|
|
0fd0f1f637 | ||
|
|
7371cdeb94 | ||
|
|
abb18d4d8c | ||
|
|
5bf13c8701 | ||
|
|
fe6af1445f | ||
|
|
8845d1aa49 | ||
|
|
0b92b40db5 | ||
|
|
27ccf253f5 | ||
|
|
c221452a00 | ||
|
|
fe38c07e7e | ||
|
|
45b5e19fb8 | ||
|
|
e08278fae2 | ||
|
|
350913b6c2 | ||
|
|
470caacfb0 | ||
|
|
e8b9bdd7c1 | ||
|
|
55cbbfb6e4 | ||
|
|
4d058cba81 | ||
|
|
872b583f06 | ||
|
|
90931b4750 | ||
|
|
a6f1a26658 | ||
|
|
4ed07388e8 | ||
|
|
bb91d0a22d | ||
|
|
e7e8d8236e | ||
|
|
fc65b8ab83 | ||
|
|
26ed8855c9 | ||
|
|
9778bdf351 | ||
|
|
2b7620ded9 | ||
|
|
c0682aa572 | ||
|
|
25aed6635f | ||
|
|
9310ff85bf | ||
|
|
5b846136ad | ||
|
|
1bf3d6a592 | ||
|
|
4e4fff87d4 | ||
|
|
995a493bd3 | ||
|
|
5a93a9a8e7 | ||
|
|
911fea3243 | ||
|
|
4aece8aff7 | ||
|
|
adb5e94511 | ||
|
|
673decf4d4 | ||
|
|
4d1cf66d96 | ||
|
|
2baad2ebae | ||
|
|
8383459f70 | ||
|
|
c34f6f3297 | ||
|
|
e4b6d918e4 | ||
|
|
bf5cf011ce | ||
|
|
6e6e47652a | ||
|
|
4c09a22add | ||
|
|
2bdf1ec76f | ||
|
|
f7e036abef | ||
|
|
f2de8f73aa | ||
|
|
643f305814 | ||
|
|
cbe1b3bd63 | ||
|
|
ae775b383c | ||
|
|
358d5af70b | ||
|
|
c6831090b5 | ||
|
|
38fc95d772 | ||
|
|
3f6638bf96 | ||
|
|
469c8aa481 | ||
|
|
df2997e9f3 | ||
|
|
f69725503f | ||
|
|
d5bc7e8031 | ||
|
|
efdffc82ae | ||
|
|
e23544a46b | ||
|
|
035ee937a0 | ||
|
|
bbb4063362 | ||
|
|
b4ba9d95b1 | ||
|
|
c18a2063c8 | ||
|
|
b636ef38bc | ||
|
|
195b5f3917 | ||
|
|
1a1e9d9542 | ||
|
|
d09b5d6abb | ||
|
|
439f5cd67b | ||
|
|
d79cb528fd | ||
|
|
9b76d2f96c | ||
|
|
bf54a3ce5f | ||
|
|
efbbb23525 | ||
|
|
3b85881af4 | ||
|
|
8cf8c4139c | ||
|
|
cd773e8f33 | ||
|
|
242f00671d | ||
|
|
7f8d7cbe20 | ||
|
|
8dd2f0c015 | ||
|
|
b3402d158f | ||
|
|
c910cf2f20 | ||
|
|
245c0a2019 | ||
|
|
add7f7d167 | ||
|
|
a29afd916d | ||
|
|
e9ce989998 | ||
|
|
7299e82c22 | ||
|
|
41052eb4ea | ||
|
|
85fd9b6704 | ||
|
|
e06c0688d9 | ||
|
|
15a3603513 | ||
|
|
c430558ab3 | ||
|
|
be38d652a5 | ||
|
|
31a75f4985 | ||
|
|
f8a8e905da | ||
|
|
5e8589bbf8 | ||
|
|
ceadf69e43 | ||
|
|
4379f76747 | ||
|
|
b3be568e36 | ||
|
|
a8cbc7b338 | ||
|
|
cf8dc80428 | ||
|
|
da74b8b9bf | ||
|
|
6944622085 | ||
|
|
2bdcebdb34 | ||
|
|
046dde44a4 | ||
|
|
74f58ea001 | ||
|
|
b515c16117 | ||
|
|
4a0d4b0964 | ||
|
|
cb8b99e0e7 | ||
|
|
e5c75702e8 | ||
|
|
4ef772d37b | ||
|
|
6986eb1937 |
@@ -7,6 +7,101 @@ edit the release notes and re-run the generator.
|
||||
|
||||
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
|
||||
|
||||
## 0.119.0 — 2026-11-13
|
||||
|
||||
### New
|
||||
|
||||
- Make a GIF of up to 30 seconds: in a player's Clip panel, or in the Downloads editor, where each kept segment becomes a GIF of its own. Both say the expected size before it is made.
|
||||
- A shared GIF's link opens the moving picture itself, with Download beside it. In Teams its preview moves when the GIF is 15 MB or less.
|
||||
|
||||
## 0.118.0 — 2026-11-08
|
||||
|
||||
### New
|
||||
|
||||
- Surprise me now leans on the channels and genres you watch, and one pick in three stays a blind draw, so it remains a surprise.
|
||||
- On a connection too slow for a Plex video, playback waits and goes on where it was, and says how fast the video needs and how fast it arrives.
|
||||
|
||||
### Fixed
|
||||
|
||||
- A 24/7 stream's archive no longer adds years to a list's total: a total names the videos it could not count as "+N without length". The Feed sorted by Surprise me opens faster.
|
||||
- Plex films the server's GPU cannot decode (VC-1, older DivX or XviD) start in a few seconds instead of about 20. Text over the Plex picture is readable on any frame.
|
||||
|
||||
## 0.117.0 — 2026-11-06
|
||||
|
||||
### New
|
||||
|
||||
- A share link's address now says what it opens: /file/ for a file, /folder/ for a folder, /watch/ for a video. Links you already sent keep working.
|
||||
- The upload dialog shows which folder the file goes to, and you can pick another one or make a new folder there. Opening Share copies a link at once.
|
||||
- The Channels tag chips show how many followed channels each holds. Continue watching in Plex has a See all, and a Plex item's Info shows the file size.
|
||||
|
||||
### Fixed
|
||||
|
||||
- An expired share link is marked as expired and is never copied. A crop of a video with wide pixels keeps exactly what its frame showed.
|
||||
- A Plex auto-skip waits 3 seconds into the marker before its countdown starts.
|
||||
|
||||
## 0.116.3 — 2026-11-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- If a stylesheet is lost on the way, the app now asks for it again, so a poor connection no longer leaves the page unstyled until a reload.
|
||||
|
||||
## 0.116.2 — 2026-11-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- A Plex video paused or muted in a background tab no longer loses its session after a minute, and if a session is lost anyway, playback comes back at once instead of after half a minute.
|
||||
|
||||
## 0.116.1 — 2026-11-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- A playlist now shows the order the server holds after a revert to YouTube, a sync or a change on another device, and the next drag no longer brings back the old one.
|
||||
- An Undo on a removed or moved row puts it back in the right place while another change is still saving.
|
||||
|
||||
## 0.116.0 — 2026-11-03
|
||||
|
||||
### New
|
||||
|
||||
- In a playlist, tick rows with their box, Shift+click or x, then move or remove them together. An Undo puts them back at their places, and a single row's move has one too.
|
||||
- Drag rows onto another list in the rail: one row, or all the selected ones, also under a sort or a filter.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Removing a video from inside the player no longer loses your place in the list when the player closes.
|
||||
|
||||
## 0.115.0 — 2026-11-01
|
||||
|
||||
### New
|
||||
|
||||
- Clear watch progress is on every YouTube video's menu, on Home, in playlists and in the player's related videos. It forgets where you left a started video and changes nothing else.
|
||||
- Channels have an Added column, and an About card opens only when the pointer rests on it.
|
||||
- In Files, a press anywhere on a row plays the file.
|
||||
- The feed's toolbar pills are named by a glyph, with Sort and View on the filters' row.
|
||||
|
||||
### Fixed
|
||||
|
||||
- In a play queue, Previous and Next pass over a video that can't play, in the direction you were going, and say why when only such videos are left.
|
||||
- A Facebook download titled only “Facebook” now takes the post's text as its name.
|
||||
- The player's big view keeps its way out in sight on a tablet, and Tab no longer lands on controls hidden under it.
|
||||
|
||||
## 0.114.3 — 2026-11-01
|
||||
|
||||
### Fixed
|
||||
|
||||
- Editing a file that is neither video nor audio now says why it can't be done, instead of a generic error.
|
||||
|
||||
## 0.114.2 — 2026-10-30
|
||||
|
||||
### Fixed
|
||||
|
||||
- Security: a library the sign-in relies on is updated, closing two published vulnerabilities.
|
||||
|
||||
## 0.114.1 — 2026-10-30
|
||||
|
||||
### Fixed
|
||||
|
||||
- Notifications follow a language switch: a notice on screen and every row in the notification center are said again in the new language, their buttons too.
|
||||
|
||||
## 0.114.0 — 2026-10-29
|
||||
|
||||
### New
|
||||
|
||||
@@ -65,7 +65,7 @@ how it gets built. Concretely:
|
||||
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
|
||||
`downloads/`, `reader/`, `discover/`, `catalog/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
|
||||
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
|
||||
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
|
||||
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`, `draw`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
|
||||
The domain is four **units**, each owning its tables: Capture (`downloads/`), Reader (`reader/`, `discover/`, the playlist
|
||||
sync), Catalog (`catalog/`, the `sync/` ingests) and Plex (`plex/`). A unit's route does no database work: it reads through
|
||||
the unit's `queries`, writes through its `service` (which flushes) and commits. A unit's code reads another unit's tables
|
||||
@@ -74,7 +74,7 @@ only along a declared edge and writes them only through that unit's service; cor
|
||||
cannot (an ORM write on another unit's loaded row, SQL in `text()`, a commit by `begin()`) is review's to catch, listed in
|
||||
[`backend/app/README.md`](backend/app/README.md) with the conventions.
|
||||
Frontend, by folder under `src/`: `main.tsx` provider tree → `shell/` (`App`, header, nav rail, palette,
|
||||
boot; a shared `/watch` link boots from its own entry, `watch.html` → `watchMain.tsx` → `watch/`, held
|
||||
boot; a share link (`/watch`, `/file`, `/folder`) boots from its own entry, `watch.html` → `watchMain.tsx` → `watch/`, held
|
||||
app-free by a contract test) → **`modules/<page>/`** (one folder per page: the page, its rail, its store, its tests,
|
||||
and its `module.ts` definition — label, icon, gate, the lazy page and rail, its store — collected in nav order by
|
||||
`features/navigation/moduleRegistry`, which also mounts every store at the root; a new page is its folder, its id in
|
||||
@@ -105,7 +105,7 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
|
||||
| Ownership check | `userscope.owned_by` (a filter) / `owned_row` (a row by id): a unit's `queries.own_x` returns None and its route answers `not_found`; a core route `owned_or_404`. A composite rule (own or shared, built-in or own) says so in its `queries` | an inline owner filter — `Model.user_id == …`, an alias's, `.in_()`, `filter_by(user_id=…)` (fence-held, every module) — or a `db.get` and a compare where `owned_row` fits |
|
||||
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
|
||||
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
|
||||
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
|
||||
| User-facing error | structured `detail` (`{code, reason}`), its sentence named in `lib/api/refusalCodes.ts` (contract-held) | a bare English string under a bilingual UI |
|
||||
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
|
||||
| Transaction commit | the handler commits; a unit's `service` and every helper `flush` (fence-held, every module) | a helper that commits — only a listed one (what owns its session, or a core step that must land whatever its caller does next), each with its reason: the fence's `UNIT_COMMITS` / `CORE_COMMITS`, a route's `COMMITTING_HELPERS` (`test_route_conventions`) |
|
||||
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
|
||||
|
||||
@@ -185,118 +185,6 @@
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 50,
|
||||
"endColumn": 56,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
@@ -364,25 +252,7 @@
|
||||
}
|
||||
}
|
||||
],
|
||||
"./app/downloads/adapters.py": [
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 30,
|
||||
"endColumn": 38,
|
||||
"lineCount": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"./app/downloads/backfill.py": [
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 34,
|
||||
"endColumn": 38,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportReturnType",
|
||||
"range": {
|
||||
@@ -413,14 +283,6 @@
|
||||
}
|
||||
],
|
||||
"./app/downloads/pipeline.py": [
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 34,
|
||||
"endColumn": 38,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportAttributeAccessIssue",
|
||||
"range": {
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
"""videos: the duration sorts ride an index on the video's length, not on the raw column
|
||||
|
||||
Revision ID: 0117_videos_length_sort_indexes
|
||||
Revises: 0116_followed_explored_channels
|
||||
Create Date: 2026-11-07
|
||||
|
||||
A YouTube length past a day is not a length (`catalog.models.LENGTH_MAX_S`): a 24/7 stream's archive
|
||||
reports the years it has run. The feed's duration sorts now order by `Video.length_seconds`, which
|
||||
is NULL for such a video, so "longest first" no longer opens on a stream archive. Migration 0096's
|
||||
two duration indexes are on the raw column and no longer match that ORDER BY; they are replaced by
|
||||
the same indexes over the length expression. The expression is spelled out here as SQLAlchemy emits
|
||||
it, with the limit as a literal (an index matches a query's constant, never a bound parameter);
|
||||
tests/test_feed_sort_indexes.py fails if the two drift apart.
|
||||
|
||||
CONCURRENTLY (autocommit_block), re-runnable, as in 0096.
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import text
|
||||
|
||||
revision: str = "0117_videos_length_sort_indexes"
|
||||
down_revision: str | None = "0116_followed_explored_channels"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
# Frozen with this migration: `catalog.models.LENGTH_MAX_S` is 24 h.
|
||||
LENGTH = "(CASE WHEN (duration_seconds > 86400) THEN NULL ELSE duration_seconds END)"
|
||||
|
||||
NEW = {
|
||||
"ix_videos_length_desc": f"{LENGTH} DESC NULLS LAST, id",
|
||||
"ix_videos_length_asc": f"{LENGTH} ASC NULLS LAST, id",
|
||||
}
|
||||
OLD = {
|
||||
"ix_videos_duration_desc": "duration_seconds DESC NULLS LAST, id",
|
||||
"ix_videos_duration_asc": "duration_seconds ASC NULLS LAST, id",
|
||||
}
|
||||
|
||||
|
||||
def _valid(name: str) -> bool | None:
|
||||
"""The index's `indisvalid`, or None when there is no index of that name."""
|
||||
row = (
|
||||
op.get_bind()
|
||||
.execute(
|
||||
text(
|
||||
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
|
||||
"WHERE c.relname = :name"
|
||||
),
|
||||
{"name": name},
|
||||
)
|
||||
.first()
|
||||
)
|
||||
return None if row is None else bool(row[0])
|
||||
|
||||
|
||||
def _build(name: str, columns: str) -> None:
|
||||
"""CREATE INDEX CONCURRENTLY, re-runnable: an INVALID leftover of a failed build is dropped first
|
||||
(see 0096 `_build`)."""
|
||||
if _valid(name) is False:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
|
||||
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} ON videos ({columns})")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
for name, columns in NEW.items():
|
||||
_build(name, columns)
|
||||
for name in OLD:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
for name, columns in OLD.items():
|
||||
_build(name, columns)
|
||||
for name in NEW:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
|
||||
@@ -0,0 +1,40 @@
|
||||
"""videos: a random key per video, so a seeded shuffle walks an index instead of sorting the scope
|
||||
|
||||
Revision ID: 0118_videos_shuffle_key
|
||||
Revises: 0117_videos_length_sort_indexes
|
||||
Create Date: 2026-11-07
|
||||
|
||||
The feed's Shuffle sort and the Home surprise ordered the whole feed scope by md5(id || seed) on every
|
||||
call: ~350 ms on prod for a scope of ~213k videos, and a Shuffle page deeper in the list cost more
|
||||
(767 ms on page 7 on localdev, against 4 ms for Newest). Each video now carries a random number drawn
|
||||
once (`Video.shuffle_key`); a seed picks a point on that ring and the draw reads the index from there
|
||||
(`feedquery.shuffle_walk`).
|
||||
|
||||
One ADD COLUMN with a volatile default rewrites the table and fills every row with its own value. On
|
||||
localdev (278k videos) that took 28 s, against 84 s for an UPDATE of a nullable column, so the rewrite
|
||||
it is: the table is locked for that time, and a deploy has the app down while it migrates anyway.
|
||||
The index is (shuffle_key, id): the walk's keyset cursor compares the pair.
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0118_videos_shuffle_key"
|
||||
down_revision: str | None = "0117_videos_length_sort_indexes"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
"ALTER TABLE videos ADD COLUMN IF NOT EXISTS shuffle_key double precision "
|
||||
"NOT NULL DEFAULT random()"
|
||||
)
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
|
||||
op.execute("CREATE INDEX CONCURRENTLY ix_videos_shuffle_key ON videos (shuffle_key, id)")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
|
||||
op.execute("ALTER TABLE videos DROP COLUMN IF EXISTS shuffle_key")
|
||||
@@ -0,0 +1,65 @@
|
||||
"""plex_items.transcode_hw: the facet says whether the GPU ENCODES, under the measured codec sets
|
||||
|
||||
Measured on prod (2026-11-07): the iGPU has no VC-1 Advanced decode, so a VC-1 film never ran on the
|
||||
GPU although the facet said it did; VC-1, XviD and DivX now take the CPU-decode → GPU-encode path; and an
|
||||
HDR film runs on the GPU through the OpenCL tonemap although the facet said software. The rule is
|
||||
`plex.sync.transcode_hw_capable`; the two codec sets below are pinned to `plex.sync` by
|
||||
tests/test_plex_playback_class.py.
|
||||
|
||||
Rewritten from the stored codec and bit depth. HDR is not stored, so a 10-bit HW-decodable item is set to
|
||||
software here and its probe watermark cleared: the bit-depth backfill (`plex.sync._probe_media_batch`,
|
||||
a bounded batch per Plex sync) reads its HDR flag again and moves an HDR one back to the GPU.
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0119_plex_transcode_gpu"
|
||||
down_revision: Union[str, None] = "0118_videos_shuffle_key"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
HW_DECODABLE_CODECS = ("h264", "hevc", "h265", "vp9", "mpeg2video")
|
||||
CPU_DECODE_CODECS = ("vc1", "mpeg4", "msmpeg4v3")
|
||||
|
||||
|
||||
def _in(codecs: tuple[str, ...]) -> str:
|
||||
return ", ".join(f"'{c}'" for c in codecs)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
f"""
|
||||
UPDATE plex_items
|
||||
SET transcode_hw = CASE
|
||||
WHEN lower(codec_video) IN ({_in(HW_DECODABLE_CODECS)})
|
||||
THEN coalesce(video_bit_depth, 8) <= 8
|
||||
WHEN lower(codec_video) IN ({_in(CPU_DECODE_CODECS)})
|
||||
THEN coalesce(video_bit_depth, 8) <= 8
|
||||
ELSE false
|
||||
END
|
||||
WHERE playable = 'transcode'
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
f"""
|
||||
UPDATE plex_items
|
||||
SET media_probed_at = NULL
|
||||
WHERE playable = 'transcode'
|
||||
AND lower(codec_video) IN ({_in(HW_DECODABLE_CODECS)})
|
||||
AND video_bit_depth > 8
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# The old rule: a decodable codec (VC-1 counted) at 8 bits; HDR and every other codec software.
|
||||
op.execute(
|
||||
f"""
|
||||
UPDATE plex_items
|
||||
SET transcode_hw = lower(codec_video) IN ({_in(HW_DECODABLE_CODECS + ("vc1",))})
|
||||
AND coalesce(video_bit_depth, 8) <= 8
|
||||
WHERE playable = 'transcode'
|
||||
"""
|
||||
)
|
||||
@@ -25,8 +25,8 @@ Three layers, by *who* runs the code — not by feature:
|
||||
background run's durable row, its words in `runstatus`),
|
||||
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
|
||||
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
|
||||
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
|
||||
reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core
|
||||
level), `debugsession` (a recording that tags its user's lines, E16), `collation` (A→Z in the
|
||||
reader's alphabet, E30) and `draw` (the weighted, seeded draw "Surprise me" deals from), … are single-file primitives many layers import. They depend on nothing above them (a core
|
||||
module must never import a `routes/…` module).
|
||||
|
||||
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
|
||||
|
||||
@@ -10,16 +10,22 @@ from sqlalchemy import (
|
||||
Boolean,
|
||||
Computed,
|
||||
DateTime,
|
||||
Float,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
JSON,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
case,
|
||||
func,
|
||||
literal_column,
|
||||
null,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql import TSVECTOR
|
||||
from sqlalchemy.ext.hybrid import hybrid_property
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
from sqlalchemy.sql.elements import ColumnElement
|
||||
|
||||
from app.db import Base
|
||||
from app.models import TimestampMixin
|
||||
@@ -114,6 +120,20 @@ class Subscription(Base, TimestampMixin):
|
||||
# hidden from feeds/search and revisited by the refresh job until they settle into a VOD.
|
||||
LIVE_OR_UPCOMING = ("live", "upcoming")
|
||||
|
||||
# A YouTube length past a day is not a length. A 24/7 stream's archive (`was_live`) reports the years
|
||||
# the stream has run (measured on prod, 2026-11-06: 122,962,914 s), and one such video made a Watch
|
||||
# later total read 44,073 hours. YouTube caps an upload at 12 hours, so nothing real is lost. Such a
|
||||
# length is unknown wherever a length is used: the API sends none (`Video.length_seconds`), sums leave
|
||||
# it out (a playlist's counts it as `without_length`), and the duration filter, sort and Home budget never
|
||||
# match it. The raw `duration_seconds` stays for what it does prove — that the video is not a Short —
|
||||
# and for the jobs that ask whether YouTube has answered yet (test_video_length holds who reads it).
|
||||
LENGTH_MAX_S = 24 * 3600
|
||||
|
||||
|
||||
def known_length(seconds: int | None) -> int | None:
|
||||
"""`seconds` as a video's length, or None when it is not one (see LENGTH_MAX_S)."""
|
||||
return seconds if seconds is not None and seconds <= LENGTH_MAX_S else None
|
||||
|
||||
|
||||
class Video(Base, TimestampMixin):
|
||||
"""A YouTube video. Shared across users; per-user state lives elsewhere."""
|
||||
@@ -141,8 +161,8 @@ class Video(Base, TimestampMixin):
|
||||
)
|
||||
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
|
||||
# The feed's views / duration / title sorts each ride a migration-defined index in their exact
|
||||
# ORDER BY, one per direction (and per ICU collation for title): migration 0096, held to
|
||||
# `feedquery.order_by` by test_feed_sort_indexes.
|
||||
# ORDER BY, one per direction (and per ICU collation for title): migration 0096 (the duration
|
||||
# pair since 0117, on `length_seconds`), held to `feedquery.order_by` by test_feed_sort_indexes.
|
||||
duration_seconds: Mapped[int | None] = mapped_column(Integer)
|
||||
view_count: Mapped[int | None] = mapped_column(BigInteger)
|
||||
like_count: Mapped[int | None] = mapped_column(BigInteger)
|
||||
@@ -197,6 +217,37 @@ class Video(Base, TimestampMixin):
|
||||
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
unavailable_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
unavailable_reason: Mapped[str | None] = mapped_column(String(24))
|
||||
# A random number in [0, 1), drawn once when the row is written: the ring a seeded shuffle walks
|
||||
# (`feedquery.shuffle_walk`, index ix_videos_shuffle_key on (shuffle_key, id), migration 0118).
|
||||
shuffle_key: Mapped[float] = mapped_column(Float, nullable=False, server_default=func.random())
|
||||
|
||||
@hybrid_property
|
||||
def length_seconds(self) -> int | None:
|
||||
"""The video's length, or None when it is unknown — not fetched yet, or not a length at all
|
||||
(`LENGTH_MAX_S`). What every reader of a length reads."""
|
||||
return known_length(self.duration_seconds)
|
||||
|
||||
@length_seconds.inplace.expression
|
||||
@classmethod
|
||||
def _length_seconds_expression(cls) -> ColumnElement[int | None]:
|
||||
# The limit is a literal, not a bound parameter: the duration sort indexes (migration 0117)
|
||||
# are built on this very expression, and Postgres matches an index to a query's constant only.
|
||||
return case(
|
||||
(cls.duration_seconds > literal_column(str(LENGTH_MAX_S)), null()),
|
||||
else_=cls.duration_seconds,
|
||||
)
|
||||
|
||||
@hybrid_property
|
||||
def without_length(self) -> bool:
|
||||
"""A video whose length is unknown — live, an archive past a day, or not fetched yet: what a
|
||||
sum of lengths leaves out and names as "+N without length". Every such video, so the count
|
||||
and the sum never disagree about what was left out."""
|
||||
return self.length_seconds is None
|
||||
|
||||
@without_length.inplace.expression
|
||||
@classmethod
|
||||
def _without_length_expression(cls) -> ColumnElement[bool]:
|
||||
return cls.length_seconds.is_(None)
|
||||
|
||||
# Weighted full-text search document (DB-generated, never written by the app): title (A) >
|
||||
# creator keywords + search queries (B) > truncated description (C). ts_rank honours the
|
||||
|
||||
@@ -44,7 +44,8 @@ def channel_titles(db: Session, ids: set[str]) -> dict[str, str]:
|
||||
|
||||
|
||||
def channel_aggregates(channel_ids: list[str]) -> Select:
|
||||
"""Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
|
||||
"""Per-channel stored count, last upload and total duration (the lengths it knows: a video with
|
||||
none is left out, `Video.length_seconds`), in one grouped pass. Every column it
|
||||
reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
|
||||
alone; a test holds that (test_channel_list_index_only)."""
|
||||
return (
|
||||
@@ -52,7 +53,7 @@ def channel_aggregates(channel_ids: list[str]) -> Select:
|
||||
Video.channel_id,
|
||||
func.count(Video.id),
|
||||
func.max(Video.published_at),
|
||||
func.coalesce(func.sum(Video.duration_seconds), 0),
|
||||
func.coalesce(func.sum(Video.length_seconds), 0),
|
||||
)
|
||||
.where(Video.channel_id.in_(channel_ids))
|
||||
.group_by(Video.channel_id)
|
||||
|
||||
+101
-29
@@ -10,16 +10,25 @@ explore-visible and in-progress for free, and cannot drift from them. The Plex h
|
||||
narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
|
||||
sidebar and wrong for a landing page that does not.
|
||||
|
||||
**One seed drives both halves.** The feed orders a shuffle by `md5(id || seed)` so a keyset cursor
|
||||
stays valid across pages; the Plex surprise rail uses `ORDER BY random()`, which is fine for a single
|
||||
un-paged draw but cannot be reproduced. A cross-source reroll needs both halves to move together —
|
||||
otherwise "Mást kérek" re-draws the Plex picks while the YouTube ones stand still — so the Plex side
|
||||
here takes the feed's seeded-md5 idiom instead of `random()`.
|
||||
**One seed drives both halves.** The YouTube half starts on the ring of stored random keys at the
|
||||
seed's point and shuffles that window by `md5(id || seed)`; the Plex surprise rail uses
|
||||
`ORDER BY random()`, which is fine for a single un-paged draw but cannot be reproduced. A
|
||||
cross-source reroll needs both halves to move together — otherwise "Mást kérek" re-draws the Plex
|
||||
picks while the YouTube ones stand still — so the Plex side here orders by the same seeded md5
|
||||
instead of `random()`.
|
||||
"""
|
||||
|
||||
from sqlalchemy import and_, func, or_, select
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import (
|
||||
and_,
|
||||
func,
|
||||
or_,
|
||||
select,
|
||||
)
|
||||
from sqlalchemy.orm import Session, aliased
|
||||
|
||||
from app import draw
|
||||
from app.discover.budget import band
|
||||
from app.feedquery import (
|
||||
feed_columns,
|
||||
@@ -27,13 +36,14 @@ from app.feedquery import (
|
||||
not_blocked_cond,
|
||||
order_by,
|
||||
serialize,
|
||||
shuffle_walk,
|
||||
sort_keys,
|
||||
)
|
||||
from app.models import User
|
||||
from app.plex.models import PlexItem, PlexShow, PlexState
|
||||
from app.catalog.models import Channel, Video
|
||||
from app.reader.models import PlaylistItem, VideoState
|
||||
from app.plex import rollup
|
||||
from app.plex import affinity, rollup
|
||||
from app.plex.cards import display_title, leaf_card, movie_card, show_card
|
||||
from app.plex.query import apply_show_state_filter, content_libs
|
||||
from app.userscope import owned_by
|
||||
@@ -67,7 +77,8 @@ _FEED_BASE: dict = {
|
||||
|
||||
|
||||
def _seeded(col, seed: int):
|
||||
"""The feed's deterministic shuffle expression, reused for Plex so one seed moves both halves."""
|
||||
"""The Home surprise's seeded order (md5 of the key and the seed), the same for Plex so one
|
||||
seed moves both halves."""
|
||||
return func.md5(func.concat(col, str(seed)))
|
||||
|
||||
|
||||
@@ -112,17 +123,75 @@ def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) ->
|
||||
Shorts are opened up only for the shortest band: "I have ten minutes" is the one answer a Short is
|
||||
actually a good response to, and the feed's own default keeps them out otherwise."""
|
||||
lo, hi = band(budget)
|
||||
params = dict(_FEED_BASE, min_duration=lo, max_duration=hi, include_shorts=budget == "short")
|
||||
query, rank_expr = filtered_query(db, user, **params)
|
||||
keys = sort_keys("shuffle", seed, rank_expr)
|
||||
# Over-fetch, then drop anything already started. The feed's `unwatched` scope keeps watched and
|
||||
# hidden out but deliberately keeps in-progress rows in (they belong in the feed). Here they must
|
||||
# go: the resume tier has already had its chance at them, and it may have passed precisely because
|
||||
# the REMAINING time did not fit the budget — offering the same item back as a surprise, whose
|
||||
# runtime does fit, would label a half-finished thing "Meglepetés" and contradict the question.
|
||||
rows = db.execute(query.order_by(*order_by(keys)).limit(limit * 4)).all()
|
||||
out = [serialize(r) for r in rows]
|
||||
return [r for r in out if not r["position_seconds"]][:limit]
|
||||
params: dict = {
|
||||
**_FEED_BASE,
|
||||
"min_duration": lo,
|
||||
"max_duration": hi,
|
||||
"include_shorts": budget == "short",
|
||||
}
|
||||
query, _rank = filtered_query(db, user, **params)
|
||||
|
||||
# A window of the ring, not the whole scope: the seed's SURPRISE_WINDOW videos read along the
|
||||
# stored random key (`shuffle_walk`, a millisecond), then shuffled by the seed. Dealing straight
|
||||
# from the ring would favour for good every video that sits after a wide gap of it (the first
|
||||
# video past a random point is the one whose gap the point fell into); shuffling the window makes
|
||||
# each video's chance the window's share, even to within a few percent. A scope no larger than
|
||||
# the window is the whole scope, in exactly the md5 order the draw always had.
|
||||
#
|
||||
# Anything already started is dropped. The feed's `unwatched` scope keeps watched and hidden out
|
||||
# but deliberately keeps in-progress rows in (they belong in the feed). Here they must go: the
|
||||
# resume tier has already had its chance at them, and it may have passed precisely because the
|
||||
# REMAINING time did not fit the budget — offering the same item back as a surprise, whose runtime
|
||||
# does fit, would label a half-finished thing "Meglepetés" and contradict the question.
|
||||
window = [
|
||||
r
|
||||
for r in map(serialize, shuffle_walk(db, query, seed, SURPRISE_WINDOW))
|
||||
if not r["position_seconds"]
|
||||
]
|
||||
blind = draw.shuffled(window, seed, key=_video_id)
|
||||
channels = _channel_affinity(db, user)
|
||||
if not channels:
|
||||
return blind[:limit]
|
||||
|
||||
def weight(row: dict) -> float:
|
||||
return 1.0 + CHANNEL_WEIGHT * channels.get(row["channel_id"], 0.0)
|
||||
|
||||
leaning = draw.leaning(window, seed, key=_video_id, weight=weight)
|
||||
return draw.blend(leaning, blind, seed, limit, key=_video_id)
|
||||
|
||||
|
||||
# How many videos of the ring a surprise draws from: wide enough that a channel the user watches has
|
||||
# a few in it, small enough to read in a millisecond or two.
|
||||
SURPRISE_WINDOW = 500
|
||||
|
||||
|
||||
def _video_id(row: dict) -> str:
|
||||
return row["id"]
|
||||
|
||||
|
||||
# How much a channel the user watches most outweighs one they never watch (1 + this), and how far back
|
||||
# the watched history counts: the latest WATCHED_RECENT watches, the recent ones weighing more.
|
||||
CHANNEL_WEIGHT = 3.0
|
||||
WATCHED_RECENT = 300
|
||||
WATCHED_HALF_LIFE_DAYS = 90
|
||||
|
||||
|
||||
def _channel_affinity(db: Session, user: User) -> dict[str, float]:
|
||||
"""Each channel the user watched lately, scored 0..1 against the most watched: the sum of its
|
||||
watches, each weighing half as much every WATCHED_HALF_LIFE_DAYS. Empty with no history."""
|
||||
rows = db.execute(
|
||||
select(Video.channel_id, VideoState.watched_at)
|
||||
.join(Video, Video.id == VideoState.video_id)
|
||||
.where(owned_by(VideoState.user_id, user), VideoState.status == "watched")
|
||||
.order_by(VideoState.watched_at.desc().nullslast())
|
||||
.limit(WATCHED_RECENT)
|
||||
).all()
|
||||
now = datetime.now(timezone.utc)
|
||||
scores: dict[str, float] = {}
|
||||
for channel_id, watched_at in rows:
|
||||
r = draw.recency(watched_at, now, WATCHED_HALF_LIFE_DAYS)
|
||||
scores[channel_id] = scores.get(channel_id, 0.0) + r
|
||||
return draw.scaled(scores)
|
||||
|
||||
|
||||
def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
|
||||
@@ -289,21 +358,24 @@ def plex_surprise(
|
||||
if not movie_libs:
|
||||
return []
|
||||
lo, hi = band(budget)
|
||||
q = (
|
||||
select(PlexItem)
|
||||
.where(
|
||||
PlexItem.kind == "movie",
|
||||
PlexItem.library_id.in_(movie_libs),
|
||||
_unseen_movie_conds(uid),
|
||||
)
|
||||
.order_by(_seeded(PlexItem.rating_key, seed))
|
||||
.limit(limit)
|
||||
q = select(PlexItem).where(
|
||||
PlexItem.kind == "movie",
|
||||
PlexItem.library_id.in_(movie_libs),
|
||||
_unseen_movie_conds(uid),
|
||||
)
|
||||
if lo is not None:
|
||||
q = q.where(PlexItem.duration_s >= lo)
|
||||
if hi is not None:
|
||||
q = q.where(PlexItem.duration_s <= hi)
|
||||
items = db.execute(q).scalars().all()
|
||||
shuffled = _seeded(PlexItem.rating_key, seed)
|
||||
blind = db.execute(q.order_by(shuffled, PlexItem.id).limit(limit)).scalars().all()
|
||||
leaning = affinity.for_user(db, uid)
|
||||
if leaning:
|
||||
key = draw.weighted_key(draw.seeded_u(PlexItem.rating_key, seed), leaning.weight(PlexItem))
|
||||
weighted = db.execute(q.order_by(key, shuffled, PlexItem.id).limit(limit)).scalars().all()
|
||||
items = draw.blend(list(weighted), list(blind), seed, limit, key=lambda it: it.id)
|
||||
else:
|
||||
items = list(blind)
|
||||
states = rollup.item_state_map(db, uid, [it.id for it in items])
|
||||
return [movie_card(it, states.get(it.id), lang=lang) for it in items]
|
||||
|
||||
|
||||
@@ -14,10 +14,10 @@ Resolution works both before extraction (from source_kind / URL host) and after
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import yt_dlp
|
||||
from yt_dlp.networking.impersonate import ImpersonateTarget
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import egress
|
||||
from app.downloads.stated import StatedFailure
|
||||
|
||||
# Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
|
||||
@@ -33,6 +33,9 @@ class SourceAdapter:
|
||||
"""Generic source: yt-dlp handles everything and the ref is already a fetchable URL."""
|
||||
|
||||
key = "generic"
|
||||
# SEC-17: every fetch of this source's download goes through the egress gate (`egress`). A
|
||||
# source's own host list is no guarantee — its pages, redirects and formats can point anywhere.
|
||||
egress_guarded = True
|
||||
|
||||
# --- extraction (worker._extract_with_fallback / backfill) ---------------------------------
|
||||
def resolve_url(self, url: str, ydl_opts: dict) -> str:
|
||||
@@ -66,6 +69,11 @@ class SourceAdapter:
|
||||
"""The download's raw title, before `titles.normalize_title` cleans it. Generic: yt-dlp's."""
|
||||
return info.get("title")
|
||||
|
||||
def is_placeholder_title(self, title: str | None) -> bool:
|
||||
"""Whether `title` is a stand-in the source gives in place of the real one (worth one more,
|
||||
signed-out read of the page). Generic: never."""
|
||||
return False
|
||||
|
||||
def uploader_url(self, info: dict) -> str | None:
|
||||
"""The channel/uploader page URL. Generic: whatever yt-dlp exposes directly (YouTube gives
|
||||
`channel_url`), else None (reddit / a raw media URL have no derivable channel page)."""
|
||||
@@ -87,6 +95,9 @@ class SourceAdapter:
|
||||
|
||||
class YouTubeAdapter(SourceAdapter):
|
||||
key = "youtube"
|
||||
# Not through the gate: YouTube's fetches go to Google's fixed hosts, and the PO-token provider
|
||||
# it needs is the sidecar on our internal network, which the gate exists to refuse.
|
||||
egress_guarded = False
|
||||
# Errors worth retrying with a different player-client set (YouTube per-client flakiness).
|
||||
_RETRIABLE = (
|
||||
"not a bot",
|
||||
@@ -174,7 +185,7 @@ class FacebookAdapter(SourceAdapter):
|
||||
post_id = facebook_group_post_id(url)
|
||||
if post_id is None or not ydl_opts.get("cookiefile"):
|
||||
return url
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
with egress.open_ydl(ydl_opts) as ydl:
|
||||
raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
|
||||
page = raw.decode("utf-8", "replace")
|
||||
# Measured: a refused or expired cookie gets the login page (~450 KB, a login
|
||||
@@ -201,11 +212,17 @@ class FacebookAdapter(SourceAdapter):
|
||||
# Measured (2026-09-14): a SIGNED-IN read gives the placeholder "Video" (plain and as Chrome), and an
|
||||
# 18+ reel only readable as Chrome gives "Facebook"; the post's text is still in `description`. A
|
||||
# signed-out read titles the page with that same text behind the engagement counts, cut short.
|
||||
# Measured (2026-11-01, yt-dlp 2026.08.19): since 2026-10-28 every signed-in reel read gives
|
||||
# "Facebook" and NO description, while a signed-out read of the same reel still carries the text —
|
||||
# so a placeholder left with nothing behind it gets one signed-out read (`backfill.public_title_info`).
|
||||
_PLACEHOLDER_TITLES = frozenset({"video", "facebook"})
|
||||
|
||||
def is_placeholder_title(self, title: str | None) -> bool:
|
||||
return (title or "").strip().lower() in self._PLACEHOLDER_TITLES
|
||||
|
||||
def title(self, info: dict) -> str | None:
|
||||
raw = info.get("title")
|
||||
if (raw or "").strip().lower() not in self._PLACEHOLDER_TITLES:
|
||||
if not self.is_placeholder_title(raw):
|
||||
return raw
|
||||
line = next(
|
||||
(ln.strip() for ln in (info.get("description") or "").splitlines() if ln.strip()), ""
|
||||
|
||||
@@ -5,7 +5,10 @@ auto-detected channel becomes a clickable link) and `poster_path` (an ffmpeg fra
|
||||
thumbnail-less sources, so the card / watch page / link preview still shows an image). Best-effort
|
||||
and safe to re-run: it only touches assets still missing the field, and only re-fetches YouTube /
|
||||
Facebook sources (a direct media URL like a reddit HLS manifest carries no channel and its signed
|
||||
URL may have expired — those just get a locally-generated poster instead).
|
||||
URL may have expired — those just get a locally-generated poster instead). A Facebook download
|
||||
titled with the source's placeholder ("Facebook", "Video") is retitled from a signed-out read of its
|
||||
page, and so is each download name that only repeated the placeholder. That page read
|
||||
(`public_title_info`) also titles every new download, from the worker.
|
||||
|
||||
Run once per environment:
|
||||
docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
|
||||
@@ -13,21 +16,23 @@ Run once per environment:
|
||||
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import yt_dlp
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import select, update
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import logs
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.downloads import adapters
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import formats, service
|
||||
from app.downloads.models import MediaAsset
|
||||
from app.downloads import egress, formats, service
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
from app.models import DISPLAY_TEXT_MAX
|
||||
from app.titles import normalize_title
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
|
||||
def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
|
||||
def extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
|
||||
"""Metadata-only yt-dlp extraction, using the source adapter's client sets + options (YouTube
|
||||
needs the POT/player-client strategy even just to extract)."""
|
||||
with formats.hiding(url):
|
||||
@@ -37,11 +42,15 @@ def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> d
|
||||
def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
|
||||
for clients in adapter.client_sets():
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "%(id)s.%(ext)s", lambda d: None, extra=adapter.ydl_opts_extra(clients)
|
||||
{},
|
||||
"%(id)s.%(ext)s",
|
||||
lambda d: None,
|
||||
extra=adapter.ydl_opts_extra(clients),
|
||||
egress=adapter.egress_guarded,
|
||||
)
|
||||
opts["skip_download"] = True
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(opts) as ydl:
|
||||
with egress.open_ydl(opts) as ydl:
|
||||
info = ydl.extract_info(url, download=False)
|
||||
if info:
|
||||
return info
|
||||
@@ -62,8 +71,43 @@ def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: in
|
||||
return None
|
||||
|
||||
|
||||
def public_title_info(info: dict, url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict:
|
||||
"""`info` to title a download from: itself, unless its title is a source's placeholder with
|
||||
nothing behind it (a signed-in Facebook read: "Facebook", no description). Then one signed-out,
|
||||
metadata-only read of the page lends its text — the description, else its title — as the
|
||||
description the adapter takes a placeholder's title from. Nothing better there: `info` as it was."""
|
||||
if not adapter.is_placeholder_title(adapter.title(info)):
|
||||
return info
|
||||
public = extract_meta(url, adapter, asset_id)
|
||||
text = public and (public.get("description") or public.get("title"))
|
||||
return {**info, "description": text} if text else info
|
||||
|
||||
|
||||
def _retitle(db: Session, a: MediaAsset, adapter: adapters.SourceAdapter, url: str) -> bool:
|
||||
"""Swap a placeholder title kept from an earlier download for the page's text, and the download
|
||||
names that only repeated it. A name its owner gave (anything else) stays."""
|
||||
if not adapter.is_placeholder_title(a.title):
|
||||
return False
|
||||
info = public_title_info({"title": a.title, "description": None}, url, adapter, a.id)
|
||||
named = normalize_title(adapter.title(info))
|
||||
if not named or adapter.is_placeholder_title(named):
|
||||
return False
|
||||
# One conditional write, not a read and a compare: a name its owner gives while the page is read
|
||||
# is not overwritten by what was read before it. The asset first, as every other writer of the
|
||||
# two locks them (`service.lock_own_job`): the session does not flush by itself.
|
||||
placeholder = a.title
|
||||
a.title = named
|
||||
db.flush()
|
||||
db.execute(
|
||||
update(DownloadJob)
|
||||
.where(DownloadJob.asset_id == a.id, DownloadJob.display_name == placeholder)
|
||||
.values(display_name=named[:DISPLAY_TEXT_MAX])
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def run() -> dict:
|
||||
filled_url = filled_poster = 0
|
||||
filled_url = filled_poster = retitled = 0
|
||||
with SessionLocal() as db:
|
||||
assets = (
|
||||
db.execute(
|
||||
@@ -78,7 +122,7 @@ def run() -> dict:
|
||||
adapter = adapters.resolve_adapter(a.source_kind, a.source_ref)
|
||||
if a.uploader_url is None and adapter.wants_metadata_backfill():
|
||||
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
|
||||
info = _extract_meta(url, adapter, a.id)
|
||||
info = extract_meta(url, adapter, a.id)
|
||||
uu = adapter.uploader_url(info) if info else None
|
||||
if uu:
|
||||
a.uploader_url = uu
|
||||
@@ -86,6 +130,14 @@ def run() -> dict:
|
||||
log.info(
|
||||
"asset %s uploader_url filled", a.id
|
||||
) # the URL names what they downloaded
|
||||
if adapter.wants_metadata_backfill():
|
||||
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
|
||||
if _retitle(db, a, adapter, url):
|
||||
# Before the poster step: it can cut a frame for 30 s, and these locks held
|
||||
# across it block a rename and deadlock a delete.
|
||||
db.commit()
|
||||
retitled += 1
|
||||
log.info("asset %s retitled", a.id) # the title names what they downloaded
|
||||
if a.poster_path is None:
|
||||
# Record a self-hosted poster for every download (the reliable og:image source):
|
||||
# ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
|
||||
@@ -99,6 +151,7 @@ def run() -> dict:
|
||||
"assets": len(assets),
|
||||
"filled_uploader_url": filled_url,
|
||||
"filled_poster": filled_poster,
|
||||
"retitled": retitled,
|
||||
}
|
||||
log.info("backfill done: %s", result)
|
||||
return result
|
||||
|
||||
@@ -12,6 +12,9 @@ given in fractions of the box it was drawn in, with that box's aspect: the YouTu
|
||||
tells its picture's size, so the frame there is drawn over the player's 16:9 box (a short's 9:16),
|
||||
and the cut maps it onto the real picture (`crop_filter`) — what lies over a letterbox bar is dropped.
|
||||
|
||||
A clip may be a GIF instead (`spec["gif"]`, downloads/gif): the same range, crop and tonemap, then a
|
||||
silent animated picture of at most 30 s rather than a video.
|
||||
|
||||
The spec is stored on the job (`DownloadJob.edit_spec`) and is the clip's whole identity: its
|
||||
signature is the asset's cache key, so marking the same range again is an instant cache hit, and a
|
||||
different range or audio track is a different clip.
|
||||
@@ -22,7 +25,9 @@ import json
|
||||
import math
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads.edit import AAC_ARGS, EVEN_FRAME, INPUT_GUARD, X264_ARGS
|
||||
from app.plex.sync import HDR_TRANSFERS
|
||||
|
||||
@@ -57,12 +62,29 @@ _CROP_ASPECTS = (0.25, 4.0)
|
||||
_EDGE_SLACK = 1e-3
|
||||
|
||||
|
||||
# Every reason a clip is refused; the UI names each one (`clip.errors.<reason>`, held by the frontend's
|
||||
# backendIds contract, which reads this Literal).
|
||||
ClipReason = Literal[
|
||||
"bad_source",
|
||||
"bad_range",
|
||||
"too_short",
|
||||
"too_long",
|
||||
"bad_audio",
|
||||
"bad_crop",
|
||||
"not_found",
|
||||
"media_missing",
|
||||
"past_end",
|
||||
"bad_gif",
|
||||
"gif_too_long",
|
||||
]
|
||||
|
||||
|
||||
class ClipError(Exception):
|
||||
"""A clip request that cannot be cut; `reason` is an i18n-mappable key, `params` the numbers its
|
||||
message names (the cap itself, so the client never says a stale one)."""
|
||||
|
||||
def __init__(self, reason: str, **params: float):
|
||||
self.reason = reason
|
||||
def __init__(self, reason: ClipReason, **params: float):
|
||||
self.reason: ClipReason = reason
|
||||
self.params = params
|
||||
super().__init__(reason)
|
||||
|
||||
@@ -74,9 +96,12 @@ def normalize(
|
||||
end_s: float,
|
||||
audio_track: int | None = None,
|
||||
crop: dict | None = None,
|
||||
gif_side: object = None,
|
||||
) -> dict:
|
||||
"""The validated, canonical clip spec. Times are rounded to the millisecond and a crop to four
|
||||
decimals, so the same marks always give the same cache key. Raises ClipError."""
|
||||
decimals, so the same marks always give the same cache key. A `gif_side` makes it a GIF of that
|
||||
size (`downloads/gif`), held to the GIF's own length cap; without one it is the video clip it
|
||||
always was, spec and cache key alike. Raises ClipError."""
|
||||
if source not in SOURCES:
|
||||
raise ClipError("bad_source")
|
||||
pattern = _YOUTUBE_ID if source == "youtube" else _RATING_KEY
|
||||
@@ -85,12 +110,27 @@ def normalize(
|
||||
if not (math.isfinite(start_s) and math.isfinite(end_s)) or start_s < 0:
|
||||
raise ClipError("bad_range")
|
||||
start, end = round(start_s, 3), round(end_s, 3)
|
||||
if end - start < MIN_CLIP_S:
|
||||
# Judged in whole milliseconds, as the ends are kept: a float difference lands a hair off
|
||||
# (60.529 - 30.529 = 30.000000000000004), which would refuse a range exactly at a cap.
|
||||
span_ms = round((end - start) * 1000)
|
||||
if span_ms < MIN_CLIP_S * 1000:
|
||||
raise ClipError("too_short")
|
||||
if end - start > MAX_CLIP_S:
|
||||
side = None
|
||||
if gif_side is not None:
|
||||
try:
|
||||
side = gifmod.normalize_side(gif_side)
|
||||
except ValueError as e:
|
||||
raise ClipError("bad_gif") from e
|
||||
if side is not None and gifmod.too_long(start, end):
|
||||
# Its own reason: a GIF's cap is said in seconds, a clip's in minutes.
|
||||
raise ClipError("gif_too_long", limit_s=gifmod.MAX_S)
|
||||
if span_ms > MAX_CLIP_S * 1000:
|
||||
raise ClipError("too_long", limit_s=MAX_CLIP_S)
|
||||
spec: dict = {"source": source, "ref": ref, "start_s": start, "end_s": end}
|
||||
if source == "plex" and audio_track is not None:
|
||||
if side is not None:
|
||||
spec["gif"] = {"side": side}
|
||||
# A GIF has no sound: the track the player was on is not part of what it is (its cache key).
|
||||
if source == "plex" and audio_track is not None and side is None:
|
||||
if not 0 <= audio_track <= _MAX_AUDIO_TRACK:
|
||||
raise ClipError("bad_audio")
|
||||
spec["audio_track"] = audio_track
|
||||
@@ -185,6 +225,16 @@ def _ratio(v: object) -> float | None:
|
||||
return n / d if n > 0 and d > 0 else None
|
||||
|
||||
|
||||
def quarter_turned(probe: dict) -> bool | None:
|
||||
"""Whether the first video stream of an ffprobe `-show_streams` JSON is stored turned a quarter
|
||||
(ffmpeg stands it upright before any filter, so its filters see width and height swapped). None
|
||||
when there is no video stream."""
|
||||
for s in probe.get("streams") or []:
|
||||
if s.get("codec_type") == "video":
|
||||
return _quarter_turned(s)
|
||||
return None
|
||||
|
||||
|
||||
def _quarter_turned(stream: dict) -> bool:
|
||||
rotation: object = (stream.get("tags") or {}).get("rotate")
|
||||
for sd in stream.get("side_data_list") or []:
|
||||
@@ -231,6 +281,12 @@ def audio_track_count(probe: dict) -> int:
|
||||
return sum(1 for s in probe.get("streams") or [] if s.get("codec_type") == "audio")
|
||||
|
||||
|
||||
def gif_pre(crop: str, hdr: bool) -> str:
|
||||
"""What a GIF clip's frames go through before the GIF's own size and palette: the crop, then an HDR
|
||||
source's tonemap (a palette made from PQ frames is as washed out as the frames are)."""
|
||||
return ",".join([*([crop] if crop else []), *([_TONEMAP] if hdr else [])])
|
||||
|
||||
|
||||
def build_cmd(
|
||||
src: Path,
|
||||
dest: Path,
|
||||
|
||||
@@ -25,6 +25,7 @@ Security controls enforced here:
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -163,9 +164,18 @@ def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredenti
|
||||
).scalar_one_or_none()
|
||||
|
||||
|
||||
# Every reason a pasted cookies.txt is refused; the UI names each one (`downloads.errors.cookies.<reason>`,
|
||||
# held by the frontend's backendIds contract, which reads this Literal).
|
||||
CookieReason = Literal["empty", "format"]
|
||||
|
||||
|
||||
class InvalidCookies(ValueError):
|
||||
"""The pasted cookies aren't a usable Netscape cookies.txt — surfaced as a 400 by the route."""
|
||||
|
||||
def __init__(self, reason: CookieReason):
|
||||
super().__init__(reason)
|
||||
self.reason: CookieReason = reason
|
||||
|
||||
|
||||
def validate_cookies(raw: str) -> str:
|
||||
"""Sanity-check a pasted cookies.txt (Netscape format) and return the cleaned text. yt-dlp reads
|
||||
|
||||
@@ -11,7 +11,9 @@ with the worker branching on the asset's `source_kind` to run ffmpeg instead of
|
||||
(the `-ss` seeks first), so a short clip is cheap even from a long source
|
||||
* crop (± trim) → always re-encode (a filter can't be stream-copied), frame-accurate
|
||||
|
||||
`edit_spec = {trim?: {start_s, end_s}, crop?: {x, y, w, h}, accurate?: bool}`. `accurate` is the
|
||||
`edit_spec = {trim?: {start_s, end_s}, crop?: {x, y, w, h, frame_w?, frame_h?}, accurate?: bool}`. A
|
||||
crop with `frame_w`/`frame_h` counts the picture as it is SHOWN, measured on a frame of that size
|
||||
(`crop_filter`); one without them is in the file's stored pixels (its older form). `accurate` is the
|
||||
per-edit user choice for a trim-only cut (ignored when a crop forces a re-encode anyway). `split`
|
||||
is a frontend concept: the
|
||||
UI fans a split out into N trim jobs, so there is one output file per job here.
|
||||
@@ -60,6 +62,46 @@ class EditAborted(Exception):
|
||||
_MAX_SEGMENTS = 100
|
||||
|
||||
|
||||
def _crop_frame(crop: dict) -> dict | None:
|
||||
"""The picture size a crop was measured on, or None when it has none it could be scaled by (a
|
||||
spec from before it was sent, which `crop_filter` cuts in the file's own pixels)."""
|
||||
try:
|
||||
w, h = int(crop["frame_w"]), int(crop["frame_h"])
|
||||
# silent-ok: no frame is a crop's older form, and it is still the crop that was asked for. An
|
||||
# infinite number (JSON `1e400`) is OverflowError, and it is no more a frame than "x" is.
|
||||
except (KeyError, TypeError, ValueError, OverflowError):
|
||||
return None
|
||||
return {"frame_w": w, "frame_h": h} if w > 0 and h > 0 else None
|
||||
|
||||
|
||||
def crop_filter(crop: dict) -> str:
|
||||
"""The ffmpeg `crop` of an edit. Its numbers count the picture as the browser reports it, which
|
||||
is its SHOWN size: a video with wide pixels (320 stored, 640 shown) would be cut off to one side
|
||||
in ffmpeg's stored ones, so they are scaled to `iw` / `ih` by the size they were measured on. A
|
||||
rotated file needs nothing more: both the browser and ffmpeg's filters see it upright."""
|
||||
frame_w, frame_h = crop.get("frame_w"), crop.get("frame_h")
|
||||
if not frame_w or not frame_h:
|
||||
return f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}"
|
||||
return (
|
||||
f"crop=iw*{crop['w']}/{frame_w}:ih*{crop['h']}/{frame_h}"
|
||||
f":iw*{crop['x']}/{frame_w}:ih*{crop['y']}/{frame_h}"
|
||||
)
|
||||
|
||||
|
||||
def crop_stored_size(crop: dict, src_w: int | None, src_h: int | None) -> tuple[int, int] | None:
|
||||
"""The size `crop_filter` cuts out of a `src_w` x `src_h` frame, evened as `EVEN_FRAME` evens it —
|
||||
the clip's size when ffprobe cannot tell. The frame is the one ffmpeg's filters see: a rotated
|
||||
file's stood upright, its stored size turned (the caller's to turn). A crop counted in the shown
|
||||
picture is a fraction of that frame, not its own pixel count; None when the size is unknown."""
|
||||
if crop.get("frame_w") and crop.get("frame_h"):
|
||||
if not src_w or not src_h:
|
||||
return None
|
||||
w, h = src_w * crop["w"] // crop["frame_w"], src_h * crop["h"] // crop["frame_h"]
|
||||
else:
|
||||
w, h = crop["w"], crop["h"]
|
||||
return w // 2 * 2, h // 2 * 2
|
||||
|
||||
|
||||
def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
"""Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
|
||||
|
||||
@@ -67,12 +109,18 @@ def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
spec = spec or {}
|
||||
out: dict = {}
|
||||
|
||||
# A GIF output (downloads/gif) keeps its size; one it cannot name is dropped here and refused by
|
||||
# the service, which reads the raw spec for exactly that (`gif_side`).
|
||||
side = gif_side(spec)
|
||||
if side is not None:
|
||||
out["gif"] = {"side": side}
|
||||
|
||||
# Crop applies to both a single trim and a multi-segment join.
|
||||
crop = spec.get("crop") or {}
|
||||
if crop:
|
||||
try:
|
||||
c = {k: int(round(float(crop[k]))) for k in ("x", "y", "w", "h")}
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
except (KeyError, TypeError, ValueError, OverflowError) as exc:
|
||||
# The clip still renders — uncropped. The user asked for a crop, got a whole frame,
|
||||
# and nothing anywhere called it a failure; this line is the only trace there is.
|
||||
log.warning(
|
||||
@@ -81,7 +129,8 @@ def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
)
|
||||
c = None
|
||||
if c and c["w"] > 0 and c["h"] > 0 and c["x"] >= 0 and c["y"] >= 0:
|
||||
out["crop"] = c
|
||||
frame = _crop_frame(crop)
|
||||
out["crop"] = {**c, **frame} if frame else c
|
||||
|
||||
# Multi-segment JOIN (cut-list → one concatenated file). Takes precedence over `trim`.
|
||||
segs = spec.get("segments")
|
||||
@@ -154,6 +203,21 @@ def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
return out
|
||||
|
||||
|
||||
def gif_side(spec: dict | None) -> int | None:
|
||||
"""The GIF size a raw edit spec asks for, or None when it asks for none — or for one that is not
|
||||
offered (the service tells the two apart by `"gif" in spec`)."""
|
||||
# Imported here: `gif` builds its command on this module's input guard.
|
||||
from app.downloads import gif as gifmod
|
||||
|
||||
raw = (spec or {}).get("gif")
|
||||
if not isinstance(raw, dict):
|
||||
return None
|
||||
try:
|
||||
return gifmod.normalize_side(raw.get("side"))
|
||||
except ValueError: # silent-ok: an unoffered size is refused by the service (`bad_gif`)
|
||||
return None
|
||||
|
||||
|
||||
def edit_sig(spec: dict) -> str:
|
||||
"""Stable 24-char signature of a (normalized) edit spec — the per-user cache identity."""
|
||||
payload = json.dumps(normalize_edit_spec(spec), sort_keys=True, separators=(",", ":"))
|
||||
@@ -165,8 +229,8 @@ def has_crop(spec: dict) -> bool:
|
||||
|
||||
|
||||
def needs_reencode(spec: dict) -> bool:
|
||||
"""A crop always re-encodes; a trim re-encodes only when the user asked for an accurate cut."""
|
||||
return bool(spec.get("crop")) or bool(spec.get("accurate"))
|
||||
"""A crop or a GIF always re-encodes; a trim only when the user asked for an accurate cut."""
|
||||
return bool(spec.get("crop")) or bool(spec.get("gif")) or bool(spec.get("accurate"))
|
||||
|
||||
|
||||
def clip_duration(spec: dict, source_duration: int | None) -> int | None:
|
||||
@@ -268,7 +332,7 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
|
||||
if reencode:
|
||||
# Input -ss + re-encode is frame-accurate (decodes from the prior keyframe, discards up to
|
||||
# the exact cut). A crop adds the filter; an accurate trim only evens the frame.
|
||||
cropf = f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}," if crop else ""
|
||||
cropf = f"{crop_filter(crop)}," if crop else ""
|
||||
cmd += ["-vf", cropf + EVEN_FRAME, *X264_ARGS, *AAC_ARGS]
|
||||
else:
|
||||
cmd += ["-c", "copy", "-avoid_negative_ts", "make_zero"]
|
||||
@@ -313,7 +377,7 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
|
||||
]
|
||||
|
||||
if needs_reencode(spec):
|
||||
cropf = f",crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}" if crop else ""
|
||||
cropf = f",{crop_filter(crop)}" if crop else ""
|
||||
parts, labels = [], []
|
||||
for i, s in enumerate(segs):
|
||||
st, en = s["start_s"], s["end_s"]
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
"""The egress gate: every fetch of a user-supplied download goes through here, and this is where the
|
||||
address is judged — at the moment of the connection, not when the URL was typed (SEC-17).
|
||||
|
||||
`urlguard` judges the URL the user typed, at enqueue and again in the worker. What that cannot see is
|
||||
every request yt-dlp makes after it: a public host answering 3xx with an internal `Location`, a page
|
||||
whose formats point inside, a name that resolves public for the check and private for the fetch (DNS
|
||||
rebinding). So a guarded download runs with yt-dlp's `proxy` set to this gate, a small HTTP proxy on
|
||||
the worker's loopback. For each request it resolves the host ONCE, refuses the connection if any
|
||||
address the name has is one the server must not fetch from (`urlguard.is_forbidden`), and otherwise
|
||||
connects to the very address it vetted — there is no second lookup for a rebinding answer to slip into.
|
||||
|
||||
Measured (yt-dlp 2026.8.19, 2026-11-06) on the worker image, with a logging proxy: the `proxy` option
|
||||
reaches every path a download takes — the Requests handler, the CurlCFFI one an impersonated request
|
||||
uses, the native HLS downloader and an ffmpeg HLS download (segments included). ffmpeg itself also
|
||||
speaks rtmp, rtsp, ftp, tcp and more, which no HTTP proxy covers, so a guarded download keeps only the
|
||||
formats it fetches over http(s) (`HttpOnlyFormats`, added by `open_ydl`) and fetches HLS natively.
|
||||
|
||||
A refused connection is answered `403` before anything is dialled, so a live internal port and a dead
|
||||
one read the same; `errors.user_error_text` then stores the one redacted sentence for both.
|
||||
|
||||
Not used for YouTube: those downloads talk to Google's fixed hosts and to the PO-token sidecar, which
|
||||
lives on the internal network on purpose (`adapters.YouTubeAdapter.egress_guarded`).
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import select
|
||||
import socket
|
||||
import socketserver
|
||||
import threading
|
||||
from collections.abc import Callable
|
||||
from typing import Any, cast
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import yt_dlp
|
||||
from yt_dlp.postprocessor.common import PostProcessor
|
||||
|
||||
from app import logs
|
||||
from app.downloads import urlguard
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
Resolver = Callable[..., list]
|
||||
Forbidden = Callable[[ipaddress.IPv4Address | ipaddress.IPv6Address], bool]
|
||||
|
||||
_BLOCKED = b"HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
_BAD_GATEWAY = b"HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
_BAD_REQUEST = b"HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
_MAX_HEAD = 64 * 1024
|
||||
# A tunnel with no traffic either way for this long is closed. Above yt-dlp's own socket_timeout
|
||||
# (30 s, `formats.build_ydl_opts`), so the gate never ends a transfer yt-dlp would still wait for.
|
||||
_IDLE_S = 120
|
||||
# Hop-by-hop headers a proxy must not pass on; `Connection: close` replaces them, so one connection
|
||||
# carries one request and a reused client connection can never be steered to a second host unvetted.
|
||||
_HOP = (b"proxy-", b"connection:", b"keep-alive:")
|
||||
|
||||
|
||||
def vetted_address(
|
||||
host: str, port: int, *, resolve: Resolver = socket.getaddrinfo, forbidden: Forbidden
|
||||
) -> tuple[str, int] | None:
|
||||
"""The address to connect to for `host:port`, or None when it must not be fetched: the name has
|
||||
no answer, or ANY of its addresses is forbidden (a round-robin answer is not a coin flip). IPv4
|
||||
only, as every download is (`source_address` 0.0.0.0 in `formats.build_ydl_opts`)."""
|
||||
try:
|
||||
infos = resolve(host, port, socket.AF_INET, socket.SOCK_STREAM)
|
||||
except (socket.gaierror, UnicodeError, OSError):
|
||||
# silent-ok: "no answer" is a refusal, and the caller's refusal is the line that is logged.
|
||||
return None
|
||||
addrs = [info[4] for info in infos]
|
||||
if not addrs:
|
||||
return None
|
||||
for addr in addrs:
|
||||
try:
|
||||
ip = ipaddress.ip_address(addr[0])
|
||||
except ValueError:
|
||||
# silent-ok: an answer that is not an address is refused like a forbidden one, and the
|
||||
# caller logs the refusal.
|
||||
return None
|
||||
if forbidden(ip):
|
||||
return None
|
||||
return addrs[0][0], addrs[0][1]
|
||||
|
||||
|
||||
def _read_head(sock: socket.socket) -> bytes | None:
|
||||
buf = b""
|
||||
while b"\r\n\r\n" not in buf:
|
||||
if len(buf) > _MAX_HEAD:
|
||||
return None
|
||||
chunk = sock.recv(8192)
|
||||
if not chunk:
|
||||
return None
|
||||
buf += chunk
|
||||
return buf
|
||||
|
||||
|
||||
def _pipe(a: socket.socket, b: socket.socket) -> None:
|
||||
try:
|
||||
_relay(a, b)
|
||||
except OSError:
|
||||
# silent-ok: one side hung up mid-transfer (yt-dlp closes a connection once it has read a
|
||||
# file's size, or gives up on a slow host); whichever side cared reports it itself.
|
||||
return
|
||||
|
||||
|
||||
def _relay(a: socket.socket, b: socket.socket) -> None:
|
||||
socks = [a, b]
|
||||
while True:
|
||||
ready, _, _ = select.select(socks, [], [], _IDLE_S)
|
||||
if not ready:
|
||||
return
|
||||
for s in ready:
|
||||
data = s.recv(65536)
|
||||
if not data:
|
||||
return
|
||||
(b if s is a else a).sendall(data)
|
||||
|
||||
|
||||
class _Handler(socketserver.BaseRequestHandler):
|
||||
def handle(self) -> None:
|
||||
client: socket.socket = self.request
|
||||
head = _read_head(client)
|
||||
if head is None:
|
||||
return
|
||||
head_part, _, body = head.partition(b"\r\n\r\n")
|
||||
lines = head_part.split(b"\r\n")
|
||||
try:
|
||||
method, target, version = lines[0].decode("latin1").split(" ", 2)
|
||||
except ValueError:
|
||||
# silent-ok: the only client is yt-dlp on this loopback; a malformed request is answered
|
||||
# 400 and reaches its own error, which the job records.
|
||||
client.sendall(_BAD_REQUEST)
|
||||
return
|
||||
if method == "CONNECT":
|
||||
host, _, port_s = target.rpartition(":")
|
||||
host = host.strip("[]")
|
||||
upstream = self._dial(host, port_s)
|
||||
if upstream is None:
|
||||
return
|
||||
with upstream:
|
||||
client.sendall(b"HTTP/1.1 200 Connection established\r\n\r\n")
|
||||
if body:
|
||||
upstream.sendall(body)
|
||||
_pipe(client, upstream)
|
||||
return
|
||||
u = urlsplit(target)
|
||||
if u.scheme != "http" or not u.hostname:
|
||||
client.sendall(_BLOCKED)
|
||||
return
|
||||
try:
|
||||
port = u.port or 80
|
||||
except ValueError:
|
||||
# silent-ok: as for a malformed request line above — answered 400, recorded by the job.
|
||||
client.sendall(_BAD_REQUEST)
|
||||
return
|
||||
upstream = self._dial(u.hostname, str(port))
|
||||
if upstream is None:
|
||||
return
|
||||
path = (u.path or "/") + (f"?{u.query}" if u.query else "")
|
||||
headers = [h for h in lines[1:] if not h.lower().startswith(_HOP)]
|
||||
request = b"\r\n".join(
|
||||
[f"{method} {path} {version}".encode("latin1"), *headers, b"Connection: close"]
|
||||
)
|
||||
with upstream:
|
||||
upstream.sendall(request + b"\r\n\r\n" + body)
|
||||
_pipe(client, upstream)
|
||||
|
||||
def _dial(self, host: str, port_s: str) -> socket.socket | None:
|
||||
"""Connect to the address `host` was vetted at, or answer the client and return None."""
|
||||
client: socket.socket = self.request
|
||||
try:
|
||||
port = int(port_s)
|
||||
except ValueError:
|
||||
# silent-ok: as for a malformed request line above — answered 400, recorded by the job.
|
||||
client.sendall(_BAD_REQUEST)
|
||||
return None
|
||||
gate = cast("_Gate", self.server)
|
||||
addr = vetted_address(host, port, resolve=gate.resolve, forbidden=gate.forbidden)
|
||||
if addr is None:
|
||||
log.warning(
|
||||
"egress refused a connection to %s:%s",
|
||||
host, # log-pii-ok: the host a download tried to reach, never its path or query
|
||||
port,
|
||||
extra=logs.event("download.egress.refused", port=port),
|
||||
)
|
||||
client.sendall(_BLOCKED)
|
||||
return None
|
||||
try:
|
||||
return socket.create_connection(addr, timeout=_IDLE_S)
|
||||
except OSError:
|
||||
# silent-ok: a public host that does not answer; yt-dlp reports it as the job's own
|
||||
# network failure, which the job records.
|
||||
client.sendall(_BAD_GATEWAY)
|
||||
return None
|
||||
|
||||
|
||||
class _Gate(socketserver.ThreadingTCPServer):
|
||||
daemon_threads = True
|
||||
allow_reuse_address = True
|
||||
|
||||
def __init__(self, resolve: Resolver, forbidden: Forbidden):
|
||||
self.resolve = resolve
|
||||
self.forbidden = forbidden
|
||||
super().__init__(("127.0.0.1", 0), _Handler)
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
return f"http://127.0.0.1:{self.server_address[1]}"
|
||||
|
||||
def handle_error(self, request, client_address) -> None:
|
||||
# socketserver's default prints a bare traceback to stderr, outside the log; a connection the
|
||||
# gate could not finish is a line with its code instead.
|
||||
log.exception("egress gate dropped a connection", extra=logs.event("download.egress.error"))
|
||||
|
||||
|
||||
def start_gate(
|
||||
*, resolve: Resolver = socket.getaddrinfo, forbidden: Forbidden = urlguard.is_forbidden
|
||||
) -> _Gate:
|
||||
"""A gate serving on its own thread. The worker uses the one `proxy_url` starts; a test starts
|
||||
its own, with a resolver and a judgement it controls."""
|
||||
gate = _Gate(resolve, forbidden)
|
||||
threading.Thread(target=gate.serve_forever, name="egress-gate", daemon=True).start()
|
||||
return gate
|
||||
|
||||
|
||||
# The options key `formats.build_ydl_opts` marks a guarded download with; `open_ydl` reads it. yt-dlp
|
||||
# keeps unknown keys in its params and never acts on them.
|
||||
GUARDED = "siftlode_egress_guarded"
|
||||
|
||||
|
||||
def open_ydl(opts: dict) -> yt_dlp.YoutubeDL:
|
||||
"""The one way this package opens yt-dlp (a contract test holds it): a guarded download also
|
||||
gets `HttpOnlyFormats`, which yt-dlp takes only as an instance, not through the options dict."""
|
||||
# yt-dlp's stub types its params as a TypedDict; these are the plain dict `formats.build_ydl_opts`
|
||||
# assembles, the same every caller handed yt-dlp before.
|
||||
ydl = yt_dlp.YoutubeDL(opts) # type: ignore[arg-type]
|
||||
if opts.get(GUARDED):
|
||||
ydl.add_post_processor(HttpOnlyFormats(), when="pre_process")
|
||||
return ydl
|
||||
|
||||
|
||||
_gate: _Gate | None = None
|
||||
_gate_lock = threading.Lock()
|
||||
|
||||
|
||||
def proxy_url() -> str:
|
||||
"""This process's gate, started on first use (the worker's jobs share it)."""
|
||||
global _gate
|
||||
with _gate_lock:
|
||||
if _gate is None:
|
||||
_gate = start_gate()
|
||||
return _gate.url
|
||||
|
||||
|
||||
# The protocols fetched over http(s) through the proxy option — the only ones the gate sees: yt-dlp's
|
||||
# own fetchers (http, f4m, ism, the DASH segments and their generator, native HLS — which a bare m3u8
|
||||
# also gets, as a guarded download sets hls_prefer_native) and ffmpeg's http input (a live stream or
|
||||
# a clip), which takes the proxy too. Everything else (rtmp, rtsp, mms, ws) ffmpeg or another tool
|
||||
# would dial directly.
|
||||
_HTTP_PROTOCOLS = frozenset(
|
||||
{
|
||||
"http",
|
||||
"https",
|
||||
"f4m",
|
||||
"ism",
|
||||
"m3u8",
|
||||
"m3u8_native",
|
||||
"http_dash_segments",
|
||||
"http_dash_segments_generator",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class HttpOnlyFormats(PostProcessor):
|
||||
"""Drops every format a guarded download would fetch past the gate. Runs as `pre_process`, which
|
||||
yt-dlp re-reads its formats after ("The pre-processors may have modified the formats"), so the
|
||||
selector never sees them. With nothing left, yt-dlp reports no formats — a refusal, not a leak."""
|
||||
|
||||
def set_downloader(self, downloader):
|
||||
# Not the job's postprocessor hooks: they would report this filter as a "processing" step
|
||||
# before a byte is fetched (yt-dlp attaches them here, once per call).
|
||||
self._downloader = downloader
|
||||
|
||||
def run(self, information):
|
||||
# yt-dlp's stub types the info dict as a TypedDict with optional keys; it is a dict here.
|
||||
info = cast("dict[str, Any]", information)
|
||||
if info.get("formats"):
|
||||
info["formats"] = [f for f in info["formats"] if _over_http(f)]
|
||||
elif info.get("url") and not _over_http(info):
|
||||
info["url"] = None
|
||||
return [], information
|
||||
|
||||
|
||||
def _over_http(fmt: dict) -> bool:
|
||||
url = fmt.get("manifest_url") or fmt.get("url") or ""
|
||||
if urlsplit(url).scheme not in ("http", "https"):
|
||||
return False
|
||||
protocol = fmt.get("protocol") or urlsplit(fmt.get("url") or "").scheme
|
||||
return all(p in _HTTP_PROTOCOLS for p in protocol.split("+"))
|
||||
@@ -68,6 +68,10 @@ CONVERTED_TOO_BIG_MESSAGE = (
|
||||
"free up space, then try again."
|
||||
)
|
||||
UPLOAD_MISSING_MESSAGE = "The uploaded file is missing."
|
||||
# A GIF over `gif.MAX_BYTES`: not the storage left, a cap of the GIF's own (its own category).
|
||||
GIF_TOO_BIG_MESSAGE = (
|
||||
"The GIF came out larger than a GIF may be — mark a shorter range or choose a smaller size."
|
||||
)
|
||||
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
|
||||
|
||||
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
|
||||
@@ -77,6 +81,8 @@ _RULES: tuple[tuple[str, str], ...] = (
|
||||
# first — it is our wording, so it can never be a false match on an upstream message.
|
||||
("download storage is full", "quota_exceeded"),
|
||||
("larger than the download storage you have left", "quota_exceeded"),
|
||||
# Our own refusal of a GIF over its size cap — the range or the size, not the storage left.
|
||||
("larger than a gif may be", "gif_too_big"),
|
||||
# Our own redaction of a transport failure (see NETWORK_REDACTED_MESSAGE) — our wording too.
|
||||
(NETWORK_REDACTED_MESSAGE.lower(), "network"),
|
||||
# Our own verdicts on a post link (adapters.py) — our wording too: its page holds no single video,
|
||||
|
||||
@@ -24,6 +24,7 @@ kept it — and its `ready_at`, the moment it finished (`stamp_done`).
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from enum import StrEnum
|
||||
from typing import Literal
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from sqlalchemy import select
|
||||
@@ -181,12 +182,22 @@ def stamp_done(db: Session, job: DownloadJob) -> None:
|
||||
job.expiry_warned = False
|
||||
|
||||
|
||||
# Every folder change the rules refuse, answered as a generic refusal; the UI names each one
|
||||
# (`errors.refused.<reason>`, held by the frontend's backendIds contract, which reads this Literal).
|
||||
FolderReason = Literal[
|
||||
"file_folder_gone",
|
||||
"file_folder_name_taken",
|
||||
"file_folder_not_its_source",
|
||||
"file_folder_not_own",
|
||||
]
|
||||
|
||||
|
||||
class FolderError(Exception):
|
||||
"""A folder change the rules refuse; `reason` is the refusal the route answers with."""
|
||||
|
||||
def __init__(self, reason: str) -> None:
|
||||
def __init__(self, reason: FolderReason) -> None:
|
||||
super().__init__(reason)
|
||||
self.reason = reason
|
||||
self.reason: FolderReason = reason
|
||||
|
||||
|
||||
def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
|
||||
|
||||
@@ -28,6 +28,7 @@ import urllib.parse
|
||||
from collections.abc import Iterator
|
||||
|
||||
from app import logs
|
||||
from app.downloads import egress as egress_gate
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
@@ -294,6 +295,8 @@ def build_ydl_opts(
|
||||
extra: dict | None = None,
|
||||
cookiefile: str | None = None,
|
||||
max_bytes: int | None = None,
|
||||
*,
|
||||
egress: bool,
|
||||
) -> dict:
|
||||
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
|
||||
|
||||
@@ -309,7 +312,11 @@ def build_ydl_opts(
|
||||
a file announced larger than it rather than filling the disk and only then being caught by the
|
||||
quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
|
||||
a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
|
||||
pipeline also holds every produced file to what is left by its real size."""
|
||||
pipeline also holds every produced file to what is left by its real size.
|
||||
|
||||
`egress` (SEC-17) routes every fetch through the worker's egress gate, which refuses a connection
|
||||
into our own network, and keeps only the formats fetched over http(s). Required, with no default:
|
||||
a new caller has to say which kind of download it builds (`SourceAdapter.egress_guarded`)."""
|
||||
s = normalize(spec)
|
||||
h = s["max_height"]
|
||||
opts: dict = {
|
||||
@@ -364,6 +371,13 @@ def build_ydl_opts(
|
||||
if cookiefile:
|
||||
opts["cookiefile"] = cookiefile
|
||||
|
||||
# SEC-17. AFTER the adapter merge, like the cookie: an adapter must not be able to point a
|
||||
# guarded download past the gate.
|
||||
if egress:
|
||||
opts["proxy"] = egress_gate.proxy_url()
|
||||
opts["hls_prefer_native"] = True
|
||||
opts[egress_gate.GUARDED] = True
|
||||
|
||||
if s["mode"] == "a":
|
||||
opts["format"] = "bestaudio/best"
|
||||
opts["postprocessors"].append(
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""The GIF output of a clip or an edit: its limits and the one ffmpeg command every GIF goes through.
|
||||
|
||||
A GIF is a picture, not a video: no sound, a library `file` that previews as an image (ffprobe's `gif`
|
||||
format is one of `probe._IMAGE_FORMATS`), and never the source of another edit. The numbers were
|
||||
measured on real sources (2026-11-09/11):
|
||||
|
||||
- 12 frames a second, the long side capped (320 / 480 / 640): a width cap made a phone's portrait
|
||||
video three times the size of a landscape one (44 MB for 21 s at "480").
|
||||
- One palette per GIF, made from the GIF's own frames (`palettegen` → `paletteuse`, light ordered
|
||||
dither): the generic 256-colour palette bands every gradient.
|
||||
- 30 s at most. 480 px fills 12–29 MB in 30 s, depending on the picture; a GIF over `MAX_BYTES` is
|
||||
not kept, and the owner is told to cut less or smaller.
|
||||
- A chat link preview: Teams moves a GIF `og:image` up to 15 MB (22 MB showed nothing), so
|
||||
`PREVIEW_MAX_BYTES` decides whether the share page offers the GIF itself or its still (`ensure_still`).
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from app.downloads.edit import INPUT_GUARD
|
||||
|
||||
SIDES = (320, 480, 640)
|
||||
DEFAULT_SIDE = 480
|
||||
FPS = 12
|
||||
MAX_S = 30
|
||||
PREVIEW_MAX_BYTES = 15 * 1024 * 1024
|
||||
MAX_BYTES = 30 * 1024 * 1024
|
||||
|
||||
# The still is cut from the GIF we made, read with the GIF demuxer alone: a library file named .gif
|
||||
# whose bytes are a playlist must not open the paths inside it (see `edit.MEDIA_DEMUXERS`).
|
||||
_GIF_GUARD = ["-format_whitelist", "gif"]
|
||||
_STILL_TIMEOUT_S = 30
|
||||
|
||||
|
||||
def normalize_side(side: object) -> int:
|
||||
"""One of `SIDES`, or ValueError. A bool or a float is not a size, even when it equals one."""
|
||||
if type(side) is not int or side not in SIDES:
|
||||
raise ValueError(side)
|
||||
return side
|
||||
|
||||
|
||||
def too_long(start_s: float, end_s: float) -> bool:
|
||||
"""Over `MAX_S`, counted in whole milliseconds: the ends are kept to the millisecond, and their
|
||||
float difference may land a hair over (60.529 - 30.529 = 30.000000000000004), which would refuse
|
||||
the very 30 s the panels allow."""
|
||||
return round((end_s - start_s) * 1000) > MAX_S * 1000
|
||||
|
||||
|
||||
# A GIF has no pixel shape a viewer honours (browsers ignore its aspect byte), so a source stored
|
||||
# with wide or narrow pixels (an anamorphic DVD: 720x480 shown at 16:9) is first resampled to square
|
||||
# ones, or it shows squeezed. `sar` is 1 for a source that names none; even, as a 4:2:0 frame wants.
|
||||
_SQUARE_PIXELS = "scale=w='trunc(iw*sar/2)*2':h=ih,setsar=1"
|
||||
|
||||
|
||||
def _scale(side: int) -> str:
|
||||
"""Fit the long side into `side`, never upscaling a smaller picture; the short side keeps the
|
||||
aspect (-1: a GIF has no chroma grid that wants it even)."""
|
||||
return (
|
||||
f"scale=w='if(gte(iw,ih),min({side},iw),-1)'"
|
||||
f":h='if(gte(iw,ih),-1,min({side},ih))':flags=lanczos"
|
||||
)
|
||||
|
||||
|
||||
def filter_graph(pre: str, side: int) -> str:
|
||||
"""The GIF's filters: `pre` (a crop, an HDR tonemap) on the source's own frame first, then the
|
||||
frame rate, square pixels and the size, then a palette made from these frames and applied to them.
|
||||
A crop counts the stored frame (both callers' crops are fractions of `iw`/`ih`), so it comes
|
||||
before the pixels are squared."""
|
||||
head = ",".join([*([pre] if pre else []), f"fps={FPS}", _SQUARE_PIXELS, _scale(side)])
|
||||
return (
|
||||
f"[0:v:0]{head},split[a][b];"
|
||||
"[a]palettegen=stats_mode=diff[p];"
|
||||
"[b][p]paletteuse=dither=bayer:bayer_scale=5:diff_mode=rectangle"
|
||||
)
|
||||
|
||||
|
||||
def build_cmd(
|
||||
src: Path, dest: Path, start_s: float, dur_s: float, *, pre: str, side: int
|
||||
) -> list[str]:
|
||||
"""ffmpeg for one GIF: an input seek (the GIF starts on the marked frame), the first video stream
|
||||
through `filter_graph`, nothing else — no sound, subtitles, data or chapters."""
|
||||
return [
|
||||
"ffmpeg",
|
||||
"-y",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-nostdin",
|
||||
"-progress",
|
||||
"pipe:1",
|
||||
"-nostats",
|
||||
"-ss",
|
||||
f"{start_s:.3f}",
|
||||
*INPUT_GUARD,
|
||||
"-i",
|
||||
str(src),
|
||||
"-t",
|
||||
f"{dur_s:.3f}",
|
||||
"-filter_complex",
|
||||
filter_graph(pre, side),
|
||||
"-an",
|
||||
"-sn",
|
||||
"-dn",
|
||||
"-map_chapters",
|
||||
"-1",
|
||||
str(dest),
|
||||
]
|
||||
|
||||
|
||||
def size(path: Path) -> tuple[int, int] | None:
|
||||
"""(width, height) from a GIF's header — ffprobe reports none for an image format, and the share
|
||||
page's preview card wants them. None for anything that is not a GIF."""
|
||||
try:
|
||||
with path.open("rb") as f:
|
||||
head = f.read(10)
|
||||
# silent-ok: no size is what an unreadable file has; the card is then built without one.
|
||||
except OSError:
|
||||
return None
|
||||
if len(head) < 10 or head[:6] not in (b"GIF87a", b"GIF89a"):
|
||||
return None
|
||||
return int.from_bytes(head[6:8], "little"), int.from_bytes(head[8:10], "little")
|
||||
|
||||
|
||||
def ensure_still(path: Path) -> Path | None:
|
||||
"""A JPEG still of the GIF's first frame beside it (`<name>.jpg`, the poster sidecar every library
|
||||
file uses): the link preview for a chat that shows no GIF (Messenger) or a GIF too big to move.
|
||||
None when it cannot be cut — the card is then built without a picture."""
|
||||
dest = path.with_suffix(".jpg")
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
|
||||
*_GIF_GUARD, "-i", str(path), "-frames:v", "1", "-q:v", "3", str(dest),
|
||||
] # fmt: skip
|
||||
try:
|
||||
subprocess.run(cmd, capture_output=True, timeout=_STILL_TIMEOUT_S, check=True)
|
||||
# silent-ok: the still is best-effort; without it the card has no picture, which is true.
|
||||
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError):
|
||||
dest.unlink(missing_ok=True)
|
||||
return None
|
||||
return dest if dest.exists() else None
|
||||
@@ -127,14 +127,35 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
|
||||
# --- serialization -------------------------------------------------------------------------
|
||||
|
||||
|
||||
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
|
||||
def opens(db: Session, link: DownloadLink) -> str:
|
||||
"""What a link opens, in the words its address and its preview use: "folder", "video" or
|
||||
"file". A file still being made reads as the video it mostly is."""
|
||||
from app.downloads import mediakind
|
||||
|
||||
if link.folder_id is not None:
|
||||
return "folder"
|
||||
job = db.get(DownloadJob, link.job_id)
|
||||
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
|
||||
kind = asset.media_kind if asset else None
|
||||
return "file" if kind and kind != mediakind.VIDEO else "video"
|
||||
|
||||
|
||||
def share_path(db: Session, link: DownloadLink, token: str) -> str:
|
||||
"""The address a link is given out on: its path says what it opens (`opens`). Only a word for the
|
||||
reader, the token is the whole key: every prefix of `main.SHARE_PREFIXES` opens every link, so
|
||||
the links already sent on /watch/ work for good."""
|
||||
prefix = {"folder": "folder", "file": "file", "video": "watch"}[opens(db, link)]
|
||||
return f"/{prefix}/{token}"
|
||||
|
||||
|
||||
def owner_view(db: Session, link: DownloadLink, token: str | None = None) -> dict:
|
||||
"""What the file owner sees when managing their links. `token` is the one just made (it is in
|
||||
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
|
||||
token = token or recover(link)
|
||||
return {
|
||||
"id": link.id,
|
||||
"token": token,
|
||||
"url": f"{base_url}/watch/{token}" if token else None,
|
||||
"url": share_path(db, link, token) if token else None,
|
||||
"allow_download": link.allow_download,
|
||||
"has_password": link.password_hash is not None,
|
||||
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
|
||||
@@ -150,7 +171,7 @@ def public_meta(
|
||||
|
||||
Resolves the owner's per-download display overrides (custom title/channel) over the asset's
|
||||
auto-extracted values, and surfaces the source + any extra reference links as clickable URLs."""
|
||||
from app.downloads import mediakind, service
|
||||
from app.downloads import mediakind, preview, service
|
||||
|
||||
source_url = service.reference_url(job, asset)
|
||||
# Audio-only assets (mp3 rips, extracted audio) can't play inside a <video> element on WebKit
|
||||
@@ -168,6 +189,8 @@ def public_meta(
|
||||
"height": asset.height,
|
||||
"audio": asset.media_kind == mediakind.AUDIO,
|
||||
"media_kind": asset.media_kind,
|
||||
# A picture is shown as itself (the file route serves it inline); other files download.
|
||||
"preview": "image" if preview.kind(asset.media_kind, asset.container) == "image" else None,
|
||||
"size_bytes": asset.size_bytes,
|
||||
"container": asset.container,
|
||||
"allow_download": link.allow_download,
|
||||
|
||||
@@ -234,7 +234,9 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
|
||||
source_asset_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("media_assets.id", ondelete="SET NULL")
|
||||
)
|
||||
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
|
||||
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h,frame_w?,frame_h?}}, its
|
||||
# crop in the SHOWN picture's pixels when it carries the frame it was measured on (`edit.crop_filter`),
|
||||
# else in the file's stored ones. For clip
|
||||
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
|
||||
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
|
||||
edit_spec: Mapped[dict | None] = mapped_column(JSON)
|
||||
|
||||
+56
-18
@@ -1,4 +1,4 @@
|
||||
"""Server-rendered Open Graph tags for the public /watch/{token} page.
|
||||
"""Server-rendered Open Graph tags for the public page of a share link (/watch/, /file/, /folder/).
|
||||
|
||||
The watch page is a client-side SPA, so a social crawler (facebookexternalhit, Twitterbot, …) that
|
||||
fetches /watch/{token} would otherwise only see the generic index.html — a blank/greyed link card.
|
||||
@@ -14,8 +14,9 @@ from pathlib import Path
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import links as linksmod
|
||||
from app.downloads import storage
|
||||
from app.downloads import preview, storage
|
||||
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
|
||||
|
||||
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
|
||||
@@ -71,21 +72,51 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
|
||||
return None
|
||||
|
||||
|
||||
def _watch_tags(token: str, db: Session) -> str | None:
|
||||
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
|
||||
# Facebook's crawler (Messenger's previews): it never showed a GIF `og:image` and did show a JPEG one
|
||||
# (measured 2026-11-11), so it is given the still. Matched on its documented agent names.
|
||||
_FACEBOOK_AGENTS = ("facebookexternalhit", "facebot")
|
||||
|
||||
|
||||
def _is_facebook(user_agent: str | None) -> bool:
|
||||
agent = (user_agent or "").lower()
|
||||
return any(name in agent for name in _FACEBOOK_AGENTS)
|
||||
|
||||
|
||||
def _moving_image(
|
||||
token: str, asset: MediaAsset, user_agent: str | None
|
||||
) -> tuple[str, tuple[int, int] | None] | None:
|
||||
"""The shared GIF itself as the card's picture, and its size from its header — when a chat can
|
||||
move it: not for Facebook's crawler, and not over `gif.PREVIEW_MAX_BYTES` (Teams moved 15 MB and
|
||||
showed nothing at 22). None: the card takes the still, as any other file's does."""
|
||||
if preview.kind(asset.media_kind, asset.container) != "image" or asset.container != "gif":
|
||||
return None
|
||||
if _is_facebook(user_agent) or (asset.size_bytes or 0) > gifmod.PREVIEW_MAX_BYTES:
|
||||
return None
|
||||
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
|
||||
return f"{settings.app_base}/api/public/watch/{token}/file", gifmod.size(path) if path else None
|
||||
|
||||
|
||||
def _watch_tags(token: str, db: Session, user_agent: str | None = None) -> str | None:
|
||||
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card.
|
||||
`user_agent` is the crawler's: a GIF's card is the GIF for a chat that moves it (`_moving_image`)."""
|
||||
link = linksmod.find(db, token)
|
||||
if link is None or linksmod.is_expired(link):
|
||||
return None
|
||||
url = f"{settings.app_base}/watch/{token}"
|
||||
folder = link.folder_id is not None
|
||||
url = f"{settings.app_base}{linksmod.share_path(db, link, token)}"
|
||||
# What the card calls it, as its address does.
|
||||
kind = linksmod.opens(db, link)
|
||||
folder, video = kind == "folder", kind == "video"
|
||||
if link.password_hash:
|
||||
# Password-gated: don't leak the title/thumbnail in the unfurl.
|
||||
return "\n ".join(
|
||||
[
|
||||
"<title>Siftlode</title>",
|
||||
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
|
||||
_tag(
|
||||
"og:title",
|
||||
"Password-protected " + ("folder" if folder else "video" if video else "file"),
|
||||
),
|
||||
_tag("og:description", "Shared via Siftlode"),
|
||||
_tag("og:type", "website" if folder else "video.other"),
|
||||
_tag("og:type", "video.other" if video else "website"),
|
||||
_tag("og:url", url),
|
||||
_tag("og:site_name", "Siftlode"),
|
||||
]
|
||||
@@ -96,7 +127,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
|
||||
if job is None or asset is None or asset.status != "ready":
|
||||
return None
|
||||
title = (job.display_name or asset.title or "Video").strip()
|
||||
title = (job.display_name or asset.title or ("Video" if video else "File")).strip()
|
||||
channel = (job.display_uploader or asset.uploader or "").strip()
|
||||
# Pick the link-preview image. Facebook's scraper REJECTS our own poster ("Corrupted Image" in the
|
||||
# Sharing Debugger): the poster is an ffmpeg-produced JPEG whose non-standard marker order FB's
|
||||
@@ -107,7 +138,12 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
# non-password links, so the poster needs no grant.
|
||||
yt_id_match = re.search(r"i\.ytimg\.com/vi/([\w-]+)/", asset.thumbnail_url or "")
|
||||
image_dims: tuple[int, int] | None = None
|
||||
if yt_id_match:
|
||||
image_type = "image/jpeg"
|
||||
moving = _moving_image(token, asset, user_agent)
|
||||
if moving:
|
||||
image, image_dims = moving
|
||||
image_type = "image/gif"
|
||||
elif yt_id_match:
|
||||
image: str | None = f"https://i.ytimg.com/vi/{yt_id_match.group(1)}/hqdefault.jpg"
|
||||
image_dims = (480, 360)
|
||||
elif asset.poster_path:
|
||||
@@ -118,7 +154,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
f"<title>{html.escape(title)}</title>",
|
||||
_tag("og:title", title),
|
||||
_tag("og:description", channel or "Shared via Siftlode"),
|
||||
_tag("og:type", "video.other"),
|
||||
_tag("og:type", "video.other" if video else "website"),
|
||||
_tag("og:url", url),
|
||||
_tag("og:site_name", "Siftlode"),
|
||||
_tag("twitter:title", title, attr="name"),
|
||||
@@ -128,12 +164,12 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
if image:
|
||||
# A real thumbnail → a large image card; without one, a plain (text) card is served.
|
||||
tags.append(_tag("og:image", image))
|
||||
# Declare the image is a public HTTPS JPEG with known dimensions, so FB (which fetches
|
||||
# og:image asynchronously on first scrape) includes it instead of unfurling title-only. For a
|
||||
# poster we read the real dimensions from its JPEG header; the ytimg hqdefault is a fixed 480×360.
|
||||
# Declare the image's type and known dimensions, so FB (which fetches og:image
|
||||
# asynchronously on first scrape) includes it instead of unfurling title-only. For a poster
|
||||
# we read the real dimensions from its JPEG header; the ytimg hqdefault is a fixed 480×360.
|
||||
if image.startswith("https://"):
|
||||
tags.append(_tag("og:image:secure_url", image))
|
||||
tags.append(_tag("og:image:type", "image/jpeg"))
|
||||
tags.append(_tag("og:image:type", image_type))
|
||||
if image_dims is None and image.endswith("poster.jpg") and asset.poster_path:
|
||||
# safe_abs_path returns None if the poster file is missing (gc'd) or escapes the root —
|
||||
# guard it, else _jpeg_size(None) would raise an uncaught AttributeError on this public page.
|
||||
@@ -173,11 +209,13 @@ def _folder_tags(db: Session, link: DownloadLink, url: str) -> str | None:
|
||||
)
|
||||
|
||||
|
||||
def render_watch_html(token: str, page_html: Path, db: Session) -> str | None:
|
||||
def render_watch_html(
|
||||
token: str, page_html: Path, db: Session, user_agent: str | None = None
|
||||
) -> str | None:
|
||||
"""Return the watch page (watch.html) with the OG block replaced for this token, or None to
|
||||
serve the generic page unchanged (invalid/expired/not-ready link, or a template without the
|
||||
markers)."""
|
||||
tags = _watch_tags(token, db)
|
||||
markers). `user_agent` is the fetcher's, which picks a GIF's card picture."""
|
||||
tags = _watch_tags(token, db, user_agent)
|
||||
if tags is None:
|
||||
return None
|
||||
text = page_html.read_text(encoding="utf-8")
|
||||
|
||||
@@ -7,6 +7,7 @@ from datetime import date, datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import yt_dlp
|
||||
from sqlalchemy import update
|
||||
from yt_dlp.utils import download_range_func
|
||||
|
||||
from app import logs, sysconfig
|
||||
@@ -15,7 +16,9 @@ from app.db import SessionLocal
|
||||
from app.downloads import clip as clipmod
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import errors as dlerrors
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import adapters, credentials, formats, mediakind, playable, probe, quota
|
||||
from app.downloads import backfill, egress
|
||||
from app.downloads import service, storage
|
||||
from app.downloads import urlguard
|
||||
from app.downloads.jobdb import (
|
||||
@@ -392,6 +395,7 @@ def _extract_with_fallback(
|
||||
_make_progress_hook(job_id, asset_id),
|
||||
cookiefile=cookiefile,
|
||||
max_bytes=max_bytes,
|
||||
egress=adapter.egress_guarded,
|
||||
),
|
||||
)
|
||||
last_exc: Exception | None = None
|
||||
@@ -403,6 +407,7 @@ def _extract_with_fallback(
|
||||
extra=adapter.ydl_opts_extra(clients),
|
||||
cookiefile=cookiefile,
|
||||
max_bytes=max_bytes,
|
||||
egress=adapter.egress_guarded,
|
||||
)
|
||||
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
|
||||
if section is not None:
|
||||
@@ -413,7 +418,7 @@ def _extract_with_fallback(
|
||||
# (Per-ATTEMPT only; a single extract_info that itself runs past the stall timeout is still
|
||||
# killed — acceptable, as socket_timeout + extractor_retries bound one attempt well under it.)
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(opts) as ydl:
|
||||
with egress.open_ydl(opts) as ydl:
|
||||
return ydl.extract_info(target, download=True)
|
||||
except yt_dlp.utils.DownloadError as exc:
|
||||
last_exc = exc
|
||||
@@ -585,9 +590,14 @@ def _download(
|
||||
# missing duration hid the trim action. Probed before the sidecars so the .nfo runtime has it
|
||||
# too. Same probe the upload ingest uses.
|
||||
vcodec, acodec, width, height, duration_s = probe.resolve_media_facts(produced, info)
|
||||
# A placeholder title with nothing behind it ("Facebook" from a signed-in read) borrows the
|
||||
# page's text from one signed-out read; the download itself stays the signed-in one. The page
|
||||
# yt-dlp read, not the link queued: a group post's link is rewritten to its video, and
|
||||
# signed out the post itself is the login wall.
|
||||
titled = backfill.public_title_info(info, info.get("webpage_url") or url, adapter, asset_id)
|
||||
meta = storage.MediaMeta(
|
||||
video_id=info.get("id") or source_ref,
|
||||
title=normalize_title(adapter.title(info)) or source_ref,
|
||||
title=normalize_title(adapter.title(titled)) or source_ref,
|
||||
uploader=info.get("uploader") or info.get("channel") or "Unknown Channel",
|
||||
upload_date=_parse_upload_date(info.get("upload_date")),
|
||||
uniqueid_type=adapter.nfo_uniqueid_type(),
|
||||
@@ -830,6 +840,18 @@ def _fail_edit(
|
||||
)
|
||||
|
||||
|
||||
def _filter_frame(
|
||||
path: Path, width: int | None, height: int | None
|
||||
) -> tuple[int | None, int | None]:
|
||||
"""A file's frame as ffmpeg's filters see it: the stored size, turned when the stream is stored
|
||||
turned a quarter (ffmpeg stands it upright first). Unknown when the file cannot be read."""
|
||||
data = probe.probe_json(path)
|
||||
turned = clipmod.quarter_turned(data) if data else None
|
||||
if turned is None:
|
||||
return None, None
|
||||
return (height, width) if turned else (width, height)
|
||||
|
||||
|
||||
def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
"""Produce a per-user trim/crop clip from the source asset's downloaded file via ffmpeg."""
|
||||
with SessionLocal() as db:
|
||||
@@ -853,7 +875,8 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
|
||||
src_ext = src_path.suffix.lstrip(".").lower() or "mp4"
|
||||
crop = editmod.has_crop(edit_spec)
|
||||
out_ext = "mp4" if editmod.needs_reencode(edit_spec) else src_ext
|
||||
gif = edit_spec.get("gif")
|
||||
out_ext = "gif" if gif else "mp4" if editmod.needs_reencode(edit_spec) else src_ext
|
||||
total_s = editmod.clip_duration(edit_spec, src_dur) or src_dur or 0
|
||||
staging = _edit_staging_dir(asset_id)
|
||||
|
||||
@@ -864,7 +887,25 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
dest_staging = staging / f"out.{out_ext}"
|
||||
|
||||
_set_job(job_id, phase="editing", progress=0, speed_bps=None, eta_s=None)
|
||||
if edit_spec.get("segments"):
|
||||
if gif:
|
||||
# One range with an end (the service's check), cropped first if a frame was drawn, then
|
||||
# tonemapped as a clip GIF is: an HDR upload keeps its HLG/PQ tags, and a GIF has none,
|
||||
# so a palette made from those frames would be washed out.
|
||||
trim = edit_spec["trim"]
|
||||
start = trim.get("start_s") or 0.0
|
||||
data = probe.probe_json(src_path)
|
||||
cmd = gifmod.build_cmd(
|
||||
src_path,
|
||||
dest_staging,
|
||||
start,
|
||||
trim["end_s"] - start,
|
||||
pre=clipmod.gif_pre(
|
||||
editmod.crop_filter(edit_spec["crop"]) if crop else "",
|
||||
bool(data) and clipmod.is_hdr(data),
|
||||
),
|
||||
side=gif["side"],
|
||||
)
|
||||
elif edit_spec.get("segments"):
|
||||
cmd, prep = editmod.build_concat_plan(
|
||||
src_path, dest_staging, edit_spec, out_ext, staging
|
||||
)
|
||||
@@ -880,9 +921,12 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
)
|
||||
if not dest_staging.exists():
|
||||
raise StatedFailure("ffmpeg produced no output file")
|
||||
# A fast trim stream-copies the source's codecs and container, so the clip is only as
|
||||
# playable as its source: hold it to the same rule as every other ready file.
|
||||
dest_staging = _ensure_phone_safe(job_id, dest_staging, staging, "editing", total_s)
|
||||
if gif:
|
||||
_hold_gif_cap(dest_staging)
|
||||
else:
|
||||
# A fast trim stream-copies the source's codecs and container, so the clip is only as
|
||||
# playable as its source: hold it to the same rule as every other ready file.
|
||||
dest_staging = _ensure_phone_safe(job_id, dest_staging, staging, "editing", total_s)
|
||||
out_ext = dest_staging.suffix.lstrip(".").lower() or out_ext
|
||||
# New storage on the owner's footprint, held to what is left by its real size, as a
|
||||
# download and a player clip are.
|
||||
@@ -896,8 +940,18 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
storage.place_file(dest_staging, settings.files_root, rel)
|
||||
final = storage.abs_path(settings.files_root, rel)
|
||||
size = final.stat().st_size if final.exists() else None
|
||||
# The clip's real frame: a re-encode evens an odd crop or source (`editmod.EVEN_FRAME`).
|
||||
# The clip's real frame: a re-encode evens an odd crop or source (`editmod.EVEN_FRAME`). A
|
||||
# GIF's is in its header (ffprobe reports no stream size for an image format).
|
||||
_, _, out_w, out_h, _ = probe.probe_media(final)
|
||||
if gif:
|
||||
out_w, out_h = gifmod.size(final) or (None, None)
|
||||
poster = _gif_still(rel) if gif else None
|
||||
# Unsized, a crop's own size stands in, cut from the frame the filters saw.
|
||||
cut = (
|
||||
editmod.crop_stored_size(edit_spec["crop"], *_filter_frame(src_path, src_w, src_h))
|
||||
if crop and not (out_w and out_h)
|
||||
else None
|
||||
)
|
||||
|
||||
with SessionLocal() as db:
|
||||
asset = db.get(MediaAsset, asset_id)
|
||||
@@ -909,11 +963,12 @@ def _process_edit(job_id: int, asset_id: int) -> None:
|
||||
asset.duration_s = total_s or asset.duration_s
|
||||
if out_w and out_h:
|
||||
asset.width, asset.height = out_w, out_h
|
||||
elif crop:
|
||||
asset.width = edit_spec["crop"]["w"]
|
||||
asset.height = edit_spec["crop"]["h"]
|
||||
elif cut:
|
||||
asset.width, asset.height = cut
|
||||
else:
|
||||
asset.width, asset.height = src_w, src_h
|
||||
asset.width, asset.height = (None, None) if crop else (src_w, src_h)
|
||||
if poster:
|
||||
asset.poster_path = poster
|
||||
asset.error = None
|
||||
asset.last_access_at = datetime.now(timezone.utc)
|
||||
job = db.get(DownloadJob, job_id)
|
||||
@@ -1031,6 +1086,19 @@ def _clip_crop(spec: dict, data: dict) -> str:
|
||||
return crop
|
||||
|
||||
|
||||
def _hold_gif_cap(out: Path) -> None:
|
||||
"""A GIF over its own size cap is not kept (`gif.MAX_BYTES`): a stated failure that names the
|
||||
range and the size, the two things that make it smaller."""
|
||||
if out.stat().st_size > gifmod.MAX_BYTES:
|
||||
raise StatedFailure(dlerrors.GIF_TOO_BIG_MESSAGE)
|
||||
|
||||
|
||||
def _gif_still(rel: str) -> str | None:
|
||||
"""The placed GIF's still (`gif.ensure_still`), as the poster's rel path beside it."""
|
||||
still = gifmod.ensure_still(storage.abs_path(settings.files_root, rel))
|
||||
return str(Path(rel).with_suffix(".jpg")) if still else None
|
||||
|
||||
|
||||
def _process_clip(job_id: int, asset_id: int) -> None:
|
||||
"""Cut an ad-hoc clip (downloads/clip): fetch or open its source, ONE re-encode to the clip
|
||||
shape, then into the user's library beside their edits."""
|
||||
@@ -1053,34 +1121,49 @@ def _process_clip(job_id: int, asset_id: int) -> None:
|
||||
raise StatedFailure("The source file could not be read.")
|
||||
audio_track = spec.get("audio_track")
|
||||
tracks = clipmod.audio_track_count(data)
|
||||
if tracks == 0:
|
||||
gif = spec.get("gif")
|
||||
if gif:
|
||||
# A GIF has no sound: no track is chosen, so none can be missing.
|
||||
audio_track = None
|
||||
elif tracks == 0:
|
||||
# A silent source: the player still names its track 0, and the clip is silent too.
|
||||
audio_track = None
|
||||
elif audio_track is not None and audio_track >= tracks:
|
||||
raise StatedFailure("The chosen audio track is not in the file.")
|
||||
crop = _clip_crop(spec, data)
|
||||
total_s = clipmod.duration(spec)
|
||||
out = staging / "clip.mp4"
|
||||
hdr = clipmod.is_hdr(data)
|
||||
out = staging / ("clip.gif" if gif else "clip.mp4")
|
||||
_set_job(job_id, phase="editing", progress=0, speed_bps=None, eta_s=None)
|
||||
editmod.run_ffmpeg(
|
||||
clipmod.build_cmd(
|
||||
cmd = (
|
||||
gifmod.build_cmd(
|
||||
src, out, start, total_s, pre=clipmod.gif_pre(crop, hdr), side=gif["side"]
|
||||
)
|
||||
if gif
|
||||
else clipmod.build_cmd(
|
||||
src,
|
||||
out,
|
||||
start,
|
||||
total_s,
|
||||
audio_track=audio_track,
|
||||
hdr=clipmod.is_hdr(data),
|
||||
hdr=hdr,
|
||||
threads=editmod.encode_thread_cap(),
|
||||
crop=crop,
|
||||
),
|
||||
)
|
||||
)
|
||||
editmod.run_ffmpeg(
|
||||
cmd,
|
||||
total_s,
|
||||
on_progress=_ffmpeg_progress_writer(job_id, "editing"),
|
||||
should_cancel=_cancel_poll(job_id),
|
||||
)
|
||||
if not out.exists():
|
||||
raise StatedFailure("ffmpeg produced no output file")
|
||||
# The phone-safe shape by construction; still held to the rule every ready file meets.
|
||||
out = _ensure_phone_safe(job_id, out, staging, "editing", round(total_s))
|
||||
if gif:
|
||||
_hold_gif_cap(out)
|
||||
else:
|
||||
# The phone-safe shape by construction; still held to the rule every ready file meets.
|
||||
out = _ensure_phone_safe(job_id, out, staging, "editing", round(total_s))
|
||||
# Held to the cap by its real size: a range fetch passes yt-dlp's `max_filesize` to its
|
||||
# ffmpeg downloader, which never reads it, and a Plex clip fetches nothing at all. Read
|
||||
# again, since other jobs of this owner may have landed during the encode.
|
||||
@@ -1092,6 +1175,8 @@ def _process_clip(job_id: int, asset_id: int) -> None:
|
||||
storage.place_file(out, settings.files_root, rel)
|
||||
final = storage.abs_path(settings.files_root, rel)
|
||||
vcodec, acodec, width, height, _ = probe.probe_media(final)
|
||||
if gif:
|
||||
width, height = gifmod.size(final) or (None, None)
|
||||
|
||||
with SessionLocal() as db:
|
||||
asset = db.get(MediaAsset, asset_id)
|
||||
@@ -1109,21 +1194,29 @@ def _process_clip(job_id: int, asset_id: int) -> None:
|
||||
asset.duration_s = round(total_s)
|
||||
asset.media_kind = mediakind.derive(vcodec, acodec)
|
||||
# The clip's own frame as its poster: the link preview shows the moment, and a Plex
|
||||
# item has no public thumbnail at all.
|
||||
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
|
||||
# item has no public thumbnail at all. A GIF's is its first frame, cut from the GIF.
|
||||
asset.poster_path = (
|
||||
_gif_still(rel) if gif else editmod.ensure_poster(asset, settings.files_root)
|
||||
)
|
||||
asset.error = None
|
||||
asset.last_access_at = datetime.now(timezone.utc)
|
||||
fields: dict = {"status": JobStatus.DONE, "progress": 100, "phase": None}
|
||||
# A video outside the catalog was queued as "Clip (clip)": the fetch has its title now.
|
||||
placeholder = service.clip_title(None, None)
|
||||
named = None
|
||||
if asset and fetched_title and asset.title == placeholder:
|
||||
named = service.clip_title(fetched_title, None)
|
||||
asset.title = named
|
||||
if job_row := db.get(DownloadJob, job_id):
|
||||
if job_row.display_name == placeholder:
|
||||
fields["display_name"] = named[:DISPLAY_TEXT_MAX]
|
||||
title = asset.title if asset else None
|
||||
finalized = service.finalize_if_running(db, job_id, **fields)
|
||||
if finalized and named:
|
||||
# Only a name that still repeats the placeholder, in one conditional write: one its
|
||||
# owner gave meanwhile stays.
|
||||
db.execute(
|
||||
update(DownloadJob)
|
||||
.where(DownloadJob.id == job_id, DownloadJob.display_name == placeholder)
|
||||
.values(display_name=named[:DISPLAY_TEXT_MAX])
|
||||
)
|
||||
db.commit()
|
||||
if finalized:
|
||||
_notify_download_done(db, user_id, title, job_id, kind="edit")
|
||||
|
||||
@@ -45,6 +45,16 @@ TEXT_EXTS: frozenset[str] = frozenset(
|
||||
|
||||
TEXT_TYPE = "text/plain; charset=utf-8"
|
||||
|
||||
# Every preview served inline — the library's viewer and a share link's picture — goes with these.
|
||||
# `sandbox` leaves the answer no script and an opaque origin, whatever the file turns out to hold (a
|
||||
# "gif" that is really a page); `frame-ancestors 'self'` lets the app's viewer frame a PDF (every
|
||||
# other answer keeps the app-wide 'none', main.py), with X-Frame-Options saying the same for old
|
||||
# engines.
|
||||
SANDBOX_HEADERS: dict[str, str] = {
|
||||
"Content-Security-Policy": "sandbox; frame-ancestors 'self'",
|
||||
"X-Frame-Options": "SAMEORIGIN",
|
||||
}
|
||||
|
||||
# A text preview is the file's head: enough to read, never a whole log loaded into a tab.
|
||||
TEXT_PREVIEW_BYTES = 1 << 20
|
||||
# A NUL byte in the first this-many bytes means the "text" is not text.
|
||||
|
||||
@@ -16,10 +16,12 @@ That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user
|
||||
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
|
||||
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
|
||||
is unknown until it finishes. CLIP jobs consult it before their fetch or encode and again with
|
||||
their output's real size before it is placed. Edit jobs do not consult it at all (tracked as SEC-19).
|
||||
their output's real size before it is placed. Edit jobs consult it with their output's real size
|
||||
before it is placed (`test_edit_cap.py`).
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
|
||||
from sqlalchemy import and_, func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -50,11 +52,16 @@ class QuotaLimits:
|
||||
unlimited: bool
|
||||
|
||||
|
||||
class QuotaExceeded(Exception):
|
||||
"""Raised by check_enqueue; `reason` is one of max_jobs|max_bytes (i18n key on the client)."""
|
||||
# Every cap a download can hit; the UI names each one (`downloads.errors.quota.<reason>`, held by the
|
||||
# frontend's backendIds contract, which reads this Literal).
|
||||
QuotaReason = Literal["max_jobs", "max_bytes"]
|
||||
|
||||
def __init__(self, reason: str, limit: int, current: int):
|
||||
self.reason = reason
|
||||
|
||||
class QuotaExceeded(Exception):
|
||||
"""Raised by check_enqueue; `reason` names the cap it hit (an i18n key on the client)."""
|
||||
|
||||
def __init__(self, reason: QuotaReason, limit: int, current: int):
|
||||
self.reason: QuotaReason = reason
|
||||
self.limit = limit
|
||||
self.current = current
|
||||
super().__init__(f"download quota exceeded: {reason} ({current}/{limit})")
|
||||
|
||||
@@ -10,6 +10,7 @@ handler); the folder, link and quota writes are `folders`, `links` and `quota`.
|
||||
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
|
||||
from sqlalchemy import delete as delete_rows
|
||||
from sqlalchemy import func, select, update
|
||||
@@ -22,6 +23,7 @@ from app.downloads import adapters
|
||||
from app.downloads import clip as clipmod
|
||||
from app.downloads import credentials
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import folders, formats, mediakind, quota, storage
|
||||
from app.downloads.status import HOLDING, JobKind, JobStatus
|
||||
from app.models import DISPLAY_TEXT_MAX, User
|
||||
@@ -39,11 +41,26 @@ from app.plex import paths as plex_paths
|
||||
from app.userscope import owned_by, owned_row
|
||||
|
||||
|
||||
class EditError(Exception):
|
||||
"""Bad edit request (empty spec / source not ready); `reason` is an i18n-mappable key."""
|
||||
# Every reason an edit is refused; the UI names each one (`downloads.errors.edit.<reason>`, held by
|
||||
# the frontend's backendIds contract, which reads this Literal).
|
||||
EditReason = Literal[
|
||||
"source_not_ready",
|
||||
"not_media",
|
||||
"empty_edit",
|
||||
"bad_gif",
|
||||
"gif_range",
|
||||
"gif_too_long",
|
||||
"gif_not_video",
|
||||
]
|
||||
|
||||
def __init__(self, reason: str):
|
||||
self.reason = reason
|
||||
|
||||
class EditError(Exception):
|
||||
"""Bad edit request (empty spec / source not ready / not media / a GIF it cannot make); `reason`
|
||||
names the sentence, `params` the numbers it names (a cap travels with its refusal)."""
|
||||
|
||||
def __init__(self, reason: EditReason, **params: float):
|
||||
self.reason: EditReason = reason
|
||||
self.params = params
|
||||
super().__init__(reason)
|
||||
|
||||
|
||||
@@ -147,7 +164,7 @@ def populate_from_catalog(db: Session, asset: MediaAsset) -> bool:
|
||||
return False
|
||||
asset.title = video.title
|
||||
asset.thumbnail_url = video.thumbnail_url
|
||||
asset.duration_s = video.duration_seconds
|
||||
asset.duration_s = video.length_seconds
|
||||
if video.published_at:
|
||||
asset.upload_date = video.published_at.date()
|
||||
channel = db.get(Channel, video.channel_id)
|
||||
@@ -249,6 +266,18 @@ def clip_title(source_title: str | None, display_name: str | None) -> str:
|
||||
return f"{base} (clip)"[:DISPLAY_TEXT_MAX]
|
||||
|
||||
|
||||
def _check_gif_edit(spec: dict) -> None:
|
||||
"""A GIF edit is one marked range with an end, at most `gif.MAX_S` long, of a size `gif` offers
|
||||
(a size it does not offer was dropped by `normalize_edit_spec`). Raises EditError."""
|
||||
if not spec.get("gif"):
|
||||
raise EditError("bad_gif")
|
||||
trim = spec.get("trim") or {}
|
||||
if spec.get("segments") or trim.get("end_s") is None:
|
||||
raise EditError("gif_range")
|
||||
if gifmod.too_long(trim.get("start_s", 0.0), trim["end_s"]):
|
||||
raise EditError("gif_too_long", limit_s=gifmod.MAX_S)
|
||||
|
||||
|
||||
def enqueue_edit(
|
||||
db: Session,
|
||||
user_id: int,
|
||||
@@ -274,6 +303,11 @@ def enqueue_edit(
|
||||
spec = editmod.normalize_edit_spec(edit_spec)
|
||||
if not spec:
|
||||
raise EditError("empty_edit")
|
||||
if "gif" in (edit_spec or {}):
|
||||
# A GIF is made of the first video stream: audio has none, or only its cover picture.
|
||||
if src_asset.media_kind != mediakind.VIDEO:
|
||||
raise EditError("gif_not_video")
|
||||
_check_gif_edit(spec)
|
||||
|
||||
quota.check_enqueue(db, user_id)
|
||||
|
||||
@@ -288,7 +322,10 @@ def enqueue_edit(
|
||||
asset.duration_s = editmod.clip_duration(spec, src_asset.duration_s)
|
||||
# A clip inherits its source's kind (a video edit stays video, an audio trim stays audio);
|
||||
# the ffmpeg edit worker doesn't re-probe codecs, so set it here where the source is known.
|
||||
asset.media_kind = src_asset.media_kind or mediakind.VIDEO
|
||||
# A GIF is a picture whatever it was cut from (downloads/gif).
|
||||
asset.media_kind = (
|
||||
mediakind.FILE if spec.get("gif") else src_asset.media_kind or mediakind.VIDEO
|
||||
)
|
||||
|
||||
job = DownloadJob(
|
||||
user_id=user_id,
|
||||
@@ -331,7 +368,7 @@ def _clip_source_facts(db: Session, spec: dict) -> dict:
|
||||
"title": video.title,
|
||||
"uploader": channel.title if channel else None,
|
||||
"thumbnail_url": video.thumbnail_url,
|
||||
"length_s": video.duration_seconds,
|
||||
"length_s": video.length_seconds,
|
||||
}
|
||||
item = plex_lookups.by_rating_key(db, PlexItem, spec["ref"])
|
||||
if item is None:
|
||||
@@ -378,7 +415,8 @@ def enqueue_clip(
|
||||
asset.uploader = facts["uploader"]
|
||||
asset.thumbnail_url = facts["thumbnail_url"]
|
||||
asset.duration_s = round(clipmod.duration(spec))
|
||||
asset.media_kind = mediakind.VIDEO
|
||||
# A GIF is a picture: no player and no editor, an image preview (downloads/gif).
|
||||
asset.media_kind = mediakind.FILE if spec.get("gif") else mediakind.VIDEO
|
||||
|
||||
job = DownloadJob(
|
||||
user_id=user_id,
|
||||
|
||||
@@ -10,22 +10,12 @@ What this closes: a URL whose host resolves to a loopback / private / link-local
|
||||
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
|
||||
differently between the two moments, and the worker is where the socket is actually opened.
|
||||
|
||||
What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
|
||||
the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
|
||||
|
||||
* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
|
||||
with an internal address.
|
||||
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
|
||||
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
|
||||
open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
|
||||
response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
|
||||
the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
|
||||
class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
|
||||
one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
|
||||
kept). Tracked as SEC-17.
|
||||
|
||||
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
|
||||
hostname into the box), which is the reachable one.
|
||||
What this alone cannot see is every request yt-dlp makes AFTER the typed URL: a public host
|
||||
answering 3xx with an internal ``Location``, and DNS rebinding (a name that passes this check and
|
||||
then resolves inside for yt-dlp's own lookup). Those are closed one layer down, by `egress`: a guarded
|
||||
download's every connection goes through a gate that judges the address it actually dials, with
|
||||
this module's `is_forbidden`. The checks here stay as the early, structured refusal — a user typing
|
||||
``http://127.0.0.1:6379`` hears why at once instead of from a failed job.
|
||||
|
||||
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
|
||||
user-controlled destination to guard.
|
||||
@@ -33,27 +23,36 @@ user-controlled destination to guard.
|
||||
|
||||
import ipaddress
|
||||
import socket
|
||||
from typing import Literal
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# Reason codes travel to the client inside the structured 422 detail, so they are part of the API
|
||||
# vocabulary (frontend `localizeDetail`) — keep them stable.
|
||||
REASON_SCHEME = "scheme" # not http(s)
|
||||
REASON_NO_HOST = "no_host" # URL carries no hostname
|
||||
REASON_MALFORMED = "malformed" # unparseable authority or port; urlparse raises late, so guard it
|
||||
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
|
||||
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
|
||||
# vocabulary — keep them stable. The UI names each one (`downloads.errors.urlBlocked.<reason>`, held by
|
||||
# the frontend's backendIds contract, which reads this Literal).
|
||||
UrlBlockedReason = Literal["scheme", "no_host", "malformed", "unresolvable", "private"]
|
||||
REASON_SCHEME: UrlBlockedReason = "scheme" # not http(s)
|
||||
REASON_NO_HOST: UrlBlockedReason = "no_host" # URL carries no hostname
|
||||
REASON_MALFORMED: UrlBlockedReason = (
|
||||
"malformed" # unparseable authority or port; urlparse raises late
|
||||
)
|
||||
REASON_UNRESOLVABLE: UrlBlockedReason = (
|
||||
"unresolvable" # DNS said nothing (not the same as "blocked")
|
||||
)
|
||||
REASON_PRIVATE: UrlBlockedReason = (
|
||||
"private" # resolves into a network the server must not fetch from
|
||||
)
|
||||
|
||||
|
||||
class BlockedUrl(Exception):
|
||||
"""A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
|
||||
REASON_* codes above."""
|
||||
|
||||
def __init__(self, reason: str):
|
||||
def __init__(self, reason: UrlBlockedReason):
|
||||
super().__init__(reason)
|
||||
self.reason = reason
|
||||
self.reason: UrlBlockedReason = reason
|
||||
|
||||
|
||||
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
def is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
"""Every address family we must not open a connection to. Checked explicitly rather than
|
||||
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
|
||||
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
|
||||
@@ -104,5 +103,5 @@ def check_url(raw: str) -> None:
|
||||
ip = ipaddress.ip_address(addr)
|
||||
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
|
||||
raise BlockedUrl(REASON_PRIVATE) from None
|
||||
if _is_forbidden(ip):
|
||||
if is_forbidden(ip):
|
||||
raise BlockedUrl(REASON_PRIVATE)
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
"""A weighted draw that stays a surprise: what "Surprise me" deals from.
|
||||
|
||||
A title's chance to come first is its weight over the pool's (Efraimidis–Spirakis: order by
|
||||
-ln(1-u)/w, u uniform per row). The weight is 1 + the title's affinity to what the user watches
|
||||
(`plex.affinity`, `discover.queries`), so a title nobody leans towards still comes up — only less
|
||||
often. Two things keep it a surprise rather than a ranking:
|
||||
|
||||
- **u is the seed's md5 of the row's key**, the Home surprise's seeded order. With every weight 1
|
||||
the order is exactly that md5 order, so a user with no history sees the unweighted draw, and one
|
||||
seed still moves both sources of the Home hero together. A draw that is not seeded (the Plex
|
||||
Discover rail) takes u from random().
|
||||
- **One slot of three is blind** (`blend`): dealt from the unweighted order, so the same few
|
||||
favourites cannot fill the rail.
|
||||
|
||||
The YouTube surprise draws over a window of the feed scope rather than all of it (`yt_surprise`):
|
||||
the window is read in SQL, and `shuffled` / `leaning` order it here with the same u.
|
||||
|
||||
Both affinities weigh a watch by `recency` and score against the strongest with `scaled`, so the two
|
||||
sources lean by one rule.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import math
|
||||
from collections.abc import Callable, Hashable
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import BigInteger, Float, cast, func, literal
|
||||
from sqlalchemy.dialects.postgresql import BIT
|
||||
from sqlalchemy.orm import InstrumentedAttribute
|
||||
from sqlalchemy.sql.elements import ColumnElement
|
||||
|
||||
# 2**32: the first 8 hex digits of an md5 as a fraction of one.
|
||||
_SPAN = 4294967296.0
|
||||
|
||||
# A watch with no date (an old import) counts as one this many half-lives ago.
|
||||
UNDATED_HALF_LIVES = 2
|
||||
|
||||
|
||||
def recency(watched_at: datetime | None, now: datetime, half_life_days: float) -> float:
|
||||
"""A watch's weight in an affinity: 1 now, half as much every `half_life_days`."""
|
||||
if watched_at is None:
|
||||
return 0.5**UNDATED_HALF_LIVES
|
||||
return 0.5 ** (max(0.0, (now - watched_at).total_seconds() / 86400) / half_life_days)
|
||||
|
||||
|
||||
def scaled(scores: dict[str, float]) -> dict[str, float]:
|
||||
"""Each score over the strongest, so the strongest is 1; empty when nothing scored."""
|
||||
top = max(scores.values(), default=0.0)
|
||||
return {k: v / top for k, v in scores.items()} if top else {}
|
||||
|
||||
|
||||
def seeded_u(
|
||||
key: ColumnElement[Any] | InstrumentedAttribute[Any], seed: int
|
||||
) -> ColumnElement[float]:
|
||||
"""A uniform u in [0, 1) per row, from md5(key || seed) — the order `md5(...)` ascending gives."""
|
||||
digest = func.md5(func.concat(key, str(seed)))
|
||||
head = cast(cast(literal("x") + func.substr(digest, 1, 8), BIT(32)), BigInteger)
|
||||
return cast(head, Float) / _SPAN
|
||||
|
||||
|
||||
def weighted_key(u: ColumnElement[float], weight: ColumnElement[float]) -> ColumnElement[float]:
|
||||
"""The row's place in a weighted draw, ascending: -ln(1-u)/w. Monotonic in u when every w is 1,
|
||||
so the unweighted order is unchanged."""
|
||||
return -func.ln(1.0 - u) / weight
|
||||
|
||||
|
||||
def random_u() -> ColumnElement[float]:
|
||||
return func.random()
|
||||
|
||||
|
||||
def u_of(key: str, seed: int) -> float:
|
||||
"""`seeded_u` in Python: the same u for the same key and seed."""
|
||||
return int(hashlib.md5(f"{key}{seed}".encode()).hexdigest()[:8], 16) / _SPAN
|
||||
|
||||
|
||||
def shuffled(items: list, seed: int, key: Callable[[Any], str]) -> list:
|
||||
"""`items` in the seed's md5 order — the order `ORDER BY md5(key || seed)` gives."""
|
||||
return sorted(items, key=lambda it: hashlib.md5(f"{key(it)}{seed}".encode()).hexdigest())
|
||||
|
||||
|
||||
def leaning(
|
||||
items: list, seed: int, key: Callable[[Any], str], weight: Callable[[Any], float]
|
||||
) -> list:
|
||||
"""`items` in the weighted draw's order (`weighted_key` in Python), ties in the md5 order."""
|
||||
order = {key(it): i for i, it in enumerate(shuffled(items, seed, key))}
|
||||
return sorted(
|
||||
items,
|
||||
key=lambda it: (-math.log(1.0 - u_of(key(it), seed)) / weight(it), order[key(it)]),
|
||||
)
|
||||
|
||||
|
||||
def blend(
|
||||
weighted: list, blind: list, offset: int, n: int, key: Callable[[Any], Hashable] = lambda x: x
|
||||
) -> list:
|
||||
"""`n` picks: slot i comes from `blind` when (i + offset) % 3 == 0, else from `weighted`, each
|
||||
skipping what an earlier slot took (`key` tells two picks apart). A side that runs out leaves the
|
||||
rest to the other."""
|
||||
out: list = []
|
||||
taken: set = set()
|
||||
sources = (iter(weighted), iter(blind))
|
||||
|
||||
def nxt(source):
|
||||
for item in source:
|
||||
if key(item) not in taken:
|
||||
return item
|
||||
return None
|
||||
|
||||
for i in range(n):
|
||||
first, second = (sources[1], sources[0]) if (i + offset) % 3 == 0 else sources
|
||||
item = nxt(first)
|
||||
if item is None:
|
||||
item = nxt(second)
|
||||
if item is None:
|
||||
break
|
||||
taken.add(key(item))
|
||||
out.append(item)
|
||||
return out
|
||||
+71
-12
@@ -13,7 +13,19 @@ import binascii
|
||||
import json
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import BigInteger, Select, and_, case, cast, false, func, or_, select
|
||||
from sqlalchemy import (
|
||||
BigInteger,
|
||||
Select,
|
||||
and_,
|
||||
case,
|
||||
cast,
|
||||
false,
|
||||
func,
|
||||
literal,
|
||||
or_,
|
||||
select,
|
||||
tuple_,
|
||||
)
|
||||
from sqlalchemy.orm import Session, aliased
|
||||
|
||||
from app.collation import collated
|
||||
@@ -68,7 +80,7 @@ def feed_columns(user: User, state) -> list:
|
||||
Channel.handle.label("channel_handle"),
|
||||
Video.published_at,
|
||||
Video.thumbnail_url,
|
||||
Video.duration_seconds,
|
||||
Video.length_seconds.label("length_seconds"),
|
||||
Video.view_count,
|
||||
Video.is_short,
|
||||
Video.live_status,
|
||||
@@ -88,7 +100,7 @@ def serialize(row) -> dict:
|
||||
"channel_url": _channel_url(row.channel_id, row.channel_handle),
|
||||
"published_at": iso(row.published_at),
|
||||
"thumbnail_url": row.thumbnail_url,
|
||||
"duration_seconds": row.duration_seconds,
|
||||
"duration_seconds": row.length_seconds,
|
||||
"view_count": row.view_count,
|
||||
"is_short": row.is_short,
|
||||
"live_status": row.live_status,
|
||||
@@ -260,9 +272,9 @@ def filtered_query(
|
||||
# OR across the picked channels; every other filter still applies on top.
|
||||
query = query.where(Video.channel_id.in_(channel_ids))
|
||||
if min_duration is not None:
|
||||
query = query.where(Video.duration_seconds >= min_duration)
|
||||
query = query.where(Video.length_seconds >= min_duration)
|
||||
if max_duration is not None:
|
||||
query = query.where(Video.duration_seconds <= max_duration)
|
||||
query = query.where(Video.length_seconds <= max_duration)
|
||||
if max_age_days is not None:
|
||||
cutoff = datetime.now(timezone.utc).timestamp() - max_age_days * 86400
|
||||
query = query.where(Video.published_at >= datetime.fromtimestamp(cutoff, tz=timezone.utc))
|
||||
@@ -503,10 +515,10 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
|
||||
"views": [{"expr": Video.view_count, "asc": False, "nullable": True, "kind": "num"}],
|
||||
"views_asc": [{"expr": Video.view_count, "asc": True, "nullable": True, "kind": "num"}],
|
||||
"duration_desc": [
|
||||
{"expr": Video.duration_seconds, "asc": False, "nullable": True, "kind": "num"}
|
||||
{"expr": Video.length_seconds, "asc": False, "nullable": True, "kind": "num"}
|
||||
],
|
||||
"duration_asc": [
|
||||
{"expr": Video.duration_seconds, "asc": True, "nullable": True, "kind": "num"}
|
||||
{"expr": Video.length_seconds, "asc": True, "nullable": True, "kind": "num"}
|
||||
],
|
||||
"title": [{"expr": title, "asc": True, "nullable": True, "kind": "str"}],
|
||||
"title_desc": [{"expr": title, "asc": False, "nullable": True, "kind": "str"}],
|
||||
@@ -536,15 +548,17 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
|
||||
},
|
||||
pub,
|
||||
],
|
||||
# Deterministic per-seed shuffle: the same seed yields the same total order, so
|
||||
# the cursor stays valid across pages of one shuffled run.
|
||||
# The ring of stored random keys from the seed's point (`shuffle_walk`): the keys at or after
|
||||
# it, then the ones before. The same seed yields the same total order, so the cursor stays
|
||||
# valid across pages; `feed_page` reads it as two index ranges, never as this sort.
|
||||
"shuffle": [
|
||||
{
|
||||
"expr": func.md5(func.concat(Video.id, str(seed))),
|
||||
"expr": case((Video.shuffle_key < shuffle_start(seed), 1), else_=0),
|
||||
"asc": True,
|
||||
"nullable": False,
|
||||
"kind": "str",
|
||||
}
|
||||
"kind": "num",
|
||||
},
|
||||
{"expr": Video.shuffle_key, "asc": True, "nullable": False, "kind": "num"},
|
||||
],
|
||||
}
|
||||
# Relevance (search-result ranking) only exists when there's a query to rank against;
|
||||
@@ -700,6 +714,43 @@ class BadCursor(ValueError):
|
||||
"""A `cursor` this module did not write (or one from another sort): the caller's bad input."""
|
||||
|
||||
|
||||
# The golden ratio's fraction: consecutive seeds land far apart on the ring, never next to each other.
|
||||
_GOLDEN = 0.6180339887498949
|
||||
|
||||
|
||||
def shuffle_start(seed: int) -> float:
|
||||
"""The point in [0, 1) where a seed's shuffle starts on the ring of `Video.shuffle_key`.
|
||||
|
||||
Folded to 32 bits first: an int past a float's range overflowed the product, and one past 2**53
|
||||
left the product no fraction to land on. A seed the client draws (under a billion) is unchanged."""
|
||||
return ((seed % 2**32) * _GOLDEN) % 1.0
|
||||
|
||||
|
||||
def shuffle_walk(
|
||||
db: Session, query: Select, seed: int, n: int, after: tuple[int, float, str] | None = None
|
||||
) -> list:
|
||||
"""Up to `n` rows of `query` in the seed's ring order: shuffle_key from the seed's point upwards,
|
||||
then from 0 up to that point. `after` is a cursor's (phase, key, id): the walk resumes past it.
|
||||
|
||||
Each arc is an ascending range read along ix_videos_shuffle_key with the scope's filters checked
|
||||
row by row, so a draw costs the rows it reads, not a sort of the scope — the md5(id || seed) order
|
||||
it replaced read and sorted every row the filters let through (~350 ms on prod for ~213k)."""
|
||||
start = shuffle_start(seed)
|
||||
phase_from, key, last_id = after if after is not None else (0, None, None)
|
||||
rows: list = []
|
||||
for phase in (0, 1):
|
||||
if phase < phase_from or len(rows) >= n:
|
||||
continue
|
||||
arc = Video.shuffle_key >= start if phase == 0 else Video.shuffle_key < start
|
||||
q = query.where(arc)
|
||||
if key is not None and phase == phase_from:
|
||||
q = q.where(
|
||||
tuple_(Video.shuffle_key, Video.id) > tuple_(literal(key), literal(last_id))
|
||||
)
|
||||
rows += db.execute(q.order_by(Video.shuffle_key, Video.id).limit(n - len(rows))).all()
|
||||
return rows
|
||||
|
||||
|
||||
def feed_page(
|
||||
db: Session,
|
||||
user: User,
|
||||
@@ -734,6 +785,14 @@ def feed_page(
|
||||
k["value"] = v
|
||||
if ranked and window is not None:
|
||||
rows = windowed_relevance_rows(db, query, keys, window, cursor_id, limit + 1)
|
||||
elif sort == "shuffle":
|
||||
after = None
|
||||
if vals is not None and cursor_id is not None:
|
||||
try:
|
||||
after = (int(vals[0]), float(vals[1]), str(cursor_id))
|
||||
except (TypeError, ValueError, OverflowError) as e: # Infinity, a 400-digit key
|
||||
raise BadCursor("Invalid feed cursor") from e
|
||||
rows = shuffle_walk(db, query, seed, limit + 1, after)
|
||||
else:
|
||||
query = query.order_by(*order_by(keys))
|
||||
if cursor_id is not None:
|
||||
|
||||
+15
-8
@@ -151,7 +151,7 @@ async def _reap_hls_sessions() -> None:
|
||||
somebody pressed play again or the process restarted.
|
||||
|
||||
A FINISHED session is reaped on the short rule here, not kept for the full idle window: an open
|
||||
player checkpoints every 10 s and that route touches the session, so a paused viewer is no longer
|
||||
player keeps checkpointing and that route touches the session, so a paused viewer is no longer
|
||||
invisible to this pass (see `reap_idle`). Keeping every finished session for ten minutes instead
|
||||
held one whole segmented copy of a film per item on the download volume."""
|
||||
while True:
|
||||
@@ -368,6 +368,10 @@ INDEX_HTML = STATIC_DIR / "index.html"
|
||||
# A public share link's own page: the player alone, without the app's bundle (frontend/watch.html).
|
||||
WATCH_HTML = STATIC_DIR / "watch.html"
|
||||
INDEX_HEADERS = {"Cache-Control": "no-cache"}
|
||||
# The paths a share link opens on, by what it opens (`links.share_path`). Any of them opens any link.
|
||||
# A new one also goes in frontend/public/robots.txt, the Vite dev entry and WatchPage's token parsing
|
||||
# (watchEntry.contract.test.ts).
|
||||
SHARE_PREFIXES = ("watch/", "file/", "folder/")
|
||||
app.mount(
|
||||
"/assets",
|
||||
StaticFiles(directory=STATIC_DIR / "assets", check_dir=False),
|
||||
@@ -381,18 +385,21 @@ async def index() -> FileResponse:
|
||||
|
||||
|
||||
@app.get("/watch/{token}")
|
||||
async def watch_page(token: str):
|
||||
"""Serve the public player page for a share link, with per-video Open Graph tags injected so the
|
||||
@app.get("/file/{token}")
|
||||
@app.get("/folder/{token}")
|
||||
async def share_page(token: str, request: Request):
|
||||
"""Serve the public page of a share link, with per-link Open Graph tags injected so the
|
||||
link unfurls richly (title/channel/thumbnail) in Messenger/social crawlers. A real browser
|
||||
ignores the extra tags and boots WatchPage; invalid/expired/password links fall back to the
|
||||
generic card."""
|
||||
generic card. A GIF's card picture depends on which crawler asks (`og._moving_image`), so the
|
||||
answer varies by User-Agent."""
|
||||
from app.downloads import og
|
||||
|
||||
with SessionLocal() as db:
|
||||
rendered = og.render_watch_html(token, WATCH_HTML, db)
|
||||
rendered = og.render_watch_html(token, WATCH_HTML, db, request.headers.get("user-agent"))
|
||||
if rendered is None:
|
||||
return FileResponse(WATCH_HTML, headers=INDEX_HEADERS)
|
||||
return HTMLResponse(rendered, headers=INDEX_HEADERS)
|
||||
return HTMLResponse(rendered, headers={**INDEX_HEADERS, "Vary": "User-Agent"})
|
||||
|
||||
|
||||
@app.get("/{full_path:path}")
|
||||
@@ -409,8 +416,8 @@ async def spa_fallback(full_path: str) -> FileResponse:
|
||||
# Real SPA-root assets (welcome images, favicon, robots.txt) rarely change and have
|
||||
# stable names, so a moderate cache is safe and satisfies the cache-lifetime audit.
|
||||
return FileResponse(candidate, headers={"Cache-Control": "public, max-age=604800"})
|
||||
# A /watch/ path the route above does not match (a trailing segment, no token) is still the
|
||||
# A share path the route above does not match (a trailing segment, no token) is still the
|
||||
# player's: WatchPage reads the token off the path and says when a link is gone.
|
||||
if full_path.startswith("watch/"):
|
||||
if full_path.startswith(SHARE_PREFIXES):
|
||||
return FileResponse(WATCH_HTML, headers=INDEX_HEADERS)
|
||||
return FileResponse(INDEX_HTML, headers=INDEX_HEADERS)
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
"""What a user's Plex history leans towards, as a weight for a surprise draw (`app.draw`).
|
||||
|
||||
Two signals, both already in the database (no TMDB call):
|
||||
- **genres** of the titles the user watched, the recent ones weighing more (a half-life of
|
||||
HALF_LIFE_DAYS), each scored 0..1 against the strongest;
|
||||
- the **TMDB recommendations** stored on those titles (`recommendations`), each scored by the most
|
||||
recent title that recommends it.
|
||||
|
||||
A title counts once however many of its episodes were watched — a binge of one show would otherwise
|
||||
outvote every film — and only the RECENT_TITLES latest titles count. A candidate's weight is
|
||||
1 + GENRE_WEIGHT × (the mean score of its genres) + REC_WEIGHT × (its recommendation score), so a
|
||||
title with no affinity keeps weight 1 and still comes up.
|
||||
"""
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import Float, cast, func, literal, literal_column, select
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.sql.elements import ColumnElement
|
||||
|
||||
from app import draw
|
||||
from app.plex.models import PlexItem, PlexShow, PlexState
|
||||
from app.userscope import owned_by
|
||||
|
||||
HALF_LIFE_DAYS = 180
|
||||
RECENT_TITLES = 200
|
||||
GENRE_WEIGHT = 2.0
|
||||
REC_WEIGHT = 3.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Affinity:
|
||||
genres: dict[str, float] = field(default_factory=dict)
|
||||
movie_recs: dict[str, float] = field(default_factory=dict)
|
||||
show_recs: dict[str, float] = field(default_factory=dict)
|
||||
|
||||
def __bool__(self) -> bool:
|
||||
return bool(self.genres or self.movie_recs or self.show_recs)
|
||||
|
||||
def weight(self, model: type[PlexItem] | type[PlexShow]) -> ColumnElement[float]:
|
||||
"""The draw weight of a `model` row (a movie or a show)."""
|
||||
scores = cast(literal(json.dumps(self.genres)), JSONB)
|
||||
recs = self.show_recs if model is PlexShow else self.movie_recs
|
||||
genre = func.jsonb_array_elements_text(model.genres).table_valued("value").alias("g")
|
||||
genre_aff = (
|
||||
select(
|
||||
func.coalesce(
|
||||
func.avg(func.coalesce(cast(scores.op("->>")(genre.c.value), Float), 0.0)),
|
||||
0.0,
|
||||
)
|
||||
)
|
||||
.select_from(genre)
|
||||
.scalar_subquery()
|
||||
)
|
||||
rec_aff = func.coalesce(
|
||||
cast(cast(literal(json.dumps(recs)), JSONB).op("->>")(model.guid), Float), 0.0
|
||||
)
|
||||
return literal_column("1.0") + GENRE_WEIGHT * genre_aff + REC_WEIGHT * rec_aff
|
||||
|
||||
|
||||
def for_user(db: Session, uid: int) -> Affinity:
|
||||
"""The user's affinity from their latest watched titles; empty (`not aff`) with no history."""
|
||||
# One key per title: a show for an episode, the item for a film. concat() of a NULL show_id is
|
||||
# just "s", which nullif turns back into NULL so the film's key takes over.
|
||||
title = func.coalesce(
|
||||
func.nullif(func.concat("s", PlexItem.show_id), "s"), func.concat("m", PlexItem.id)
|
||||
).label("title")
|
||||
latest = (
|
||||
select(title, PlexState.watched_at, PlexItem.id.label("item_id"), PlexItem.show_id)
|
||||
.join(PlexItem, PlexItem.id == PlexState.item_id)
|
||||
.where(owned_by(PlexState.user_id, uid), PlexState.status == "watched")
|
||||
.distinct(title)
|
||||
.order_by(title, PlexState.watched_at.desc().nullslast())
|
||||
.subquery()
|
||||
)
|
||||
rows = db.execute(
|
||||
select(
|
||||
latest.c.watched_at,
|
||||
PlexItem.genres,
|
||||
PlexItem.recommendations,
|
||||
PlexShow.genres,
|
||||
PlexShow.recommendations,
|
||||
)
|
||||
.select_from(latest)
|
||||
.join(PlexItem, PlexItem.id == latest.c.item_id)
|
||||
.outerjoin(PlexShow, PlexShow.id == latest.c.show_id)
|
||||
.order_by(latest.c.watched_at.desc().nullslast())
|
||||
.limit(RECENT_TITLES)
|
||||
).all()
|
||||
if not rows:
|
||||
return Affinity()
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
genres: dict[str, float] = {}
|
||||
movie_recs: dict[str, float] = {}
|
||||
show_recs: dict[str, float] = {}
|
||||
for watched_at, item_genres, item_recs, show_genres, show_recs_json in rows:
|
||||
r = draw.recency(watched_at, now, HALF_LIFE_DAYS)
|
||||
for g in (show_genres if show_genres is not None else item_genres) or []:
|
||||
genres[g] = genres.get(g, 0.0) + r
|
||||
for rec in (show_recs_json if show_recs_json is not None else item_recs) or []:
|
||||
if not rec.get("tmdb_id"):
|
||||
continue
|
||||
bucket = show_recs if rec.get("media_type") == "tv" else movie_recs
|
||||
guid = f"tmdb:{rec['tmdb_id']}"
|
||||
bucket[guid] = max(bucket.get(guid, 0.0), r)
|
||||
return Affinity(draw.scaled(genres), draw.scaled(movie_recs), draw.scaled(show_recs))
|
||||
@@ -22,7 +22,8 @@ from sqlalchemy.orm import Session
|
||||
from app.collation import sorted_words
|
||||
from app.discover.queries import RAIL_LIMIT
|
||||
from app.models import User
|
||||
from app.plex import rollup
|
||||
from app import draw
|
||||
from app.plex import affinity, rollup
|
||||
from app.plex.cards import SHOW_NEW, display_title, leaf_card, movie_card, show_card
|
||||
from app.plex.models import PlexItem, PlexShow, PlexState
|
||||
from app.plex.query import (
|
||||
@@ -399,22 +400,25 @@ def discover_page(db: Session, user: User, scope: str, lang: str, f: LibraryFilt
|
||||
}
|
||||
|
||||
|
||||
def _draw(db: Session, stmt, key_col, per: int, exclude: list[str]) -> list:
|
||||
def _draw(db: Session, stmt, key_col, per: int, exclude: list[str], weight=None) -> list:
|
||||
"""`per` random rows of `stmt`, the previous draw's (`exclude`) left out while the pool allows: a
|
||||
reshuffle over a small filtered pool kept dealing the same few titles. What the fresh rows leave
|
||||
short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail."""
|
||||
fresh = (
|
||||
db.execute(stmt.where(key_col.not_in(exclude)).order_by(func.random()).limit(per))
|
||||
.scalars()
|
||||
.all()
|
||||
if exclude
|
||||
else []
|
||||
)
|
||||
short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail.
|
||||
With a `weight` (`plex.affinity`) the draw leans on it, every third slot left blind (`app.draw`)."""
|
||||
|
||||
def pick(q, n: int) -> list:
|
||||
blind = db.execute(q.order_by(func.random()).limit(n)).scalars().all()
|
||||
if weight is None:
|
||||
return list(blind)
|
||||
key = draw.weighted_key(draw.random_u(), weight)
|
||||
leaning = db.execute(q.order_by(key).limit(n)).scalars().all()
|
||||
return draw.blend(list(leaning), list(blind), 1, n, key=lambda r: r.rating_key)
|
||||
|
||||
fresh = pick(stmt.where(key_col.not_in(exclude)), per) if exclude else []
|
||||
if len(fresh) == per:
|
||||
return list(fresh)
|
||||
return fresh
|
||||
taken = [key_col.not_in([r.rating_key for r in fresh])] if fresh else []
|
||||
rest = db.execute(stmt.where(*taken).order_by(func.random()).limit(per - len(fresh))).scalars()
|
||||
return [*fresh, *rest.all()]
|
||||
return [*fresh, *pick(stmt.where(*taken), per - len(fresh))]
|
||||
|
||||
|
||||
def _kept(
|
||||
@@ -474,6 +478,8 @@ def surprise(
|
||||
if keep:
|
||||
return {"scope": scope, "items": _kept(db, user.id, movie_ids, show_ids, keep, lang)}
|
||||
per = min(limit, SURPRISE_PER_KIND)
|
||||
# What the user's history leans towards; with none, every title weighs the same (a plain draw).
|
||||
leaning = affinity.for_user(db, user.id)
|
||||
movies = []
|
||||
if movie_ids:
|
||||
# Exclude movies this user has watched.
|
||||
@@ -501,6 +507,7 @@ def surprise(
|
||||
PlexItem.rating_key,
|
||||
per,
|
||||
exclude,
|
||||
leaning.weight(PlexItem) if leaning else None,
|
||||
)
|
||||
shows = []
|
||||
if show_ids:
|
||||
@@ -523,6 +530,7 @@ def surprise(
|
||||
PlexShow.rating_key,
|
||||
per,
|
||||
exclude,
|
||||
leaning.weight(PlexShow) if leaning else None,
|
||||
)
|
||||
mcards = _movie_cards(db, user.id, movies, lang)
|
||||
scards = _show_cards(db, user.id, shows, lang)
|
||||
|
||||
@@ -227,16 +227,17 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
|
||||
playable: Mapped[str | None] = mapped_column(String(12), index=True)
|
||||
probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# Transcode HW-capability facet (the playback-class filter). NULL for non-transcode items; for a
|
||||
# `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
|
||||
# AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
|
||||
# breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
|
||||
# set codec-only at sync (bit depth unknown from the cheap listing), then refined by the bit-depth
|
||||
# backfill (`_probe_media_batch`) which downgrades a 10-bit/HDR HEVC to False. The one filterable
|
||||
# home of the invariant `_want_hw`'s source gate computes at play time — kept in sync via the shared
|
||||
# classifier so the catalog facet and the runtime encoder choice never disagree by codec.
|
||||
# `playable="transcode"` item: True = the prod iGPU ENCODES it (a HW-decodable 8-bit SDR source, a
|
||||
# true-HDR one through the OpenCL tonemap, or an 8-bit source in a codec the CPU decodes for it —
|
||||
# VC-1, XviD, DivX), False = software libx264 (AV1, or a 10-bit SDR source — see plex/stream.py).
|
||||
# Derived by `plex.sync.transcode_hw_capable`: set codec-only at sync (bit depth unknown from the
|
||||
# cheap listing), then refined by the bit-depth backfill (`_probe_media_batch`), which downgrades a
|
||||
# 10-bit SDR HEVC to False. The filterable twin of the rule `_select_encode` applies at play time;
|
||||
# the two share their codec sets, not their code, so a change to one tree is a change to both.
|
||||
transcode_hw: Mapped[bool | None] = mapped_column(Boolean, index=True)
|
||||
# Video bit depth (8/10/12…) from the Plex media Stream, filled by the bit-depth backfill. NULL =
|
||||
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw`.
|
||||
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw` (a
|
||||
# 10-bit SDR source drops to software; an HDR one keeps the GPU through the tonemap).
|
||||
video_bit_depth: Mapped[int | None] = mapped_column(Integer)
|
||||
# Watermark for the bit-depth backfill: when this item's media Stream was last probed for
|
||||
# bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are
|
||||
|
||||
+113
-54
@@ -4,9 +4,10 @@ Playback is from the LOCAL physical file. Browser-compatible files (playable="di
|
||||
raw with HTTP range requests. Everything else that is h264 video (playable="remux") is remuxed on
|
||||
the fly to HLS with **video stream-copy** (cheap, I/O-bound — no video re-encode) and audio to AAC
|
||||
when needed. Non-h264 video (playable="transcode": HEVC/VP9/AV1/MPEG-2/XviD…) needs a full re-encode
|
||||
to H.264: on the prod host that runs on the Intel iGPU via VAAPI (`h264_vaapi`, near-zero CPU), with
|
||||
software libx264 as the automatic fallback where the render node is absent or the source codec isn't
|
||||
hardware-decodable. Same HLS session/seek machinery either way — only the video encoder differs.
|
||||
to H.264: on the prod host that is encoded on the Intel iGPU via VAAPI (`h264_vaapi`) — decoded there
|
||||
too when it can, else decoded on the CPU and uploaded (VC-1, XviD, DivX) — with software libx264 as
|
||||
the fallback where the render node is absent, for a source neither GPU route takes (AV1, 10-bit SDR),
|
||||
and when a GPU session produces nothing. Same HLS session/seek machinery either way.
|
||||
|
||||
Seek model (Jellyfin-style): one ffmpeg session per item, started at a given offset via `-ss`
|
||||
(fast keyframe seek). A seek beyond the generated region restarts the session at the new offset,
|
||||
@@ -35,7 +36,7 @@ from app.config import settings
|
||||
from app.downloads.edit import encode_thread_cap
|
||||
from app.plex.models import PlexItem
|
||||
from app.plex import paths
|
||||
from app.plex.sync import HDR_TRANSFERS, HW_DECODABLE_CODECS
|
||||
from app.plex.sync import CPU_DECODE_CODECS, HDR_TRANSFERS, HW_DECODABLE_CODECS
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
_HLS_ROOT = (
|
||||
@@ -120,12 +121,13 @@ class HlsSession:
|
||||
hw: bool = False,
|
||||
tonemap: bool = False,
|
||||
user_id: int = 0,
|
||||
cpu_decode: bool = False,
|
||||
):
|
||||
# Unguessable per-session id, carried in the URLs the client is handed. A seek-restart
|
||||
# replaces the viewer's registered session for the item — and the old player's next request
|
||||
# for `seg_12.ts`, resolved by (viewer, rating_key) alone, was answered 200 with the NEW session's twelfth segment, produced from a different offset
|
||||
# under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it.
|
||||
# A request naming a session that is no longer registered is a 404,
|
||||
# A request naming a session that is no longer registered is a 410,
|
||||
# which the player recovers from by restarting at the playhead.
|
||||
self.id = secrets.token_urlsafe(9)
|
||||
self.key = key # the item's rating_key (what the logs name)
|
||||
@@ -135,6 +137,7 @@ class HlsSession:
|
||||
self.transcode = transcode # video re-encoded (non-h264 source), not stream-copied
|
||||
self.hw = hw # transcode ran on the VAAPI hardware encoder (else libx264 software)
|
||||
self.tonemap = tonemap # HW HDR/10-bit path: OpenCL tonemap between VAAPI decode and encode
|
||||
self.cpu_decode = cpu_decode # HW encode of frames the CPU decoded (a codec the GPU can't)
|
||||
self.start_s = start_s # the REQUESTED seek offset
|
||||
# The REAL media start: with `-ss X -c:v copy` ffmpeg can only cut at the nearest earlier
|
||||
# keyframe K<=X, and hls.js zero-bases the timeline to K (so video.currentTime=0 is K, not X).
|
||||
@@ -188,8 +191,11 @@ _HLS_TAIL = [
|
||||
# * VAAPI hardware (`h264_vaapi`) — the prod default once the iGPU is passed in. A full GPU pipeline
|
||||
# (decode + scale + encode all on-device) offloads the CPU almost entirely, so several concurrent
|
||||
# 1080p transcodes are fine and the API stays responsive.
|
||||
# * CPU decode → VAAPI encode (`format=nv12,hwupload` + `h264_vaapi`) — a codec the GPU cannot decode
|
||||
# but whose 8-bit picture it encodes (`CPU_DECODE_CODECS`: VC-1, XviD, DivX). The encode is the
|
||||
# expensive half, so this still takes most of the load off the CPU.
|
||||
# * libx264 software — the automatic fallback where `/dev/dri` is absent (dev box, or prod before the
|
||||
# iGPU enablement) or the source codec isn't hardware-decodable (AV1/XviD on Gen 9.5). CPU-bound, so
|
||||
# iGPU enablement) or the source fits neither GPU route (AV1, a 10-bit SDR source). CPU-bound, so
|
||||
# it reserves a core via `-threads` (see encode_thread_cap).
|
||||
# Everything else — the `-copyts`/`_HLS_TAIL`/segment/seek-restart model and the AAC audio path — is
|
||||
# unchanged, so hls.js, subtitles, multi-audio and seeking keep working exactly as for remux.
|
||||
@@ -198,10 +204,10 @@ _VAAPI_DEVICE = "/dev/dri/renderD128"
|
||||
# here; its presence is the precondition for the HW HDR→SDR tonemap graph (see _opencl_present).
|
||||
_OPENCL_ICD_DIR = Path("/etc/OpenCL/vendors")
|
||||
# Codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for the
|
||||
# full-GPU VAAPI pipeline. A source outside it (AV1, MPEG-4 ASP/XviD…) falls back to software even with
|
||||
# the render node present, because the GPU can't decode it to feed its own encoder. THE single source is
|
||||
# plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't disagree by
|
||||
# codec; aliased to the old name to keep the call sites below unchanged.
|
||||
# full-GPU VAAPI pipeline. A source outside it is decoded on the CPU when its codec is in
|
||||
# `CPU_DECODE_CODECS` (the GPU still encodes), else transcoded in software. THE single source of both
|
||||
# sets is plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't
|
||||
# disagree by codec; aliased to the old name to keep the call sites below unchanged.
|
||||
_HW_DECODABLE = HW_DECODABLE_CODECS
|
||||
# Color-transfer values that mark a true-HDR source (PQ / HLG). A HW-decodable HDR source can't take the
|
||||
# plain scale_vaapi=format=nv12 path (Gen 9.5 VPP refuses it) — it needs the OpenCL tonemap graph
|
||||
@@ -302,16 +308,22 @@ def _opencl_present() -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _hwaccel_in_opts(hw: bool, tonemap: bool = False) -> list[str]:
|
||||
def _hwaccel_in_opts(hw: bool, tonemap: bool = False, cpu_decode: bool = False) -> list[str]:
|
||||
"""VAAPI decode options, placed before `-i` so the source is decoded (and kept) on the GPU.
|
||||
`-hwaccel_output_format vaapi` keeps frames as GPU surfaces all the way to `h264_vaapi` — no
|
||||
round-trip to system memory. Empty for the software path.
|
||||
|
||||
The tonemap variant additionally derives an OpenCL device FROM the VAAPI one (`opencl=ocl@va`) so
|
||||
tonemap_opencl can map the decoded surfaces zero-copy (see _venc_opts); it needs named devices, so it
|
||||
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`."""
|
||||
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`.
|
||||
|
||||
The CPU-decode variant opens the VAAPI device for the FILTER graph only (`hwupload` needs one) and
|
||||
asks for no `-hwaccel`: the codec is one the GPU refuses to decode. This exact form is the one
|
||||
measured on prod with the HLS muxer and `-copyts` (and to the end of the file)."""
|
||||
if not hw:
|
||||
return []
|
||||
if cpu_decode:
|
||||
return ["-init_hw_device", f"vaapi=va:{_VAAPI_DEVICE}", "-filter_hw_device", "va"]
|
||||
if tonemap:
|
||||
return [
|
||||
"-init_hw_device",
|
||||
@@ -359,6 +371,7 @@ def _venc_opts(
|
||||
preset: str = "veryfast",
|
||||
crf: int = 20,
|
||||
max_height: int = 0,
|
||||
cpu_decode: bool = False,
|
||||
) -> list[str]:
|
||||
"""The video-encoder args that replace `-c:v copy` for a transcode session (returns the copy args
|
||||
when not transcoding). A re-encode must MINT its own keyframes at the segment boundaries — the copy
|
||||
@@ -385,6 +398,10 @@ def _venc_opts(
|
||||
"h264_vaapi",
|
||||
*force_kf,
|
||||
]
|
||||
if hw and cpu_decode:
|
||||
# The CPU decoded the frames: convert them to the encoder's 8-bit nv12 and upload them to the
|
||||
# device `-filter_hw_device` named (see _hwaccel_in_opts).
|
||||
return ["-vf", "format=nv12,hwupload", "-c:v", "h264_vaapi", *force_kf]
|
||||
if hw:
|
||||
# scale_vaapi with format=nv12 downconverts a 10-bit (p010) HEVC source to 8-bit nv12 so
|
||||
# h264_vaapi (8-bit High) can encode it; for an 8-bit source it's a cheap format pass-through.
|
||||
@@ -422,13 +439,18 @@ def _render_node_present() -> bool:
|
||||
return Path(_VAAPI_DEVICE).exists()
|
||||
|
||||
|
||||
def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> tuple[bool, bool]:
|
||||
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap)`:
|
||||
* `(False, False)` → software libx264 (the fallback: no render node, non-decodable source, or a
|
||||
10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
|
||||
* `(True, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR source.
|
||||
* `(True, True)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG) source, when
|
||||
the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
|
||||
def _select_encode(
|
||||
db: DbSession, item: PlexItem, transcode: bool, src: Path
|
||||
) -> tuple[bool, bool, bool]:
|
||||
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap, cpu_decode)`:
|
||||
* `(False, False, False)` → software libx264 (the fallback: no render node, a source neither GPU
|
||||
route takes, or a 10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
|
||||
* `(True, False, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR
|
||||
source the GPU decodes.
|
||||
* `(True, True, False)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG)
|
||||
source, when the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
|
||||
* `(True, False, True)` → CPU decode → `hwupload` → VAAPI encode — an 8-bit SDR source in a codec
|
||||
the GPU cannot decode (`CPU_DECODE_CODECS`).
|
||||
|
||||
A 10-bit *SDR* source is deliberately NOT a `(True, True)` case: it needs a plain 10→8-bit downconvert,
|
||||
not the HDR tone curve (which would visibly darken/flatten an SDR picture), so it drops to software.
|
||||
@@ -438,31 +460,40 @@ def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) ->
|
||||
backstops a session that produces nothing — but still routing a true-HDR source through the tonemap
|
||||
graph so it doesn't hit the scale_vaapi wall); "auto" (default) → HW only when the render node is
|
||||
present AND the iGPU can hardware-decode the source, with tonemap chosen only for a true-HDR (PQ/HLG)
|
||||
source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
|
||||
source that has an OpenCL runtime, while a 10-bit SDR source drops to software. In both "on" and
|
||||
"auto" a codec the GPU is known not to decode goes the CPU-decode route: asking the GPU to decode it
|
||||
only bought the fallback's wasted second before libx264."""
|
||||
sw = (False, False, False)
|
||||
if not transcode:
|
||||
return (False, False)
|
||||
return sw
|
||||
mode = sysconfig.get_str(db, "plex_transcode_hw")
|
||||
if mode == "off" or not _render_node_present():
|
||||
return (False, False)
|
||||
return sw
|
||||
codec = (item.codec_video or "").lower()
|
||||
if mode == "on":
|
||||
if codec in CPU_DECODE_CODECS:
|
||||
return (True, False, True)
|
||||
# Trust the admin: HW wherever the render node exists, skipping the decodability gate (the runtime
|
||||
# backstop catches a miss). Only a true-HDR (PQ/HLG) source needs the tonemap graph — a 10-bit SDR
|
||||
# source takes the plain scale path here, NOT the HDR tone curve. Probe only when OpenCL exists
|
||||
# (short-circuit), so "on" stays probe-free where tonemap can't apply anyway.
|
||||
tonemap = _opencl_present() and _probe_video_kind(src) == _VK_HDR
|
||||
return (True, tonemap)
|
||||
return (True, tonemap, False)
|
||||
# auto
|
||||
if (item.codec_video or "").lower() not in _HW_DECODABLE:
|
||||
return (False, False)
|
||||
if codec in CPU_DECODE_CODECS:
|
||||
# The upload converts to 8-bit nv12: only an 8-bit SDR picture comes through it unchanged.
|
||||
return (True, False, True) if _probe_video_kind(src) == _VK_HWSAFE else sw
|
||||
if codec not in _HW_DECODABLE:
|
||||
return sw
|
||||
kind = _probe_video_kind(src)
|
||||
if kind == _VK_HWSAFE:
|
||||
return (True, False) # 8-bit SDR → full-GPU scale path
|
||||
return (True, False, False) # 8-bit SDR → full-GPU scale path
|
||||
if kind == _VK_HDR:
|
||||
# PQ/HLG: HW OpenCL tonemap when the runtime is present, else software libx264.
|
||||
return (True, True) if _opencl_present() else (False, False)
|
||||
return (True, True, False) if _opencl_present() else sw
|
||||
# _VK_SDR10 — 10-bit SDR: a plain 10→8-bit downconvert, done exactly on the CPU. Never the HDR tone
|
||||
# curve (it would alter an SDR picture), and off the unverified scale_vaapi 10-bit path — software.
|
||||
return (False, False)
|
||||
return sw
|
||||
|
||||
|
||||
def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
|
||||
@@ -471,16 +502,17 @@ def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
|
||||
return _select_encode(db, item, transcode, src)[0]
|
||||
|
||||
|
||||
# CPU-nice increment for the SOFTWARE transcode child. The Plex ffmpeg is a Popen in the long-lived API
|
||||
# process (which must NOT itself be niced), so — unlike the download worker's _deprioritize (run once in
|
||||
# its own main()) — we lower priority per-child via preexec_fn in the fork. The GPU path needs none of
|
||||
# this. Reuses the download child's nice default so all co-located re-encoders share one handicap.
|
||||
# CPU-nice increment for a transcode child that works the CPU: a software encode, or a CPU decode
|
||||
# feeding the GPU encoder. The Plex ffmpeg is a Popen in the long-lived API process (which must NOT
|
||||
# itself be niced), so — unlike the download worker's _deprioritize (run once in its own main()) — we
|
||||
# lower priority per-child via preexec_fn in the fork. The full-GPU path needs none of this. Reuses
|
||||
# the download child's nice default so all co-located re-encoders share one handicap.
|
||||
_SW_TRANSCODE_NICE = settings.download_child_nice
|
||||
|
||||
|
||||
def _sw_child_preexec() -> None:
|
||||
"""Runs in the forked child (POSIX) before exec: renice this software-transcode ffmpeg down so it
|
||||
can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
|
||||
"""Runs in the forked child (POSIX) before exec: renice this CPU-bound transcode ffmpeg down so
|
||||
it can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
|
||||
playback."""
|
||||
try:
|
||||
os.nice(_SW_TRANSCODE_NICE)
|
||||
@@ -540,13 +572,15 @@ def _ffmpeg_cmd(
|
||||
crf: int = 20,
|
||||
max_height: int = 0,
|
||||
loudnorm: bool = False,
|
||||
cpu_decode: bool = False,
|
||||
) -> list[str]:
|
||||
# For a transcode session the video is re-encoded (VAAPI or libx264) instead of stream-copied; the
|
||||
# hwaccel decode options go before -i, the encoder args replace `-c:v copy` below. Everything else is
|
||||
# identical to remux. `tonemap` is the HDR/10-bit HW variant (OpenCL tonemap between VAAPI decode and
|
||||
# encode) — it needs matching decode-side (named devices) and encode-side (the OpenCL filter graph) opts.
|
||||
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height)
|
||||
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap)]
|
||||
# `cpu_decode` is the HW encode of CPU-decoded frames: a device for the filter graph, no -hwaccel.
|
||||
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height, cpu_decode)
|
||||
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap, cpu_decode)]
|
||||
# Keep ORIGINAL timestamps on the output so the first segment carries the true absolute PTS of the
|
||||
# keyframe ffmpeg actually seeked to. hls.js still zero-bases playback to that PTS; we read it back
|
||||
# (see _probe_first_pts) to learn the real content offset. Without -copyts the segment PTS is
|
||||
@@ -729,7 +763,7 @@ def _enforce_caps(
|
||||
whole segmented copy of the film under `_HLS_ROOT`, written faster than realtime. Counted only
|
||||
by the transcode cap, N distinct items were N ffmpegs and N film-sized scratch directories.
|
||||
|
||||
Somebody's playback stops at each drop. The route turns the missing session into a retryable 404
|
||||
Somebody's playback stops at each drop. The route turns the missing session into a 410
|
||||
and the player restarts at the playhead, so from the server nothing failed — and "my film keeps
|
||||
rebuffering" has no other explanation anywhere, hence the line."""
|
||||
|
||||
@@ -800,7 +834,11 @@ def start_session(
|
||||
return directory / segment_name(i, _VIDEO_VARIANT if variants else None)
|
||||
|
||||
def _spawn(
|
||||
hw: bool, tonemap: bool = False, *, replace_only: "HlsSession | None" = None
|
||||
hw: bool,
|
||||
tonemap: bool = False,
|
||||
cpu_decode: bool = False,
|
||||
*,
|
||||
replace_only: "HlsSession | None" = None,
|
||||
) -> "HlsSession | None":
|
||||
# `replace_only` makes the supersede identity-aware for the runtime fallback: the whole
|
||||
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
|
||||
@@ -815,10 +853,12 @@ def start_session(
|
||||
preset = sysconfig.get_str(db, "plex_transcode_preset")
|
||||
crf = sysconfig.get_int(db, "plex_transcode_crf")
|
||||
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
|
||||
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
|
||||
# can't starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a
|
||||
# platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
|
||||
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
|
||||
# Nice the child that works the CPU — a SOFTWARE transcode, or a CPU decode feeding the GPU
|
||||
# encoder (not the full-GPU path, and not remux) — so it can't starve the co-located
|
||||
# API/playback. preexec_fn runs in the forked child (POSIX); on a platform without fork (the API
|
||||
# always runs in a Linux container in prod) it's simply omitted.
|
||||
cpu_bound = transcode and (not hw or cpu_decode)
|
||||
preexec = _sw_child_preexec if (cpu_bound and os.name == "posix") else None
|
||||
doomed: list[tuple[HlsSession, Path | None]] = []
|
||||
# Finished even when the new session fails to start (no space, no ffmpeg, a DB error): what
|
||||
# was moved aside under the lock is otherwise left for a restart's sweep.
|
||||
@@ -851,6 +891,7 @@ def start_session(
|
||||
crf=crf,
|
||||
max_height=max_height,
|
||||
loudnorm=loudnorm,
|
||||
cpu_decode=cpu_decode,
|
||||
),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
@@ -866,6 +907,7 @@ def start_session(
|
||||
hw=hw,
|
||||
tonemap=tonemap,
|
||||
user_id=user_id,
|
||||
cpu_decode=cpu_decode,
|
||||
)
|
||||
_sessions[reg] = s
|
||||
# The one line that says which pipeline a session actually got. Everything downstream
|
||||
@@ -879,6 +921,7 @@ def start_session(
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
tonemap=tonemap,
|
||||
cpu_decode=cpu_decode,
|
||||
variants=len(variants or ()),
|
||||
start_s=round(start_s, 1),
|
||||
),
|
||||
@@ -918,7 +961,7 @@ def start_session(
|
||||
# stale and stops the session it names (`stop_session`), which stopped the playback that won.
|
||||
# Never None either: the POST /session route turns a None return into a "Media file not found
|
||||
# on disk" 404 — dishonest for a file that exists. A dead session instead degrades to a
|
||||
# retryable 404 ("No active playback session" / "Not available yet") the client re-requests.
|
||||
# 410 ("session_gone") on its URLs, and the client asks again.
|
||||
return s
|
||||
s = respawned
|
||||
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
|
||||
@@ -937,7 +980,8 @@ def start_session(
|
||||
if not wait_for(_seg(i), timeout=_WARMUP_SEG_TIMEOUT_S, proc=s.proc):
|
||||
break
|
||||
log.info(
|
||||
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s tonemap=%s",
|
||||
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s"
|
||||
" tonemap=%s cpu_decode=%s",
|
||||
key,
|
||||
start_s,
|
||||
s.media_start_s,
|
||||
@@ -946,6 +990,7 @@ def start_session(
|
||||
transcode,
|
||||
s.hw,
|
||||
s.tonemap,
|
||||
s.cpu_decode,
|
||||
)
|
||||
return s
|
||||
|
||||
@@ -1160,9 +1205,13 @@ def sweep_orphans(older_than: float | None = None) -> int:
|
||||
return dropped
|
||||
|
||||
|
||||
# How long a session whose ffmpeg has FINISHED may go unfetched before a new session request reaps
|
||||
# it (see reap_idle).
|
||||
_FINISHED_IDLE_S = 30
|
||||
# The slowest heartbeat a live player sends: a browser cuts the timers of a hidden page that makes no
|
||||
# sound to one a minute (measured on Chromium: a paused and a muted player, about a minute after
|
||||
# hiding; a hidden player with sound keeps its 10 s).
|
||||
_THROTTLED_HEARTBEAT_S = 60
|
||||
# How long a session whose ffmpeg has FINISHED may go without a signal before `reap_idle` takes it:
|
||||
# the beat before the reaper looks, one lost on the way, and the time the next one takes to arrive.
|
||||
_FINISHED_IDLE_S = 3 * _THROTTLED_HEARTBEAT_S
|
||||
|
||||
|
||||
def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
|
||||
@@ -1171,20 +1220,23 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
|
||||
|
||||
A finished session is also what a PAUSED viewer has: a remux runs far faster than realtime, so
|
||||
ffmpeg is done a minute or two in and writes ENDLIST; hls.js then stops asking for the playlist,
|
||||
and with its forward buffer full a paused player fetches nothing at all. With the 30 s rule that
|
||||
killed a one-minute pause, and playback died ~30 s after resuming.
|
||||
and with its forward buffer full a paused player fetches nothing at all.
|
||||
|
||||
What makes the short rule safe again is that fetching is no longer the only signal: a player
|
||||
parked MID-ITEM checkpoints its position every 10 s, and that route calls `touch_session`, so a
|
||||
viewer who is merely paused says so far more often than this rule asks. The heartbeat
|
||||
What makes the short rule safe is that fetching is not the only signal: a player parked MID-ITEM
|
||||
checkpoints its position and that route calls `touch_session`. That beat is every 10 s only while
|
||||
the page is visible or making sound; a hidden, silent one sends it once a minute, which a 30 s
|
||||
rule killed whenever the reaper happened to run more than 30 s after the beat (measured: a paused
|
||||
session gone that way, and one that lived on the same cadence because its phase was luckier).
|
||||
So the rule is sized to the throttled beat (`_THROTTLED_HEARTBEAT_S`). The heartbeat
|
||||
stops once the item is finished or before a position is known (`usePlexProgress.flush`) — both
|
||||
states where reaping is correct, and where the player's 404 → reclaim recovers a viewer who
|
||||
scrubs back in anyway. Do not restate this as "every 10 s whether or not it is playing":
|
||||
states where reaping is correct, and where the player's 410 → reclaim recovers a viewer who
|
||||
scrubs back in anyway. Do not restate this as "whether or not it is playing":
|
||||
a review found that claim in two comments and not in the code. An earlier blanket
|
||||
`reap_abandoned` bought the same safety by keeping every finished session for ten minutes — each
|
||||
one a whole segmented copy of the film on a volume shared with the download center."""
|
||||
now = time.time()
|
||||
doomed: list[tuple[HlsSession, Path | None]] = []
|
||||
reaped: list[tuple[str, str, int]] = []
|
||||
with _lock:
|
||||
for reg, s in list(_sessions.items()):
|
||||
done = s.proc.poll() is not None
|
||||
@@ -1192,6 +1244,13 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
|
||||
if idle > _SESSION_IDLE_S or (done and idle > finished_idle_s):
|
||||
_sessions.pop(reg, None)
|
||||
doomed.append(_detach(s))
|
||||
reaped.append((s.key, "finished_idle" if done else "idle", int(idle)))
|
||||
# The viewer of a session wrongly taken sees only a 410 later, so this line is the one record of why.
|
||||
for key, reason, idle_s in reaped:
|
||||
log.info(
|
||||
"an idle session was reaped",
|
||||
extra=logs.event("plex.stream.session_reaped", key=key, reason=reason, idle_s=idle_s),
|
||||
)
|
||||
_finish(doomed)
|
||||
return len(doomed)
|
||||
|
||||
|
||||
+34
-21
@@ -60,14 +60,20 @@ def classify_playable(vcodec: str | None, acodec: str | None, container: str | N
|
||||
|
||||
# Video codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for
|
||||
# the full-GPU VAAPI transcode pipeline. THE single source of this set: plex.stream imports it here so
|
||||
# the runtime encoder choice (`_want_hw`) and this catalog facet can never disagree by codec. A source
|
||||
# outside it (AV1, MPEG-4 ASP/XviD…) is software-only even on the iGPU (it can't decode it to feed its
|
||||
# own encoder).
|
||||
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"})
|
||||
# the runtime encoder choice (`_select_encode`) and this catalog facet can never disagree by codec. VC-1
|
||||
# is NOT in it: measured on prod, the iHD driver answers "No support for codec vc1 profile 3" (VC-1
|
||||
# Advanced, what a Blu-ray remux carries), so every session that asked the GPU to decode one died and
|
||||
# fell back to libx264.
|
||||
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "mpeg2video"})
|
||||
# Codecs the iGPU cannot decode but whose picture it can still ENCODE: the CPU decodes, the frames go up
|
||||
# to the GPU as 8-bit nv12, and h264_vaapi encodes (plex.stream's `cpu_decode` path). Measured on prod:
|
||||
# three segments in ~5 s for a 1080p VC-1 remux (libx264: ~17 s) and ~1 s for an SD XviD or DivX
|
||||
# episode. Only codecs that are always 8-bit SDR, and only ones measured; any other codec stays software.
|
||||
CPU_DECODE_CODECS = frozenset({"vc1", "mpeg4", "msmpeg4v3"})
|
||||
# Color-transfer values marking an HDR source (PQ / HLG). THE single source of this set: plex.stream
|
||||
# (the tonemap choice) and downloads.clip (a clip's tonemap) import it here, so a transfer added for
|
||||
# one is tonemapped by all. A 10-bit/HDR source breaks the VAAPI scale=nv12 path on Gen 9.5 (see
|
||||
# plex.stream) → software transcode, so it counts as NOT HW-capable.
|
||||
# one is tonemapped by all. An HDR source breaks the plain VAAPI scale=nv12 path on Gen 9.5, so it takes
|
||||
# the OpenCL tonemap graph (see plex.stream) — still the GPU encoder.
|
||||
HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
|
||||
|
||||
|
||||
@@ -78,18 +84,24 @@ def transcode_hw_capable(
|
||||
hdr: bool = False,
|
||||
) -> bool | None:
|
||||
"""The stored `PlexItem.transcode_hw` facet: for a `playable="transcode"` item, whether the prod
|
||||
iGPU can HARDWARE-transcode it (else software libx264 only); None for a non-transcode item. HW needs
|
||||
a HW-decodable codec AND an 8-bit SDR source — a 10-bit/HDR source (bit_depth>8 or an HDR transfer)
|
||||
breaks the VAAPI nv12 pipeline and falls back to software. `bit_depth`/`hdr` are unknown at sync
|
||||
time (the cheap listing carries no Stream), so they default to 8-bit SDR (codec-only) and the
|
||||
bit-depth backfill refines a 10-bit/HDR HEVC to False later."""
|
||||
iGPU ENCODES it ("Transcoded on the hardware encoder"), else software libx264; None for a
|
||||
non-transcode item. The same answer plex.stream's `_select_encode` gives on prod (render node and
|
||||
OpenCL runtime present, policy "auto"):
|
||||
* a HW-decodable codec: 8-bit SDR → the full-GPU path; HDR → the OpenCL tonemap, also on the GPU;
|
||||
10-bit SDR → software (an exact downconvert, never the HDR tone curve);
|
||||
* a CPU-decode codec (`CPU_DECODE_CODECS`) at 8 bits → CPU decode, GPU encode;
|
||||
* any other codec → software.
|
||||
`bit_depth`/`hdr` are unknown at sync time (the cheap listing carries no Stream), so they default to
|
||||
8-bit SDR and the bit-depth backfill refines a 10-bit SDR HEVC to False later."""
|
||||
if playable != "transcode":
|
||||
return None
|
||||
if (vcodec or "").lower() not in HW_DECODABLE_CODECS:
|
||||
return False
|
||||
if bit_depth is not None and bit_depth > 8:
|
||||
return False
|
||||
return not hdr
|
||||
codec = (vcodec or "").lower()
|
||||
deep = bit_depth is not None and bit_depth > 8
|
||||
if codec in HW_DECODABLE_CODECS:
|
||||
return hdr or not deep
|
||||
if codec in CPU_DECODE_CODECS:
|
||||
return not deep and not hdr
|
||||
return False
|
||||
|
||||
|
||||
def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
|
||||
@@ -266,8 +278,9 @@ def sync(db: Session) -> dict:
|
||||
# a Plex shape change, a DB error or our own bug, and `%s` told them apart badly.
|
||||
log.exception("the Plex people-enrich batch failed")
|
||||
# Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
|
||||
# playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
|
||||
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
|
||||
# playback-class filter's HW/SW split is precise (a 10-bit SDR HEVC → software) over
|
||||
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are classed
|
||||
# by codec alone.
|
||||
try:
|
||||
stats["media_probed"] = _probe_media_batch(db, plex)
|
||||
except Exception: # best-effort; never fail the sync over a backfill
|
||||
@@ -294,15 +307,15 @@ def sync(db: Session) -> dict:
|
||||
# whole library is covered over successive runs (each fetched item is stamped people_enriched_at once).
|
||||
_ENRICH_BATCH = 60
|
||||
# Same bounded-per-run pattern for the bit-depth probe. Only HW-decodable-codec transcode items are
|
||||
# probed (typically a small minority — the bulk of transcodes are old codecs that are SW by codec), so
|
||||
# probed (typically a small minority — the bulk of transcodes are old codecs classed by codec), so
|
||||
# a modest batch covers the work-list in a run or two; a stuck item retries next run (watermark unset).
|
||||
_PROBE_BATCH = 60
|
||||
|
||||
|
||||
def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH) -> int:
|
||||
"""Rich-fetch up to `limit` not-yet-probed HW-decodable transcode items, read the video Stream's
|
||||
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit/HDR HEVC →
|
||||
False/software). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
|
||||
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit SDR HEVC →
|
||||
False/software; an HDR one stays on the GPU through the tonemap). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
|
||||
so an item is probed once and not re-fetched every sync; a transient fetch FAILURE leaves the
|
||||
watermark unset to retry. Oldest-id first. Returns how many items were probed this run."""
|
||||
rows = (
|
||||
|
||||
@@ -41,11 +41,25 @@ def video(db: Session, video_id: str) -> Video | None:
|
||||
# --- tags ------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int]]:
|
||||
"""The tags this user sees (the shared system ones and their own), each with how many channel
|
||||
links it has, by category then name."""
|
||||
def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int, int]]:
|
||||
"""The tags this user sees (the shared system ones and their own), by category then name, each
|
||||
with how many channel links it has and how many of the user's own links sit on a channel they
|
||||
subscribe to and have not hidden — the rows the Channels page's tag filter shows (a hidden
|
||||
channel has its own tab). A link outlives an unsubscribe and a hide, so the two differ."""
|
||||
subscribed = select(Subscription.channel_id).where(
|
||||
owned_by(Subscription.user_id, user), Subscription.hidden.is_(False)
|
||||
)
|
||||
on_subscribed = dict(
|
||||
db.execute(
|
||||
select(ChannelTag.tag_id, func.count(ChannelTag.id))
|
||||
.where(owned_by(ChannelTag.user_id, user), ChannelTag.channel_id.in_(subscribed))
|
||||
.group_by(ChannelTag.tag_id)
|
||||
)
|
||||
.tuples()
|
||||
.all()
|
||||
)
|
||||
return [
|
||||
(tag, count)
|
||||
(tag, count, on_subscribed.get(tag.id, 0))
|
||||
for tag, count in db.execute(
|
||||
select(Tag, func.count(ChannelTag.id))
|
||||
.outerjoin(ChannelTag, ChannelTag.tag_id == Tag.id)
|
||||
@@ -132,12 +146,17 @@ def item_counts(db: Session, ids: list[int]) -> dict[int, int]:
|
||||
)
|
||||
|
||||
|
||||
def durations(db: Session, ids: list[int]) -> dict[int, int]:
|
||||
"""Each playlist's running time: the sum of its videos' lengths."""
|
||||
def durations(db: Session, ids: list[int]) -> dict[int, tuple[int, int]]:
|
||||
"""Each playlist's running time — the sum of its videos' known lengths — and how many videos
|
||||
it left out for having none (`Video.without_length`)."""
|
||||
return {
|
||||
pid: int(total or 0)
|
||||
for pid, total in db.execute(
|
||||
select(PlaylistItem.playlist_id, func.coalesce(func.sum(Video.duration_seconds), 0))
|
||||
pid: (int(total or 0), int(left_out or 0))
|
||||
for pid, total, left_out in db.execute(
|
||||
select(
|
||||
PlaylistItem.playlist_id,
|
||||
func.coalesce(func.sum(Video.length_seconds), 0),
|
||||
func.count().filter(Video.without_length),
|
||||
)
|
||||
.join(Video, Video.id == PlaylistItem.video_id)
|
||||
.where(PlaylistItem.playlist_id.in_(ids))
|
||||
.group_by(PlaylistItem.playlist_id)
|
||||
@@ -177,11 +196,16 @@ def covers(db: Session, ids: list[int]) -> dict[int, str | None]:
|
||||
)
|
||||
|
||||
|
||||
def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None]:
|
||||
"""One playlist's item count, running time and cover. An outer join, so an empty list still
|
||||
answers (0, 0, None)."""
|
||||
count, total = db.execute(
|
||||
select(func.count(PlaylistItem.id), func.coalesce(func.sum(Video.duration_seconds), 0))
|
||||
def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, int, str | None]:
|
||||
"""One playlist's item count, running time, the videos the running time left out for having no
|
||||
length (`Video.without_length`) and cover. An outer join, so an empty list still answers
|
||||
(0, 0, 0, None)."""
|
||||
count, total, left_out = db.execute(
|
||||
select(
|
||||
func.count(PlaylistItem.id),
|
||||
func.coalesce(func.sum(Video.length_seconds), 0),
|
||||
func.count().filter(Video.without_length),
|
||||
)
|
||||
.select_from(PlaylistItem)
|
||||
.outerjoin(Video, Video.id == PlaylistItem.video_id)
|
||||
.where(PlaylistItem.playlist_id == playlist_id)
|
||||
@@ -193,7 +217,7 @@ def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None
|
||||
.order_by(PlaylistItem.position)
|
||||
.limit(1)
|
||||
)
|
||||
return count or 0, total or 0, cover
|
||||
return count or 0, total or 0, left_out or 0, cover
|
||||
|
||||
|
||||
def playlist_rows(db: Session, user: User, playlist_id: int) -> list:
|
||||
|
||||
+159
-65
@@ -9,7 +9,7 @@ unique key, a row lock taken before a presence check. The comfort shelf's writes
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from typing import cast
|
||||
from typing import Literal, cast
|
||||
|
||||
from sqlalchemy import Table, bindparam, delete, exists, func, select, text, update
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
@@ -255,17 +255,6 @@ def mark(db: Session, user: User, video_id: str, status: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def clear_state(db: Session, user: User, video_id: str) -> None:
|
||||
"""The video back to pristine for this user: status, resume point and `watched_at` all go.
|
||||
Playlist membership lives elsewhere and stays."""
|
||||
lock_takeout(db, user) # as `unmark`
|
||||
row = queries.video_state(db, user, video_id)
|
||||
if row is not None:
|
||||
db.delete(row)
|
||||
_remember_unmark(db, user, video_id)
|
||||
db.flush()
|
||||
|
||||
|
||||
def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
|
||||
"""The player's resume point. 0 clears it: a row holding nothing else goes, one with a status
|
||||
(watched, hidden) keeps it with the position zeroed. The status itself is never changed here.
|
||||
@@ -301,12 +290,17 @@ def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
|
||||
# --- playlists -------------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Every reason a move is refused; the UI names each one (`playlists.moveErrors.<reason>`, held by the
|
||||
# frontend's backendIds contract, which reads this Literal).
|
||||
MoveReason = Literal["same_playlist"]
|
||||
|
||||
|
||||
class MoveRefused(Exception):
|
||||
"""A move between playlists that cannot happen: `reason` is the code the UI translates."""
|
||||
|
||||
def __init__(self, reason: str):
|
||||
def __init__(self, reason: MoveReason):
|
||||
super().__init__(reason)
|
||||
self.reason = reason
|
||||
self.reason: MoveReason = reason
|
||||
|
||||
|
||||
def touch(pl: Playlist) -> None:
|
||||
@@ -412,30 +406,6 @@ def remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) -
|
||||
return True
|
||||
|
||||
|
||||
def place_item(db: Session, pl: Playlist, video_id: str, position: int) -> None:
|
||||
"""Put a video already in the list at `position` (clamped to the end), renumbering the whole
|
||||
list: positions can have gaps after removals, so "index N" is only meaningful over the
|
||||
ordered rows."""
|
||||
rows = queries.items(db, pl.id)
|
||||
new = next(r for r in rows if r.video_id == video_id)
|
||||
rows.remove(new)
|
||||
rows.insert(min(position, len(rows)), new)
|
||||
for pos, row in enumerate(rows):
|
||||
row.position = pos
|
||||
db.flush()
|
||||
|
||||
|
||||
def add_to_playlist(db: Session, pl: Playlist, video_id: str, position: int | None) -> None:
|
||||
"""Add a video to one of the user's lists, at `position` when given (the list page's "Undo" of
|
||||
a removal puts the row back where it was in ONE request), else at the end. A video already in
|
||||
the list stays where it is."""
|
||||
added = add_item(db, pl, video_id, mark_dirty=False)
|
||||
if added and position is not None:
|
||||
place_item(db, pl, video_id, position)
|
||||
recompute_dirty(db, pl)
|
||||
db.flush()
|
||||
|
||||
|
||||
def rename_playlist(db: Session, pl: Playlist, name: str) -> None:
|
||||
"""A new name moves `dirty`; the caller holds the playlist's lock (the scheduled mirror refresh
|
||||
reads `dirty` under it)."""
|
||||
@@ -453,39 +423,163 @@ def delete_playlist(db: Session, pl: Playlist) -> None:
|
||||
db.flush()
|
||||
|
||||
|
||||
def move_item(db: Session, src: Playlist, dst: Playlist, video_id: str) -> None:
|
||||
"""Move one video from `src` into `dst`, in the caller's one transaction.
|
||||
def remove_items(db: Session, pl: Playlist, video_ids: list[str]) -> list[str]:
|
||||
"""Remove these videos from a playlist, in one transaction; a video not in it any more is simply
|
||||
not among them. Says which went: an Undo must not bring back one this removal never took."""
|
||||
lock_playlists(db, pl.id)
|
||||
wanted = set(video_ids)
|
||||
rows = [it for it in queries.items(db, pl.id) if it.video_id in wanted]
|
||||
for row in rows:
|
||||
db.delete(row)
|
||||
if rows:
|
||||
recompute_dirty(db, pl)
|
||||
touch(pl)
|
||||
db.flush()
|
||||
return [row.video_id for row in rows]
|
||||
|
||||
The client-side shape — add to the target, then remove from the source — leaves the video in
|
||||
BOTH lists whenever the second call fails. Idempotent: a video already in the target is simply
|
||||
dropped from the source (moving into a list that has it is a removal, not a duplicate). Raises
|
||||
`MoveRefused` for the same list twice, or a video a concurrent edit already took out."""
|
||||
|
||||
def _insert_at(
|
||||
db: Session,
|
||||
pl: Playlist,
|
||||
placed: list[tuple[int, str]],
|
||||
leaving: dict[str, PlaylistItem] | None = None,
|
||||
staying_out: frozenset[str] = frozenset(),
|
||||
) -> int:
|
||||
"""Add videos at their indices (ascending, so each index is where the row stood when the rows
|
||||
before it were back) and renumber the list 0..n-1, as positions can have gaps. A video already in
|
||||
the list stays where it is, and one the catalog has lost or in `staying_out` is left out, the
|
||||
ones after it closing up. A video with a row in `leaving` brings that row along, for the reason
|
||||
`move_items` re-parents. Says how many were added; the caller holds the playlist's lock."""
|
||||
leaving = leaving or {}
|
||||
rows = queries.items(db, pl.id)
|
||||
present = {r.video_id for r in rows}
|
||||
known = queries.known_video_ids(db, [vid for _, vid in placed])
|
||||
added = 0
|
||||
left_out = 0
|
||||
for at, vid in sorted(placed, key=lambda p: p[0]):
|
||||
if vid in present or vid not in known or vid in staying_out:
|
||||
left_out += 1
|
||||
continue
|
||||
row = leaving.pop(vid, None)
|
||||
if row is None:
|
||||
row = PlaylistItem(playlist_id=pl.id, video_id=vid, position=0)
|
||||
db.add(row)
|
||||
else:
|
||||
row.playlist_id = pl.id
|
||||
rows.insert(min(at - left_out, len(rows)), row)
|
||||
present.add(vid)
|
||||
added += 1
|
||||
for pos, row in enumerate(rows):
|
||||
row.position = pos
|
||||
return added
|
||||
|
||||
|
||||
def restore_items(db: Session, pl: Playlist, video_ids: list[str], positions: list[int]) -> int:
|
||||
"""Put removed videos back at their places, in ONE request: an append plus a whole-list reorder
|
||||
leaves them at the end whenever the second call fails. Says how many came back."""
|
||||
lock_playlists(db, pl.id)
|
||||
added = _insert_at(db, pl, list(zip(positions, video_ids, strict=True)))
|
||||
if added:
|
||||
recompute_dirty(db, pl)
|
||||
touch(pl)
|
||||
db.flush()
|
||||
return added
|
||||
|
||||
|
||||
class ListGone(Exception):
|
||||
"""A list of a two-list write was deleted between the route's read and the lock."""
|
||||
|
||||
def __init__(self, *, source: bool):
|
||||
super().__init__("source" if source else "target")
|
||||
self.source = source
|
||||
|
||||
|
||||
def _lock_both(db: Session, src: Playlist, dst: Playlist) -> None:
|
||||
"""Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise. A list
|
||||
already gone is said here: a write into it would answer with its foreign key's 500."""
|
||||
found = lock_playlists(db, src.id, dst.id)
|
||||
if src.id not in found:
|
||||
raise ListGone(source=True)
|
||||
if dst.id not in found:
|
||||
raise ListGone(source=False)
|
||||
|
||||
|
||||
def move_items(
|
||||
db: Session, src: Playlist, dst: Playlist, video_ids: list[str]
|
||||
) -> tuple[list[str], list[str]]:
|
||||
"""Move these videos from `src` into `dst`, in the caller's one transaction, in the source's
|
||||
order whatever order they were picked in. Returns the ones that landed in the target and the ones
|
||||
it already held, which only left the source: the Undo needs to tell them apart. A video a
|
||||
concurrent edit already took out of the source is in neither. Raises `MoveRefused` for the same
|
||||
list twice."""
|
||||
if src.id == dst.id:
|
||||
raise MoveRefused("same_playlist")
|
||||
# Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise.
|
||||
lock_playlists(db, src.id, dst.id)
|
||||
row = queries.item(db, src.id, video_id)
|
||||
if row is None:
|
||||
raise MoveRefused("not_in_playlist")
|
||||
dst_changed = False
|
||||
if queries.item(db, dst.id, video_id) is None:
|
||||
# Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
|
||||
# survives the move, and there is no window where the video belongs to neither list. The
|
||||
# check above holds: both lists are locked, and an insert into the target (another tab's
|
||||
# add, a YouTube merge) waits for the lock — its foreign key needs the target row.
|
||||
row.playlist_id = dst.id
|
||||
row.position = next_item_position(db, dst.id)
|
||||
dst_changed = True
|
||||
if not dst_changed:
|
||||
db.delete(row)
|
||||
# The source always changed; the target only when the row landed in it. Touching a target
|
||||
# that already had the video promoted an untouched list in the rail's "recently changed" sort.
|
||||
recompute_dirty(db, src)
|
||||
touch(src)
|
||||
if dst_changed:
|
||||
_lock_both(db, src, dst)
|
||||
wanted = set(video_ids)
|
||||
rows = [it for it in queries.items(db, src.id) if it.video_id in wanted]
|
||||
held = queries.in_playlist(db, dst.id, [r.video_id for r in rows])
|
||||
position = next_item_position(db, dst.id)
|
||||
moved: list[str] = []
|
||||
already_there: list[str] = []
|
||||
for row in rows:
|
||||
if row.video_id in held:
|
||||
already_there.append(row.video_id)
|
||||
db.delete(row)
|
||||
else:
|
||||
moved.append(row.video_id)
|
||||
# Re-parented rather than deleted and inserted: `added_at` ("when I saved this") survives
|
||||
# the move, and there is no window where the video belongs to neither list. `held` stays
|
||||
# true: both lists are locked, and an insert into the target (another tab's add, a
|
||||
# YouTube merge) waits for the lock, as its foreign key needs the target row.
|
||||
row.playlist_id = dst.id
|
||||
row.position = position
|
||||
position += 1
|
||||
if rows:
|
||||
recompute_dirty(db, src)
|
||||
touch(src)
|
||||
# Only when a row landed in it: touching a target that already had the video promoted an untouched
|
||||
# list in the rail's "recently changed" sort.
|
||||
if moved:
|
||||
recompute_dirty(db, dst)
|
||||
touch(dst)
|
||||
db.flush()
|
||||
return moved, already_there
|
||||
|
||||
|
||||
def undo_move(
|
||||
db: Session,
|
||||
src: Playlist,
|
||||
dst: Playlist,
|
||||
moved: list[str],
|
||||
already_there: list[str],
|
||||
positions: list[int],
|
||||
) -> int:
|
||||
"""Take a move back in one transaction: the `moved` videos leave `dst` and return to their places
|
||||
in `src`, keeping their rows, as the move did. The `already_there` ones return too and stay in
|
||||
`dst`, which held them before. A moved video `dst` no longer holds has gone on from there, and
|
||||
bringing it back would leave it in lists the reader never put it in, so it stays out, as a
|
||||
move skips a video a concurrent edit took out. `positions` pairs with `moved` + `already_there`.
|
||||
Says how many came back. Raises `MoveRefused` for the same list twice, as a move does."""
|
||||
if src.id == dst.id:
|
||||
raise MoveRefused("same_playlist")
|
||||
_lock_both(db, src, dst)
|
||||
wanted = set(moved)
|
||||
rows = {it.video_id: it for it in queries.items(db, dst.id) if it.video_id in wanted}
|
||||
changed = bool(rows)
|
||||
placed = list(zip(positions, [*moved, *already_there], strict=True))
|
||||
back = _insert_at(db, src, placed, leaving=rows, staying_out=frozenset(wanted - rows.keys()))
|
||||
# What `src` holds again by now only leaves `dst`.
|
||||
for row in rows.values():
|
||||
db.delete(row)
|
||||
db.flush()
|
||||
if changed:
|
||||
recompute_dirty(db, dst)
|
||||
touch(dst)
|
||||
if back:
|
||||
recompute_dirty(db, src)
|
||||
touch(src)
|
||||
db.flush()
|
||||
return back
|
||||
|
||||
|
||||
def reorder_items(db: Session, pl: Playlist, order: list[str]) -> int:
|
||||
|
||||
+20
-10
@@ -1,29 +1,39 @@
|
||||
"""A request the app refuses, said in a way the UI can put in the reader's language.
|
||||
|
||||
Every refusal the API answers carries a structured `detail` — `{code, reason, …params}` — and the client
|
||||
translates it (`lib/api/core.ts` `localizeDetail`). A bare English string reached the Hungarian UI
|
||||
translates it (`lib/api/refusalText.ts` `localizeDetail`). A bare English string reached the Hungarian UI
|
||||
verbatim: the blocking error dialog, and every inline surface that printed `HttpError.detail`, showed
|
||||
whatever sentence the route happened to hand-write. `refused` is the generic family
|
||||
(`errors.refused.<reason>`, the params interpolated into the sentence); the older specific codes
|
||||
(`not_found`, `youtube`, `playlist`, the admin refusals, …) stay, since each is already translated.
|
||||
|
||||
A test (`test_http_details.py`) reads every `HTTPException` in the app and fails on a string detail,
|
||||
and the frontend's contract test requires every `refused(…, "<reason>")` to be named in both
|
||||
languages — so the rule is enforced, not described.
|
||||
A test (`test_http_details.py`) reads every place an `HTTPException` or a `JSONResponse` sends a detail
|
||||
and fails on one the client cannot say: a sentence, a dict without a `code`, a variable, none at all.
|
||||
`tests/refusal_sites.py` reads every `{"code": …}` detail from the syntax tree, written where it is sent
|
||||
(or returned by a factory that only ever is), with any `**` spread before its `code` and `reason`, and
|
||||
follows each reason along a closed list of shapes: a literal, a `Literal` type on the parameter or
|
||||
attribute it travels in, every call of a helper like `refused` below, a map's values; anything else is
|
||||
reported, never guessed. The frontend's contract test
|
||||
(`i18n/backendIds.contract.test.ts`) runs it and requires each code in one record of
|
||||
`lib/api/refusalCodes.ts`, and each reason's sentence in both languages; a code or reason it cannot
|
||||
follow fails it, unless no screen shows that code. So the rule is enforced, not described.
|
||||
"""
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app.sysconfig import InvalidSetting, SettingReason
|
||||
|
||||
|
||||
def refused(status: int, reason: str, **params: object) -> HTTPException:
|
||||
"""The refusal to raise: `raise refused(409, "view_name_taken", name=name)`. `params` travel in
|
||||
the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see."""
|
||||
return HTTPException(status_code=status, detail={"code": "refused", "reason": reason, **params})
|
||||
the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see.
|
||||
They are spread first, so a `code` or `reason` among them could never replace these two."""
|
||||
return HTTPException(status_code=status, detail={**params, "code": "refused", "reason": reason})
|
||||
|
||||
|
||||
# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the frontend's
|
||||
# contract test can read every reason this can produce.
|
||||
_SETTING_REASONS = {
|
||||
# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the refusal
|
||||
# scan reads every reason this can produce, and holds its keys to `SettingReason`.
|
||||
_SETTING_REASONS: dict[SettingReason, str] = {
|
||||
"not_a_number": "setting_not_a_number",
|
||||
"too_low": "setting_too_low",
|
||||
"too_high": "setting_too_high",
|
||||
@@ -32,6 +42,6 @@ _SETTING_REASONS = {
|
||||
}
|
||||
|
||||
|
||||
def setting_refused(exc) -> HTTPException:
|
||||
def setting_refused(exc: InvalidSetting) -> HTTPException:
|
||||
"""A value the admin entered that `sysconfig.set_value` refused (`exc`: its `InvalidSetting`)."""
|
||||
return refused(400, _SETTING_REASONS[exc.reason], key=exc.key, **exc.params)
|
||||
|
||||
@@ -6,9 +6,9 @@ and "fetch a row this user owns or 404" idioms have one definition, one status c
|
||||
one existence-leak policy.
|
||||
"""
|
||||
|
||||
from typing import Any, Protocol, TypeVar
|
||||
from typing import Annotated, Any, Protocol, TypeVar
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
from fastapi import HTTPException, Query, Request
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -21,6 +21,10 @@ from app.youtube.client import YouTubeError, refusal_reason
|
||||
|
||||
M = TypeVar("M", bound=Base)
|
||||
|
||||
# A shuffle / surprise seed off the query string. The client draws one under a billion; bounded so a
|
||||
# hand-typed 400-digit one is a 422, not an int no float or bigint can hold further down.
|
||||
Seed = Annotated[int, Query(ge=-(2**53), le=2**53)]
|
||||
|
||||
# Resume-position thresholds of a player that only keeps a resume point (the YouTube player, the
|
||||
# library's): positions below this are "didn't really start" and within this of the end are
|
||||
# "basically finished" — neither is worth storing, so the position is cleared instead (no progress
|
||||
|
||||
+19
-10
@@ -976,7 +976,16 @@ def _resend_verify_if_eligible(email: str) -> None:
|
||||
send_verification_link(db, user)
|
||||
|
||||
|
||||
def _login_refused(status: int, reason: str, user: User | None = None, **extra) -> HTTPException:
|
||||
# Every refusal a password sign-in answers with; the UI names each one (`welcome.auth.blocked.<reason>`,
|
||||
# held by the frontend's backendIds contract, which reads this Literal).
|
||||
LoginRefusal = Literal[
|
||||
"rate_limited", "invalid", "suspended", "email_unverified", "pending_approval"
|
||||
]
|
||||
|
||||
|
||||
def _login_refused(
|
||||
status: int, reason: LoginRefusal, user: User | None = None, **extra
|
||||
) -> HTTPException:
|
||||
"""Every refusal password_login can answer with, in the codebase's structured `{code, reason}`
|
||||
shape so the SPA can say it in the user's own language (the old bare English strings were
|
||||
unreadable under a bilingual UI, and unusable as anything but a one-line error).
|
||||
@@ -992,15 +1001,15 @@ def _login_refused(status: int, reason: str, user: User | None = None, **extra)
|
||||
`email_verified` is false, and vice versa — the SPA renders whichever is actually outstanding
|
||||
rather than assuming a sequence.
|
||||
"""
|
||||
detail: dict = {"code": "login", "reason": reason}
|
||||
if user is not None:
|
||||
detail |= {
|
||||
"email": user.email,
|
||||
"email_verified": user.email_verified,
|
||||
"approved": user.is_active,
|
||||
}
|
||||
detail |= {k: v for k, v in extra.items() if v is not None}
|
||||
return HTTPException(status_code=status, detail=detail)
|
||||
account = (
|
||||
{"email": user.email, "email_verified": user.email_verified, "approved": user.is_active}
|
||||
if user is not None
|
||||
else {}
|
||||
)
|
||||
given = {k: v for k, v in extra.items() if v is not None}
|
||||
return HTTPException(
|
||||
status_code=status, detail={**account, **given, "code": "login", "reason": reason}
|
||||
)
|
||||
|
||||
|
||||
@router.post("/password-login")
|
||||
|
||||
@@ -76,6 +76,10 @@ def list_channels(user: User = Depends(current_user), db: Session = Depends(get_
|
||||
"details_synced": ch.details_synced_at is not None,
|
||||
"recent_synced": ch.recent_synced_at is not None,
|
||||
"backfill_done": ch.backfill_done,
|
||||
# When the follow reached this list: written by the subscription import the run that
|
||||
# first saw it, or by a subscribe made here — not the channel row's own date, which an
|
||||
# explored or discovered channel had before it was followed.
|
||||
"added_at": iso(sub.created_at),
|
||||
}
|
||||
for sub, ch in rows
|
||||
]
|
||||
|
||||
@@ -23,7 +23,7 @@ from app.discover import pins, rails
|
||||
from app.discover.budget import Budget, DEFAULT_BUDGET
|
||||
from app.models import User
|
||||
from app.plex.cards import norm_lang
|
||||
from app.routes._common import not_found
|
||||
from app.routes._common import Seed, not_found
|
||||
|
||||
router = APIRouter(prefix="/api/discover", tags=["discover"])
|
||||
|
||||
@@ -44,7 +44,7 @@ class ReorderIn(BaseModel):
|
||||
@router.get("")
|
||||
def get_discover(
|
||||
budget: Budget = DEFAULT_BUDGET,
|
||||
seed: int = 0,
|
||||
seed: Seed = 0,
|
||||
lang: str = "en",
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
@@ -79,7 +79,7 @@ MAX_EXCLUDE = 32
|
||||
@router.get("/pick")
|
||||
def get_pick(
|
||||
budget: Budget = DEFAULT_BUDGET,
|
||||
seed: int = 0,
|
||||
seed: Seed = 0,
|
||||
exclude: str = "",
|
||||
start: int = 0,
|
||||
lang: str = "en",
|
||||
|
||||
@@ -208,7 +208,7 @@ def set_credential(
|
||||
try:
|
||||
cleaned = dlcreds.validate_cookies(body.cookies)
|
||||
except dlcreds.InvalidCookies as e:
|
||||
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)}) from e
|
||||
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": e.reason}) from e
|
||||
dlcreds.upsert_credential(db, user.id, host_key, cleaned)
|
||||
audit.record(
|
||||
db,
|
||||
@@ -318,7 +318,9 @@ def enqueue_edit(
|
||||
except quota.QuotaExceeded as e:
|
||||
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
|
||||
except service.EditError as e:
|
||||
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason}) from e
|
||||
raise HTTPException(
|
||||
status_code=400, detail={**e.params, "code": "edit", "reason": e.reason}
|
||||
) from e
|
||||
db.commit()
|
||||
return serialize_job(db, job)
|
||||
|
||||
@@ -341,6 +343,8 @@ class EnqueueClipIn(BaseModel):
|
||||
audio_track: int | None = None
|
||||
# A crop frame (downloads/clip): fractions of the box it was drawn in, and that box's aspect.
|
||||
crop: ClipCropIn | None = None
|
||||
# A GIF of this long side (downloads/gif) instead of a video; None is the video clip.
|
||||
gif_side: int | None = None
|
||||
display_name: str | None = None
|
||||
|
||||
|
||||
@@ -353,14 +357,15 @@ def enqueue_clip(
|
||||
try:
|
||||
crop = body.crop.model_dump() if body.crop else None
|
||||
spec = clipmod.normalize(
|
||||
body.source, body.ref, body.start_s, body.end_s, body.audio_track, crop
|
||||
body.source, body.ref, body.start_s, body.end_s, body.audio_track, crop, body.gif_side
|
||||
)
|
||||
display_name = display_text(body.display_name)
|
||||
job = service.enqueue_clip(db, user.id, spec, display_name=display_name)
|
||||
except clipmod.ClipError as e:
|
||||
status = 404 if e.reason == "not_found" else 422
|
||||
detail = {"code": "clip", "reason": e.reason, **e.params}
|
||||
raise HTTPException(status_code=status, detail=detail) from e
|
||||
raise HTTPException(
|
||||
status_code=status, detail={**e.params, "code": "clip", "reason": e.reason}
|
||||
) from e
|
||||
except quota.QuotaExceeded as e:
|
||||
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
|
||||
db.commit()
|
||||
@@ -610,7 +615,7 @@ def list_links(
|
||||
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
) -> list[dict]:
|
||||
own_job(db, user, job_id)
|
||||
return [linksmod.owner_view(link) for link in queries.links_to(db, job_id=job_id)]
|
||||
return [linksmod.owner_view(db, link) for link in queries.links_to(db, job_id=job_id)]
|
||||
|
||||
|
||||
class CreateLinkIn(BaseModel):
|
||||
@@ -652,7 +657,7 @@ def create_link(
|
||||
raise refused(409, "share_unfinished")
|
||||
link, token = _new_link(db, user, body, job_id=job_id)
|
||||
db.commit()
|
||||
return linksmod.owner_view(link, token=token)
|
||||
return linksmod.owner_view(db, link, token=token)
|
||||
|
||||
|
||||
def _shareable_job(db: Session, user: User, job_id: int) -> "DownloadJob":
|
||||
@@ -683,7 +688,7 @@ def list_folder_links(
|
||||
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
) -> list[dict]:
|
||||
own_folder(db, user, folder_id)
|
||||
return [linksmod.owner_view(link) for link in queries.links_to(db, folder_id=folder_id)]
|
||||
return [linksmod.owner_view(db, link) for link in queries.links_to(db, folder_id=folder_id)]
|
||||
|
||||
|
||||
@router.post("/folders/{folder_id}/links")
|
||||
@@ -696,7 +701,7 @@ def create_folder_link(
|
||||
_shareable_folder(db, user, folder_id)
|
||||
link, token = _new_link(db, user, body, folder_id=folder_id)
|
||||
db.commit()
|
||||
return linksmod.owner_view(link, token=token)
|
||||
return linksmod.owner_view(db, link, token=token)
|
||||
|
||||
|
||||
class UpdateLinkIn(BaseModel):
|
||||
@@ -723,7 +728,7 @@ def update_link(
|
||||
sent=frozenset(body.model_fields_set),
|
||||
)
|
||||
db.commit()
|
||||
return linksmod.owner_view(link)
|
||||
return linksmod.owner_view(db, link)
|
||||
|
||||
|
||||
@router.delete("/links/{link_id}")
|
||||
|
||||
+13
-12
@@ -10,6 +10,7 @@ from sqlalchemy.orm.exc import StaleDataError
|
||||
|
||||
from app import jobruns, quota, scheduler
|
||||
from app.catalog import queries as catalog_queries, service as catalog_service
|
||||
from app.catalog.models import known_length
|
||||
from app.auth import current_user
|
||||
from app.db import get_db, release_read_connection
|
||||
from app.mailtext import lang_for_user
|
||||
@@ -19,6 +20,7 @@ from app.models import User
|
||||
from app.reader import queries, service, takeout
|
||||
from app.routes._common import (
|
||||
ProgressIn,
|
||||
Seed,
|
||||
capped_body,
|
||||
drain_body,
|
||||
kept_position,
|
||||
@@ -88,7 +90,7 @@ def _feed_params(
|
||||
def get_feed(
|
||||
params: dict = Depends(_feed_params),
|
||||
sort: str = "newest",
|
||||
seed: int = 0,
|
||||
seed: Seed = 0,
|
||||
limit: int = Query(default=60, le=200),
|
||||
cursor: str | None = None,
|
||||
user: User = Depends(current_user),
|
||||
@@ -204,22 +206,21 @@ def set_video_state(
|
||||
return {"ok": True, "video_id": video_id, "status": status}
|
||||
|
||||
|
||||
@router.delete("/videos/{video_id}/state")
|
||||
def clear_video_state(
|
||||
@router.delete("/videos/{video_id}/progress")
|
||||
def clear_video_progress(
|
||||
video_id: str,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Reset a video to pristine for this user — drop the whole VideoState row (status,
|
||||
watch position and watched_at), as if it had never been opened. Saved/playlist membership
|
||||
lives elsewhere and is untouched."""
|
||||
"""Forget where the user left a video: its resume point goes, its status (watched, hidden) stays.
|
||||
Saved/playlist membership lives elsewhere and is untouched."""
|
||||
try:
|
||||
service.clear_state(db, user, video_id)
|
||||
service.set_progress(db, user, video_id, 0)
|
||||
db.commit()
|
||||
# silent-ok: the row went under the delete, which is the outcome the delete wanted.
|
||||
# silent-ok: the row went under the write, which is the outcome the write wanted.
|
||||
except StaleDataError:
|
||||
db.rollback()
|
||||
return {"ok": True, "video_id": video_id, "status": "new", "position_seconds": 0}
|
||||
return {"ok": True, "video_id": video_id, "position_seconds": 0}
|
||||
|
||||
|
||||
@router.post("/videos/{video_id}/progress")
|
||||
@@ -288,7 +289,7 @@ def get_video_detail(
|
||||
"channel_title": v.channel.title if v.channel else None,
|
||||
"published_at": iso(v.published_at),
|
||||
"view_count": v.view_count,
|
||||
"duration_seconds": v.duration_seconds,
|
||||
"duration_seconds": v.length_seconds,
|
||||
}
|
||||
|
||||
# The DB path above is free; the off-catalog YouTube lookup below spends a quota unit.
|
||||
@@ -316,8 +317,8 @@ def get_video_detail(
|
||||
"channel_title": snippet.get("channelTitle"),
|
||||
"published_at": snippet.get("publishedAt"),
|
||||
"view_count": int(views) if views is not None else None,
|
||||
"duration_seconds": parse_iso8601_duration(
|
||||
items[0].get("contentDetails", {}).get("duration")
|
||||
"duration_seconds": known_length(
|
||||
parse_iso8601_duration(items[0].get("contentDetails", {}).get("duration"))
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -139,7 +139,7 @@ def own_job_for_update(db: Session, user: User, job_id: int) -> "DownloadJob":
|
||||
|
||||
def quota_detail(e: quota.QuotaExceeded) -> dict:
|
||||
"""Structured error the trilingual client localizes (reason key + numbers), instead of a
|
||||
pre-baked English string with a dot-decimal GB value. See lib/api.ts localizeDetail()."""
|
||||
pre-baked English string with a dot-decimal GB value. See lib/api/refusalText.ts localizeDetail()."""
|
||||
return {"code": "quota", "reason": e.reason, "limit": e.limit, "current": e.current}
|
||||
|
||||
|
||||
@@ -715,13 +715,8 @@ def get_file(
|
||||
return response
|
||||
|
||||
|
||||
# A preview's own headers. `sandbox` leaves the answer no script and an opaque origin, whatever the
|
||||
# file turns out to hold; `frame-ancestors 'self'` lets the app's viewer frame a PDF (every other
|
||||
# answer keeps the app-wide 'none', main.py), with X-Frame-Options saying the same for old engines.
|
||||
PREVIEW_HEADERS = {
|
||||
"Content-Security-Policy": "sandbox; frame-ancestors 'self'",
|
||||
"X-Frame-Options": "SAMEORIGIN",
|
||||
}
|
||||
# A preview's own headers, shared with a share link's picture (`preview.SANDBOX_HEADERS`).
|
||||
PREVIEW_HEADERS = preview.SANDBOX_HEADERS
|
||||
# Set on a text preview cut at `preview.TEXT_PREVIEW_BYTES`, so the viewer can say it shows the start.
|
||||
TRUNCATED_HEADER = "X-Preview-Truncated"
|
||||
|
||||
|
||||
+177
-49
@@ -3,10 +3,10 @@ CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-w
|
||||
mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
|
||||
out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, model_validator
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import quota
|
||||
@@ -41,11 +41,16 @@ router = APIRouter(prefix="/api/playlists", tags=["playlists"])
|
||||
# It guards EVERY route that spends under QuotaAction.PLAYLISTS_PULL, not just the bulk one: a
|
||||
# scripted loop over the single-playlist re-pull drains the same shared budget, and limiting one of
|
||||
# two routes with the same quota action is the asymmetric fix this codebase keeps re-learning.
|
||||
# Deliberately NOT on the push side (`push-plan` / `push`): those run on every push attempt,
|
||||
# including the no-op "already up to date" path, so a 3-per-10-minute allowance would refuse
|
||||
# legitimate work. Their exposure is tracked as SEC-18 instead.
|
||||
_playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600)
|
||||
|
||||
# SEC-18, the push side: `push-plan` reads the live list (1 unit a page) and `push` reads and writes
|
||||
# it, both against the same instance-wide budget. They get a budget of their own, counted in calls
|
||||
# and far wider than the pull's: a press in the UI is one plan and, unless nothing changed, one push,
|
||||
# so no person reaches it, while a scripted loop stops at it. A big push is not this limit's job —
|
||||
# `push` already refuses one the day's remaining budget cannot cover.
|
||||
PUSH_ALLOWANCE = 20
|
||||
_playlist_push_limiter = RateLimiter(max_events=PUSH_ALLOWANCE, window_seconds=600)
|
||||
|
||||
|
||||
def _check_pull_cooldown(user: User) -> None:
|
||||
"""Shared gate so the two PLAYLISTS_PULL routes cannot drift apart again."""
|
||||
@@ -53,7 +58,23 @@ def _check_pull_cooldown(user: User) -> None:
|
||||
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
|
||||
|
||||
|
||||
def _refused(status: int, reason: str) -> HTTPException:
|
||||
# Every playlist action the server refuses; the UI names each one (`playlists.errors.<reason>`, held by
|
||||
# the frontend's backendIds contract, which reads this Literal).
|
||||
PlaylistRefusal = Literal[
|
||||
"name_required",
|
||||
"not_on_youtube",
|
||||
"push_cooldown",
|
||||
"quota_exceeded",
|
||||
"sync_all_busy",
|
||||
"sync_busy",
|
||||
"watch_later_not_pushable",
|
||||
"watch_later_undeletable",
|
||||
"youtube_editing_off",
|
||||
"youtube_read_scope",
|
||||
]
|
||||
|
||||
|
||||
def _refused(status: int, reason: PlaylistRefusal) -> HTTPException:
|
||||
"""A playlist action the server refuses, as a reason the UI translates
|
||||
(`playlists.errors.<reason>`) — a bare English detail reached the Hungarian UI verbatim."""
|
||||
return HTTPException(status_code=status, detail={"code": "playlist", "reason": reason})
|
||||
@@ -74,10 +95,13 @@ def _video_or_404(db: Session, video_id: str) -> None:
|
||||
def _summary(
|
||||
pl: "Playlist",
|
||||
count: int,
|
||||
total_duration: int = 0,
|
||||
duration: tuple[int, int] = (0, 0),
|
||||
has_video: bool | None = None,
|
||||
cover: str | None = None,
|
||||
) -> dict:
|
||||
"""`duration` is `queries.durations`'s pair: the known running time, and the videos it left out
|
||||
for having no length — the "+N without length" beside it."""
|
||||
total_duration, left_out = duration
|
||||
out = {
|
||||
"id": pl.id,
|
||||
"name": pl.name,
|
||||
@@ -87,6 +111,7 @@ def _summary(
|
||||
"dirty": pl.dirty,
|
||||
"item_count": count,
|
||||
"total_duration_seconds": total_duration,
|
||||
"without_length": left_out,
|
||||
"cover_thumbnail": cover,
|
||||
# When this list last CHANGED (rename, add/remove/move/reorder) — the rail's
|
||||
# "recently changed first" sort reads it. See `service.touch` for why it isn't automatic.
|
||||
@@ -117,7 +142,7 @@ def list_playlists(
|
||||
_summary(
|
||||
p,
|
||||
counts.get(p.id, 0),
|
||||
durations.get(p.id, 0),
|
||||
durations.get(p.id, (0, 0)),
|
||||
(p.id in member) if contains else None,
|
||||
covers.get(p.id),
|
||||
)
|
||||
@@ -229,9 +254,16 @@ def revert_youtube(
|
||||
raise youtube_refusal(exc, playlist=True) from exc
|
||||
|
||||
|
||||
def _check_push_cooldown(user: User) -> None:
|
||||
"""Shared gate of the two push routes, so they spend one allowance and cannot drift apart."""
|
||||
if not _playlist_push_limiter.allow(str(user.id)):
|
||||
raise _refused(429, "push_cooldown")
|
||||
|
||||
|
||||
def _pushable(db: Session, user: User, playlist_id: int) -> "Playlist":
|
||||
"""A local playlist the user may push to YouTube (create or update). The built-in
|
||||
Watch later and read-only YouTube mirrors are never pushable."""
|
||||
"""A playlist the user may push to YouTube: a local one (created there on its first push) or a
|
||||
mirror of their own YouTube list (its local edits merged back). Only the built-in Watch later is
|
||||
never pushable."""
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
if pl.kind == "watch_later":
|
||||
raise _refused(400, "watch_later_not_pushable")
|
||||
@@ -248,6 +280,7 @@ def push_plan(
|
||||
the quota estimate, so the UI can show a confirm. For a linked playlist this reads the live
|
||||
YouTube state (cheap, 1 unit/page). Write scope is required — syncing is a YouTube write."""
|
||||
pl = _pushable(db, user, playlist_id)
|
||||
_check_push_cooldown(user) # after the refusal: a list that cannot be pushed spends nothing
|
||||
try:
|
||||
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
|
||||
plan = plan_merge(db, user, pl)
|
||||
@@ -273,6 +306,8 @@ def push(
|
||||
with sync_claim(db, pl.id) as free:
|
||||
if not free:
|
||||
raise _refused(409, "sync_busy")
|
||||
# Spent only by a push that runs, as Revert's pull: a busy list or Watch later costs nothing.
|
||||
_check_push_cooldown(user)
|
||||
try:
|
||||
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
|
||||
# Read the live playlist ONCE (linked playlists only) and share it with both the
|
||||
@@ -334,8 +369,8 @@ def rename_playlist(
|
||||
raise _refused(400, "name_required")
|
||||
service.rename_playlist(db, pl, name)
|
||||
db.commit()
|
||||
count, total, cover = queries.playlist_totals(db, pl.id)
|
||||
return _summary(pl, count, total, cover=cover)
|
||||
count, total, left_out, cover = queries.playlist_totals(db, pl.id)
|
||||
return _summary(pl, count, (total, left_out), cover=cover)
|
||||
|
||||
|
||||
@router.delete("/{playlist_id}")
|
||||
@@ -371,24 +406,17 @@ def delete_playlist(
|
||||
return {"deleted": playlist_id}
|
||||
|
||||
|
||||
class AddItemIn(VideoRefIn):
|
||||
# Where the video lands among the list's items, in their stored order (0 = first). Omitted, it
|
||||
# is appended. The list page's "Undo" of a removal puts the row back where it was with this —
|
||||
# ONE request, because append-then-reorder leaves it at the end whenever the second call fails.
|
||||
position: int | None = Field(default=None, ge=0)
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items")
|
||||
def add_item(
|
||||
playlist_id: int,
|
||||
body: AddItemIn,
|
||||
body: VideoRefIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
pl = _own_playlist(db, user, playlist_id, lock=True)
|
||||
video_id = body.video_id
|
||||
_video_or_404(db, video_id)
|
||||
service.add_to_playlist(db, pl, video_id, body.position)
|
||||
service.add_item(db, pl, video_id, mark_dirty=True)
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
|
||||
|
||||
@@ -400,58 +428,158 @@ def remove_item(
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
pl = _own_playlist(db, user, playlist_id, lock=True)
|
||||
service.remove_item(db, pl, video_id, mark_dirty=True)
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
|
||||
|
||||
|
||||
class MoveItemIn(BaseModel):
|
||||
class VideoIdsIn(BaseModel):
|
||||
# Required: with a default, a misnamed key answered 200 having acted on no video.
|
||||
video_ids: list[str]
|
||||
|
||||
|
||||
def _paired(video_count: int, positions: list[int]) -> None:
|
||||
"""One place per video, none before the start and none twice: both undo bodies come back from
|
||||
the client, and the insert's arithmetic counts on places that only rise."""
|
||||
if len(positions) != video_count:
|
||||
raise ValueError("positions must pair one-to-one with the videos")
|
||||
if any(at < 0 for at in positions):
|
||||
raise ValueError("positions cannot be negative")
|
||||
if len(set(positions)) != len(positions):
|
||||
raise ValueError("positions cannot repeat")
|
||||
|
||||
|
||||
class RestoreItemsIn(VideoIdsIn):
|
||||
# Where each video goes back: an index over the ordered rows, one per video.
|
||||
positions: list[int] = []
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _one_position_per_video(self):
|
||||
_paired(len(self.video_ids), self.positions)
|
||||
return self
|
||||
|
||||
|
||||
class MoveItemsIn(VideoIdsIn):
|
||||
target_playlist_id: int
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/{video_id}/move")
|
||||
def move_item(
|
||||
class UndoMoveIn(BaseModel):
|
||||
from_playlist_id: int
|
||||
# Required, as `video_ids` is. An empty one is still a real body: the target held them all.
|
||||
moved: list[str]
|
||||
already_there: list[str] = []
|
||||
# One per video of `moved` + `already_there`, in that order.
|
||||
positions: list[int]
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _one_position_per_video(self):
|
||||
_paired(len(self.moved) + len(self.already_there), self.positions)
|
||||
return self
|
||||
|
||||
|
||||
def _list_gone(e: service.ListGone) -> HTTPException:
|
||||
"""Each list of a two-list write under the reason the route's own read gives it."""
|
||||
return not_found("playlist") if e.source else not_found("move_target")
|
||||
|
||||
|
||||
def _move_refused(e: service.MoveRefused) -> HTTPException:
|
||||
return HTTPException(status_code=400, detail={"code": "playlist_move", "reason": e.reason})
|
||||
|
||||
|
||||
def _other_list(db: Session, user: User, playlist_id: int) -> "Playlist":
|
||||
"""The other list of a move. Its own reason: "this playlist doesn't exist any more" about a list
|
||||
that is gone was read as the one on screen, which is still there."""
|
||||
other = queries.own_playlist(db, user, playlist_id)
|
||||
if other is None:
|
||||
raise not_found("move_target")
|
||||
return other
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/remove-bulk")
|
||||
def remove_items(
|
||||
playlist_id: int,
|
||||
video_id: str,
|
||||
body: MoveItemIn,
|
||||
body: VideoIdsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Move one video from a playlist into another of the caller's playlists, atomically.
|
||||
"""Remove many videos from a playlist at once. Returns the ones that went."""
|
||||
pl = _own_playlist(db, user, playlist_id, lock=True)
|
||||
removed = service.remove_items(db, pl, body.video_ids)
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": pl.id, "removed": removed}
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/restore")
|
||||
def restore_items(
|
||||
playlist_id: int,
|
||||
body: RestoreItemsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""The "Undo" of a removal: the videos back at their places. Returns how many came back."""
|
||||
pl = _own_playlist(db, user, playlist_id, lock=True)
|
||||
added = service.restore_items(db, pl, body.video_ids, body.positions)
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": pl.id, "added": added}
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/move-bulk")
|
||||
def move_items(
|
||||
playlist_id: int,
|
||||
body: MoveItemsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Move videos into another of the caller's playlists, atomically.
|
||||
|
||||
ONE transaction on purpose. The obvious client-side shape — add to the target, then remove
|
||||
from the source — leaves the video in BOTH lists whenever the second call fails, which is
|
||||
exactly the kind of half-applied state this codebase treats as a bug rather than an edge
|
||||
case. Idempotent: if the video is already in the target, the source row is simply dropped
|
||||
(moving into a list that already has it is a removal, not a duplicate)."""
|
||||
from the source — leaves a video in BOTH lists whenever the second call fails. A video the
|
||||
target already has only leaves the source, and one a concurrent edit (another tab, a YouTube
|
||||
re-pull) already took out is skipped. Says which landed there and which it already held, for
|
||||
the "Undo"."""
|
||||
src = _own_playlist(db, user, playlist_id)
|
||||
# Its own reason: "this playlist doesn't exist any more" about a TARGET that is gone was read as
|
||||
# the list on screen — which is still there.
|
||||
dst = queries.own_playlist(db, user, body.target_playlist_id)
|
||||
if dst is None:
|
||||
raise not_found("move_target")
|
||||
dst = _other_list(db, user, body.target_playlist_id)
|
||||
try:
|
||||
service.move_item(db, src, dst, video_id)
|
||||
moved, already_there = service.move_items(db, src, dst, body.video_ids)
|
||||
except service.MoveRefused as e:
|
||||
# A same-list move is a bad request; a video a concurrent edit (another tab, a YouTube
|
||||
# re-pull) already took out is gone.
|
||||
status = 400 if e.reason == "same_playlist" else 404
|
||||
raise HTTPException(
|
||||
status_code=status, detail={"code": "playlist_move", "reason": e.reason}
|
||||
) from None
|
||||
raise _move_refused(e) from None
|
||||
except service.ListGone as e:
|
||||
raise _list_gone(e) from None
|
||||
db.commit()
|
||||
return {"ok": True, "from_playlist_id": src.id, "to_playlist_id": dst.id, "video_id": video_id}
|
||||
return {
|
||||
"ok": True,
|
||||
"from_playlist_id": src.id,
|
||||
"to_playlist_id": dst.id,
|
||||
"moved": moved,
|
||||
"already_there": already_there,
|
||||
}
|
||||
|
||||
|
||||
class ReorderItemsIn(BaseModel):
|
||||
video_ids: list[str] = []
|
||||
@router.post("/{playlist_id}/items/undo-move")
|
||||
def undo_move(
|
||||
playlist_id: int,
|
||||
body: UndoMoveIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""The "Undo" of a move, back into this playlist in one transaction. Returns how many came back."""
|
||||
src = _own_playlist(db, user, playlist_id)
|
||||
dst = _other_list(db, user, body.from_playlist_id)
|
||||
try:
|
||||
added = service.undo_move(db, src, dst, body.moved, body.already_there, body.positions)
|
||||
except service.MoveRefused as e:
|
||||
raise _move_refused(e) from None
|
||||
except service.ListGone as e:
|
||||
raise _list_gone(e) from None
|
||||
db.commit()
|
||||
return {"ok": True, "playlist_id": src.id, "added": added}
|
||||
|
||||
|
||||
@router.put("/{playlist_id}/order")
|
||||
def reorder_items(
|
||||
playlist_id: int,
|
||||
body: ReorderItemsIn,
|
||||
body: VideoIdsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
|
||||
@@ -432,6 +432,7 @@ def item_detail(
|
||||
"summary": (it.summary_hu or it.summary) if hu else it.summary,
|
||||
"year": it.year,
|
||||
"duration_seconds": it.duration_s,
|
||||
"size_bytes": it.size_bytes,
|
||||
"playable": it.playable,
|
||||
# Playback-class detail: HW/SW transcode split + the codec/bit-depth the info-page chip shows.
|
||||
"transcode_hw": it.transcode_hw,
|
||||
|
||||
@@ -323,13 +323,14 @@ def item_progress(
|
||||
`_should_push_resume`). `final` (sent on pause / seek / pagehide / unmount) pushes immediately and
|
||||
resets the throttle window, so Plex sees a settled position the instant the user pauses or seeks."""
|
||||
it = item_or_404(db, rating_key)
|
||||
# A player parked mid-item — playing or PAUSED — checkpoints every 10 s, so this is the signal
|
||||
# A player parked mid-item — playing or PAUSED — keeps checkpointing (every 10 s, once a minute
|
||||
# in a hidden silent page), so this is the signal
|
||||
# that such a viewer still wants their HLS session: the one thing `last_access` could not see,
|
||||
# since it moved only on a file fetch and a paused player fetches nothing (see
|
||||
# stream.reap_idle). It is NOT every open player: `usePlexProgress.flush` sends nothing once the
|
||||
# finish latch is up, or before the first position is known. Those are the states in which
|
||||
# reaping is the right answer anyway, and a viewer who scrubs back into a reaped session gets
|
||||
# the same 404 → reclaim the player already handles.
|
||||
# the same 410 → reclaim the player already handles.
|
||||
plex_stream.touch_session(user.id, str(rating_key))
|
||||
position = max(0, int(body.position_seconds))
|
||||
duration = int(body.duration_seconds)
|
||||
@@ -609,7 +610,9 @@ def stream_hls_file(
|
||||
is per (key, viewer, offset, tag) — so resolving by the key alone handed the old player the new
|
||||
session's `seg_12.ts`, produced from a different offset under `-copyts`: wrong
|
||||
bytes with a 200, which the player has no way to notice. A request for a
|
||||
session that is no longer the registered one is a 404, and the player reclaims at the playhead.
|
||||
session that is no longer the registered one is a 410, and the player reclaims at the playhead.
|
||||
410, not 404: a 404 here means "that segment is not there yet", worth hls.js's retries, and the
|
||||
player can only read the status (a gone session retried for half a minute behind that 404).
|
||||
|
||||
That reclaim is also why a request with no live session is no longer auto-started here: the
|
||||
auto-start could only ever begin at 0, silently throwing away the viewer's position."""
|
||||
@@ -625,7 +628,7 @@ def stream_hls_file(
|
||||
raise not_found("file")
|
||||
s = plex_stream.current_session(user.id, str(rating_key))
|
||||
if s is None or s.id != session_id:
|
||||
raise refused(404, "session_gone")
|
||||
raise refused(410, "session_gone")
|
||||
path = s.dir / filename
|
||||
# Only a name THIS session can actually produce is worth waiting for: its own playlists, and a
|
||||
# segment of one of its own renditions whose predecessor is already on disk (ffmpeg writes them
|
||||
@@ -648,13 +651,13 @@ def stream_hls_file(
|
||||
except OSError:
|
||||
# The session was torn down between the wait and the read — teardown moves the whole
|
||||
# directory aside and deletes it, and nothing holds `_lock` across those two steps. It has
|
||||
# to present as the same 404 a gone session does: the player's recovery is keyed on the
|
||||
# STATUS (`code === 404`), so
|
||||
# to present as the same 410 a gone session does: the player's recovery is keyed on the
|
||||
# STATUS, so
|
||||
# a 500 here skipped the reclaim and ended playback with "couldn't play" instead of
|
||||
# restarting at the playhead. Ordinary trigger: `loadSession` awaits the new session — which
|
||||
# kills and rmtrees the old one — before destroying the old hls instance, so the outgoing
|
||||
# instance's in-flight segment request races the rmtree on every seek-restart.
|
||||
raise refused(404, "session_gone") from None
|
||||
raise refused(410, "session_gone") from None
|
||||
return Response(
|
||||
content=content,
|
||||
media_type=_HLS_CT[ext],
|
||||
|
||||
@@ -183,7 +183,9 @@ def playlist_remove_item(
|
||||
|
||||
|
||||
class PlexItemKeysIn(BaseModel):
|
||||
item_rating_keys: list[str] = []
|
||||
# Required: with a default, a misnamed key answered 200 having acted on no item, and a reorder
|
||||
# took it for an empty order.
|
||||
item_rating_keys: list[str]
|
||||
|
||||
|
||||
class PlexItemAddBulkIn(PlexItemKeysIn):
|
||||
|
||||
@@ -16,7 +16,7 @@ from sqlalchemy.orm import Session
|
||||
from app.config import settings
|
||||
from app.db import get_db
|
||||
from app.downloads import links as linksmod
|
||||
from app.downloads import queries, storage
|
||||
from app.downloads import preview, queries, storage
|
||||
from app.downloads.status import JobStatus
|
||||
from app.ratelimit import RateLimiter, client_key
|
||||
from app.security import verify_password
|
||||
@@ -176,8 +176,15 @@ def watch_meta(token: str, db: Session = Depends(get_db)) -> dict:
|
||||
returns only `needs_password` until the viewer unlocks it (no title/thumbnail leak)."""
|
||||
link = _link_or_404(db, token)
|
||||
if link.password_hash:
|
||||
# Which kind it opens, for the password prompt's title — never what is in it.
|
||||
return {"needs_password": True, "kind": "folder" if link.folder_id is not None else "file"}
|
||||
# Which kind it opens, for the password prompt's words — never what is in it. `video` tells
|
||||
# a video (watched) from any other file (opened); the address is only a hint, every
|
||||
# prefix opens every link.
|
||||
opens = linksmod.opens(db, link)
|
||||
return {
|
||||
"needs_password": True,
|
||||
"kind": "folder" if opens == "folder" else "file",
|
||||
"video": opens == "video",
|
||||
}
|
||||
meta = _opened(db, link, token)
|
||||
db.commit()
|
||||
return meta
|
||||
@@ -219,8 +226,9 @@ def _granted(link: "DownloadLink", token: str, g: str | None) -> None:
|
||||
|
||||
|
||||
def _serve(link: "DownloadLink", asset: "MediaAsset") -> FileResponse:
|
||||
"""Range-aware stream of a shared file. Inline by default (plays in the page); an
|
||||
`allow_download` link serves it as an attachment."""
|
||||
"""Range-aware stream of a shared file. Inline by default (a video plays and a picture shows in
|
||||
the page, and a chat's link preview can fetch the picture); an `allow_download` link serves it
|
||||
as an attachment."""
|
||||
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
|
||||
if path is None:
|
||||
raise refused(404, "video_gone")
|
||||
@@ -234,6 +242,17 @@ def _serve(link: "DownloadLink", asset: "MediaAsset") -> FileResponse:
|
||||
# served with its real audio/video type, inline unless the link forces a download so it can play
|
||||
# in the watch page. Both are range-aware (FileResponse → 206) for seeking/resume.
|
||||
media_type = storage.served_media_type(asset.media_kind, asset.container)
|
||||
disposition = "attachment" if link.allow_download else "inline"
|
||||
if media_type is None and preview.kind(asset.media_kind, asset.container) == "image":
|
||||
# A picture (a GIF made from a clip, an uploaded photo) is shown as itself: its type from the
|
||||
# preview allow-list, under the preview's sandbox, so a mislabelled one still runs nothing.
|
||||
return FileResponse(
|
||||
path,
|
||||
filename=filename,
|
||||
media_type=preview.media_type("image", asset.container),
|
||||
content_disposition_type=disposition,
|
||||
headers=preview.SANDBOX_HEADERS,
|
||||
)
|
||||
if media_type is None:
|
||||
return FileResponse(
|
||||
path,
|
||||
@@ -241,7 +260,6 @@ def _serve(link: "DownloadLink", asset: "MediaAsset") -> FileResponse:
|
||||
media_type="application/octet-stream",
|
||||
content_disposition_type="attachment",
|
||||
)
|
||||
disposition = "attachment" if link.allow_download else "inline"
|
||||
return FileResponse(
|
||||
path, filename=filename, media_type=media_type, content_disposition_type=disposition
|
||||
)
|
||||
|
||||
@@ -128,7 +128,8 @@ def update_view(
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
raise refused(409, "view_name_taken", name=view.name) from None
|
||||
# The name asked for: the rollback has put the view's old name back on `view`.
|
||||
raise refused(409, "view_name_taken", name=name) from None
|
||||
return _serialize(view)
|
||||
|
||||
|
||||
|
||||
@@ -314,7 +314,8 @@ def search_youtube(
|
||||
|
||||
|
||||
class ClearSearchIn(BaseModel):
|
||||
video_ids: list[str] = []
|
||||
# Required: a default read a misnamed key as an empty list, a 200 that cleared nothing.
|
||||
video_ids: list[str]
|
||||
|
||||
|
||||
@router.post("/clear")
|
||||
|
||||
@@ -48,8 +48,9 @@ def list_tags(user: User = Depends(current_user), db: Session = Depends(get_db))
|
||||
"category": tag.category,
|
||||
"system": tag.user_id is None,
|
||||
"channel_count": count,
|
||||
"subscribed_count": subscribed,
|
||||
}
|
||||
for tag, count in queries.tags_with_counts(db, user)
|
||||
for tag, count, subscribed in queries.tags_with_counts(db, user)
|
||||
]
|
||||
|
||||
|
||||
@@ -107,7 +108,8 @@ def update_tag(
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
raise refused(409, "tag_name_taken", name=tag.name) from None
|
||||
# The name asked for: the rollback has put the tag's old name back on `tag`.
|
||||
raise refused(409, "tag_name_taken", name=name) from None
|
||||
return {
|
||||
"ok": True,
|
||||
"id": tag.id,
|
||||
|
||||
@@ -443,15 +443,20 @@ def _default(s: ConfigSpec):
|
||||
return getattr(settings, s.default_attr)
|
||||
|
||||
|
||||
# Every reason `set_value` refuses with. `refusal.setting_refused` looks each one up in its map, which
|
||||
# the refusal scan holds to exactly these keys: a reason the map lacks would answer a 500.
|
||||
SettingReason = Literal["not_a_number", "too_low", "too_high", "not_a_choice", "no_encryption_key"]
|
||||
|
||||
|
||||
class InvalidSetting(ValueError):
|
||||
"""A value `set_value` refuses, as a reason the UI translates (`errors.refused.setting_<reason>`)
|
||||
plus what the sentence needs — the English message is for logs only. A ValueError still, so a
|
||||
caller that only needs "refused or not" keeps working."""
|
||||
|
||||
def __init__(self, key: str, reason: str, **params: object):
|
||||
def __init__(self, key: str, reason: SettingReason, **params: object):
|
||||
super().__init__(f"{key}: {reason} {params}")
|
||||
self.key = key
|
||||
self.reason = reason
|
||||
self.reason: SettingReason = reason
|
||||
self.params = params
|
||||
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
# `pip install -r requirements.lock && pip check` must pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177); yt-dlp 2026.7.4 -> 2026.8.19
|
||||
# (it marks the clients that need a stream PO token, so the downloader stops fetching into a 403);
|
||||
# certifi 2026.7.22 -> 2026.10.19 (GHSA-6q48-6c7w-9gw2); brotli 1.2.0 -> 1.2.1 (PYSEC-2026-4322);
|
||||
# Authlib 1.7.2 -> 1.9.1 (CVE-2026-96760, CVE-2026-114616).
|
||||
# Authlib 1.7.2 -> 1.9.1 (CVE-2026-96760, CVE-2026-114616); joserfc 1.7.4 -> 1.7.6 (CVE-2026-119283, CVE-2026-119284).
|
||||
APScheduler==3.11.3
|
||||
Authlib==1.9.1
|
||||
Mako==1.4.1
|
||||
@@ -39,7 +39,7 @@ httptools==0.8.0
|
||||
httpx==0.28.1
|
||||
idna==3.18
|
||||
itsdangerous==2.2.0
|
||||
joserfc==1.7.4
|
||||
joserfc==1.7.6
|
||||
markdown-it-py==4.2.0
|
||||
mdurl==0.1.2
|
||||
mutagen==1.48.1
|
||||
|
||||
@@ -0,0 +1,738 @@
|
||||
"""Every refusal the backend can answer, read from its syntax tree: each `{"code": …, "reason": …}` detail,
|
||||
with every reason its code can be sent with.
|
||||
|
||||
The frontend's contract (`i18n/backendIds.contract.test.ts`) runs this and requires a sentence for each
|
||||
code and reason in both languages. A scan that tries to follow every shape opens a silent gap with each one
|
||||
it learns, and following how a value flows through Python (variables, scopes, aliases, a dict changed in
|
||||
place) never closes. So the CODE keeps to a narrow form, and this reads a CLOSED list of it, reporting
|
||||
everything else, never guessing:
|
||||
|
||||
- where a detail stands: where it is sent — `HTTPException`'s second argument or `detail=`, the `"detail"`
|
||||
value of a `JSONResponse`'s content dict — through the arms of `a if c else b`, or as what a module-level
|
||||
factory returns when every return of it is a detail and every reference to it is a call sent as one. A
|
||||
detail held in a variable is not followed;
|
||||
- the code: one literal. A `**` spread, or a key that is not a literal, stands BEFORE a literal `code` and
|
||||
`reason`, both of which the detail then carries: the later key wins, so no spread can change them;
|
||||
- a reason: a string literal (`None` sends none: the site counts as reasonless); both arms of
|
||||
`a if c else b`; a function's local bound only by plain `x = …`; a module constant reached as `mod.NAME`;
|
||||
a call's `Literal` return annotation;
|
||||
- a parameter typed with a `Literal` (basedpyright holds every value passed in), or a `str` one of a
|
||||
module-level helper (`refused`, `not_found`), followed to every call, each of which must pass it by
|
||||
position or by name; a parameter its body binds again is unread;
|
||||
- an attribute: `e.x` of the exception an `except C as e` catches, or of a parameter typed `C`, when a
|
||||
method of `C` declares `self.x: <Literal>`;
|
||||
- a map lookup (`_SETTING_REASONS[exc.reason]`): the map's values, and its keys must be exactly what the
|
||||
index can be.
|
||||
|
||||
A name bound twice at module level, inside a module-level `if`/`try`/loop, named by a `global`, or imported
|
||||
from two places is not looked up; nor is a local a lambda or a comprehension binds, or one a `nonlocal` or
|
||||
`global` names. A module constant or map that anything stores into, deletes from or (a map) uses but by
|
||||
`map[k]` or a read method is unread. A reference named like a followed helper that the scan cannot resolve
|
||||
(`import app.x`, a relative import, an app module that does not bind it) is unread. Anything else is UNREAD,
|
||||
or an unread code, and the contract fails on it unless no screen shows that code. Stdlib only: it runs on
|
||||
the host, beside the frontend's tests (`python -I backend/tests/refusal_sites.py` → JSON).
|
||||
"""
|
||||
|
||||
import ast
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
APP = pathlib.Path(__file__).resolve().parents[1] / "app"
|
||||
|
||||
# Files whose `"code"` dicts are data, not refusals: a subtitle track's status payload, TMDB country codes.
|
||||
NOT_REFUSALS = {
|
||||
"plex/subtitles.py": "a subtitle track's status payload (its own `code`)",
|
||||
"plex/tmdb.py": "a production country's ISO code",
|
||||
}
|
||||
|
||||
# What a map may be used by besides `map[k]`: a method that only reads it.
|
||||
READS = {"get", "items", "keys", "values"}
|
||||
|
||||
Node = ast.AST
|
||||
FUNCS = (ast.FunctionDef, ast.AsyncFunctionDef)
|
||||
SCOPES = (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda, ast.ClassDef)
|
||||
COMPREHENSIONS = (ast.ListComp, ast.SetComp, ast.DictComp, ast.GeneratorExp)
|
||||
COMPOUND = (
|
||||
ast.If,
|
||||
ast.Try,
|
||||
ast.TryStar,
|
||||
ast.For,
|
||||
ast.AsyncFor,
|
||||
ast.While,
|
||||
ast.With,
|
||||
ast.AsyncWith,
|
||||
ast.Match,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Res:
|
||||
values: set[str] = field(default_factory=set)
|
||||
unread: list[str] = field(default_factory=list)
|
||||
none: bool = False # an arm sends no reason
|
||||
|
||||
def __or__(self, other: "Res") -> "Res":
|
||||
return Res(self.values | other.values, self.unread + other.unread, self.none or other.none)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Module:
|
||||
path: str # relative to app/, posix
|
||||
name: str # dotted, `app.…`
|
||||
tree: ast.Module
|
||||
parents: dict[Node, Node]
|
||||
# local name → (module, symbol or None for the module)
|
||||
imports: dict[str, tuple[str, str | None]]
|
||||
top: dict[str, Node] # module-level bindings: def, class, or the assignment
|
||||
ambiguous: set[
|
||||
str
|
||||
] # bound more than once, conditionally, by a `global`, or imported from two places
|
||||
slots: set[int] # the ids of the nodes sent as a detail
|
||||
|
||||
|
||||
def _own_scope(node: Node) -> Iterator[Node]:
|
||||
"""`node` and what it holds in its own scope: a nested def, class or lambda is itself, not its body."""
|
||||
stack = [node]
|
||||
while stack:
|
||||
n = stack.pop()
|
||||
yield n
|
||||
if n is not node and isinstance(n, SCOPES):
|
||||
continue
|
||||
stack.extend(ast.iter_child_nodes(n))
|
||||
|
||||
|
||||
def _stored(target: Node) -> set[str]:
|
||||
"""The names an assignment target binds, not the ones it reads (`LABELS[k] = v` binds none)."""
|
||||
return {
|
||||
n.id for n in ast.walk(target) if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)
|
||||
}
|
||||
|
||||
|
||||
def _binds(node: Node) -> set[str]:
|
||||
"""The names one node binds in the scope it sits in."""
|
||||
if isinstance(node, ast.Assign):
|
||||
return set().union(*(_stored(t) for t in node.targets))
|
||||
if isinstance(node, (ast.AnnAssign, ast.AugAssign, ast.For, ast.AsyncFor, ast.NamedExpr)):
|
||||
return _stored(node.target)
|
||||
if isinstance(node, (ast.With, ast.AsyncWith)):
|
||||
return set().union(*(_stored(i.optional_vars) for i in node.items if i.optional_vars))
|
||||
if isinstance(node, (ast.Import, ast.ImportFrom)):
|
||||
return {(a.asname or a.name).split(".")[0] for a in node.names}
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
return {node.name}
|
||||
if isinstance(node, (ast.MatchAs, ast.MatchStar, ast.ExceptHandler)) and node.name:
|
||||
return {node.name}
|
||||
return set()
|
||||
|
||||
|
||||
def _sender(call: ast.Call) -> str | None:
|
||||
return getattr(call.func, "id", None) or getattr(call.func, "attr", None)
|
||||
|
||||
|
||||
def detail_slots(tree: Node) -> Iterator[tuple[ast.Call, Node | None]]:
|
||||
"""Every place a detail is sent from, with what it holds: `HTTPException`'s second argument or
|
||||
`detail=` (None: sent without one), and the `"detail"` value of a `JSONResponse`'s content dict (a
|
||||
content without one is an answer, not a refusal)."""
|
||||
for node in ast.walk(tree):
|
||||
if not isinstance(node, ast.Call):
|
||||
continue
|
||||
kw = {k.arg: k.value for k in node.keywords}
|
||||
if _sender(node) == "HTTPException":
|
||||
yield node, kw.get("detail", node.args[1] if len(node.args) > 1 else None)
|
||||
elif _sender(node) == "JSONResponse":
|
||||
content = kw.get("content", node.args[0] if node.args else None)
|
||||
if isinstance(content, ast.Dict):
|
||||
for k, v in zip(content.keys, content.values, strict=True):
|
||||
if isinstance(k, ast.Constant) and k.value == "detail":
|
||||
yield node, v
|
||||
|
||||
|
||||
def _code_dict(node: Node) -> bool:
|
||||
"""A `{"code": …}` dict, or a `dict(code=…)` call."""
|
||||
if isinstance(node, ast.Dict):
|
||||
return any(isinstance(k, ast.Constant) and k.value == "code" for k in node.keys)
|
||||
return (
|
||||
isinstance(node, ast.Call)
|
||||
and getattr(node.func, "id", None) == "dict"
|
||||
and any(k.arg == "code" for k in node.keywords)
|
||||
)
|
||||
|
||||
|
||||
def _module_name(path: str) -> str:
|
||||
parts = path.removesuffix(".py").split("/")
|
||||
if parts[-1] == "__init__":
|
||||
parts = parts[:-1]
|
||||
return ".".join(["app", *parts])
|
||||
|
||||
|
||||
def _load(sources: dict[str, str]) -> dict[str, Module]:
|
||||
mods: dict[str, Module] = {}
|
||||
for path, src in sources.items():
|
||||
tree = ast.parse(src)
|
||||
parents = {child: node for node in ast.walk(tree) for child in ast.iter_child_nodes(node)}
|
||||
origins: dict[str, set[tuple[str, str | None]]] = {}
|
||||
ambiguous: set[str] = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
|
||||
for a in node.names:
|
||||
origins.setdefault(a.asname or a.name, set()).add((node.module, a.name))
|
||||
elif isinstance(node, ast.Import):
|
||||
for a in node.names:
|
||||
if a.asname:
|
||||
origins.setdefault(a.asname, set()).add((a.name, None))
|
||||
elif isinstance(node, ast.Global):
|
||||
ambiguous |= set(node.names)
|
||||
top: dict[str, Node] = {}
|
||||
count: dict[str, int] = {}
|
||||
ambiguous |= {name for name, o in origins.items() if len(o) > 1}
|
||||
for stmt in tree.body:
|
||||
if isinstance(stmt, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
top[stmt.name] = stmt
|
||||
elif isinstance(stmt, ast.Assign):
|
||||
for t in stmt.targets:
|
||||
if isinstance(t, ast.Name):
|
||||
top[t.id] = stmt
|
||||
elif isinstance(stmt, ast.AnnAssign) and isinstance(stmt.target, ast.Name):
|
||||
top[stmt.target.id] = stmt
|
||||
for n in [stmt] if isinstance(stmt, SCOPES) else _own_scope(stmt):
|
||||
for name in _binds(n):
|
||||
count[name] = count.get(name, 0) + 1
|
||||
if isinstance(stmt, COMPOUND):
|
||||
ambiguous.add(name)
|
||||
ambiguous |= {name for name, c in count.items() if c > 1}
|
||||
imports = {name: next(iter(o)) for name, o in origins.items() if name not in ambiguous}
|
||||
slots = {id(d) for _, d in detail_slots(tree) if d is not None}
|
||||
name = _module_name(path)
|
||||
mods[name] = Module(path, name, tree, parents, imports, top, ambiguous, slots)
|
||||
return mods
|
||||
|
||||
|
||||
class Scan:
|
||||
def __init__(self, sources: dict[str, str]):
|
||||
self.mods = _load(sources)
|
||||
self.errors: list[str] = []
|
||||
self._following: set[tuple[str, str, str]] = set()
|
||||
self._followed: dict[tuple[str, str, str], Res] = {}
|
||||
self._resolving: set[int] = set()
|
||||
self._factory: dict[int, bool] = {}
|
||||
self._changed: dict[int, bool] = {}
|
||||
|
||||
# --- names ---------------------------------------------------------------------------------------
|
||||
|
||||
def lookup(
|
||||
self, m: Module, name: str, seen: frozenset[tuple[str, str]] = frozenset()
|
||||
) -> tuple[Module, Node] | tuple[str, None] | None:
|
||||
"""What a module-level name is: (its module, its binding), or (a module's name, None)."""
|
||||
if name in m.ambiguous:
|
||||
return None
|
||||
if name in m.top:
|
||||
return m, m.top[name]
|
||||
if name in m.imports:
|
||||
mod, sym = m.imports[name]
|
||||
if sym is None:
|
||||
return (mod, None) if mod in self.mods else None
|
||||
if f"{mod}.{sym}" in self.mods:
|
||||
return f"{mod}.{sym}", None
|
||||
if mod in self.mods and (mod, sym) not in seen:
|
||||
return self.lookup(self.mods[mod], sym, seen | {(mod, sym)})
|
||||
return None
|
||||
|
||||
def lookup_expr(self, m: Module, e: Node) -> tuple[Module, Node] | tuple[str, None] | None:
|
||||
if isinstance(e, ast.Name):
|
||||
return None if self.binder(m, e) is not None else self.lookup(m, e.id)
|
||||
if isinstance(e, ast.Attribute):
|
||||
base = self.lookup_expr(m, e.value)
|
||||
if base and base[1] is None and isinstance(base[0], str):
|
||||
return self.lookup(self.mods[base[0]], e.attr)
|
||||
return None
|
||||
|
||||
def binder(self, m: Module, e: ast.Name) -> Node | None:
|
||||
"""The innermost scope `e` is bound in: a function (a parameter, or a binding in its own body), a
|
||||
lambda (a parameter), a comprehension (its target); None: a module-level name."""
|
||||
node = m.parents.get(e)
|
||||
while node is not None:
|
||||
if isinstance(node, (*FUNCS, ast.Lambda)):
|
||||
a = node.args
|
||||
params = [*a.posonlyargs, *a.args, *a.kwonlyargs, a.vararg, a.kwarg]
|
||||
if e.id in {p.arg for p in params if p}:
|
||||
return node
|
||||
if isinstance(node, FUNCS) and self.assignments(node, e.id):
|
||||
return node
|
||||
elif isinstance(node, COMPREHENSIONS) and any(
|
||||
e.id in _stored(g.target) for g in node.generators
|
||||
):
|
||||
return node
|
||||
node = m.parents.get(node)
|
||||
return None
|
||||
|
||||
def at(self, m: Module, e: Node) -> str:
|
||||
return f"{m.path}:{getattr(e, 'lineno', '?')}: {ast.unparse(e)}"
|
||||
|
||||
def enclosing(self, m: Module, e: Node, kind: type | tuple[type, ...]) -> Node | None:
|
||||
node = m.parents.get(e)
|
||||
while node is not None and not isinstance(node, kind):
|
||||
node = m.parents.get(node)
|
||||
return node
|
||||
|
||||
def assignments(self, fn: Node, name: str) -> list[Node]:
|
||||
"""What binds `name` in this function's own body (not in a nested one)."""
|
||||
return [n for n in _own_scope(fn) if n is not fn and name in _binds(n)]
|
||||
|
||||
def caught_as(self, m: Module, e: Node, name: str) -> ast.ExceptHandler | None:
|
||||
node = m.parents.get(e)
|
||||
while node is not None and not isinstance(node, FUNCS):
|
||||
if isinstance(node, ast.ExceptHandler) and node.name == name:
|
||||
return node
|
||||
node = m.parents.get(node)
|
||||
return None
|
||||
|
||||
def references(
|
||||
self, m: Module, target: Node
|
||||
) -> Iterator[tuple[Module, Node, ast.Call | None, bool]]:
|
||||
"""Every reference to a module-level def or class: the call it makes (None: not a call), and
|
||||
whether it resolved. A reference named like it that the scan cannot resolve is yielded too."""
|
||||
name = getattr(target, "name", None)
|
||||
for cm in self.mods.values():
|
||||
for node in ast.walk(cm.tree):
|
||||
if not isinstance(node, (ast.Name, ast.Attribute)) or not isinstance(
|
||||
node.ctx, ast.Load
|
||||
):
|
||||
continue
|
||||
if (node.id if isinstance(node, ast.Name) else node.attr) != name:
|
||||
continue
|
||||
parent = cm.parents.get(node)
|
||||
if isinstance(parent, ast.Attribute) and parent.value is node:
|
||||
continue
|
||||
found = self.lookup_expr(cm, node)
|
||||
if found:
|
||||
if not (found[0] is m and found[1] is target):
|
||||
continue
|
||||
elif not self.unresolved(cm, node):
|
||||
continue
|
||||
call = parent if isinstance(parent, ast.Call) and parent.func is node else None
|
||||
yield cm, node, call, found is not None
|
||||
|
||||
def unresolved(self, m: Module, node: Node) -> bool:
|
||||
"""A name, or a dotted path, the app could mean that the scan cannot resolve: not a local, nor an
|
||||
outside module's (an attribute of an object the function holds is not one)."""
|
||||
root = node
|
||||
while isinstance(root, ast.Attribute):
|
||||
root = root.value
|
||||
if not isinstance(root, ast.Name) or self.binder(m, root) is not None:
|
||||
return False
|
||||
if root.id in m.ambiguous or (root.id not in m.top and root.id not in m.imports):
|
||||
return True
|
||||
if root.id in m.top:
|
||||
return False
|
||||
if root is node: # imported from an app module that does not bind it where the scan can see
|
||||
mod = m.imports[root.id][0]
|
||||
return (mod == "app" or mod.startswith("app.")) and self.lookup(m, root.id) is None
|
||||
found = self.lookup(m, root.id)
|
||||
return (
|
||||
found is not None and found[1] is None
|
||||
) # an app module whose dotted path went nowhere
|
||||
|
||||
def changed(self, m: Module, stmt: Node, name: str, *, only_read: bool) -> bool:
|
||||
"""Something in the app stores into, deletes or rebinds the module-level `name` that `stmt` binds,
|
||||
or (`only_read`, a map) uses it but by `map[k]` or a method that only reads it."""
|
||||
key = id(stmt) * 2 + only_read
|
||||
if key in self._changed:
|
||||
return self._changed[key]
|
||||
own = stmt.targets if isinstance(stmt, ast.Assign) else [getattr(stmt, "target", None)]
|
||||
out = False
|
||||
for cm in self.mods.values():
|
||||
for node in ast.walk(cm.tree):
|
||||
if out:
|
||||
break
|
||||
if not (
|
||||
isinstance(node, ast.Name)
|
||||
and node.id == name
|
||||
or isinstance(node, ast.Attribute)
|
||||
and node.attr == name
|
||||
) or (cm is m and node in own):
|
||||
continue
|
||||
found = self.lookup_expr(cm, node)
|
||||
if not (found and found[0] is m and found[1] is stmt):
|
||||
continue
|
||||
if not isinstance(node.ctx, ast.Load):
|
||||
out = True
|
||||
elif only_read:
|
||||
p = cm.parents.get(node)
|
||||
out = not (
|
||||
isinstance(p, ast.Subscript)
|
||||
and p.value is node
|
||||
and isinstance(p.ctx, ast.Load)
|
||||
or isinstance(p, ast.Attribute)
|
||||
and p.attr in READS
|
||||
)
|
||||
self._changed[key] = out
|
||||
return out
|
||||
|
||||
# --- types ---------------------------------------------------------------------------------------
|
||||
|
||||
def ann_values(self, m: Module, ann: Node | None, depth: int = 0) -> set[str] | None:
|
||||
"""The values a `Literal[…]` annotation, or an alias of one, allows; None when it is not one."""
|
||||
if isinstance(ann, ast.Subscript) and getattr(ann.value, "id", None) == "Literal":
|
||||
items = ann.slice.elts if isinstance(ann.slice, ast.Tuple) else [ann.slice]
|
||||
if all(isinstance(i, ast.Constant) and isinstance(i.value, str) for i in items):
|
||||
return {i.value for i in items}
|
||||
return None
|
||||
if isinstance(ann, (ast.Name, ast.Attribute)) and depth < 8:
|
||||
found = self.lookup_expr(m, ann)
|
||||
if found and isinstance(found[0], Module) and isinstance(found[1], ast.Assign):
|
||||
return self.ann_values(found[0], found[1].value, depth + 1)
|
||||
return None
|
||||
|
||||
def classes(self, m: Module, e: Node | None) -> list[tuple[Module, ast.ClassDef]] | None:
|
||||
found = self.lookup_expr(m, e) if e is not None else None
|
||||
if found and isinstance(found[0], Module) and isinstance(found[1], ast.ClassDef):
|
||||
return [(found[0], found[1])]
|
||||
return None
|
||||
|
||||
def typed(self, m: Module, at: Node, name: str) -> list[tuple[Module, ast.ClassDef]] | None:
|
||||
"""The class `name` holds where `at` sits: the one an `except C as name` catches, or a parameter's
|
||||
annotation (a parameter its body binds again holds no known class)."""
|
||||
handler = self.caught_as(m, at, name)
|
||||
if handler is not None:
|
||||
return self.classes(m, handler.type)
|
||||
fn = self.enclosing(m, at, FUNCS)
|
||||
if fn is None:
|
||||
return None
|
||||
params = {a.arg: a for a in [*fn.args.posonlyargs, *fn.args.args, *fn.args.kwonlyargs]}
|
||||
if name in params and not self.assignments(fn, name):
|
||||
return self.classes(m, params[name].annotation)
|
||||
return None
|
||||
|
||||
def attr_values(self, cm: Module, cls: ast.ClassDef, attr: str) -> set[str] | None:
|
||||
"""An attribute's `Literal` values, from `self.attr: T` in a method of the class."""
|
||||
for node in ast.walk(cls):
|
||||
if (
|
||||
isinstance(node, ast.AnnAssign)
|
||||
and isinstance(node.target, ast.Attribute)
|
||||
and node.target.attr == attr
|
||||
and getattr(node.target.value, "id", None) == "self"
|
||||
):
|
||||
return self.ann_values(cm, node.annotation)
|
||||
return None
|
||||
|
||||
# --- reasons -------------------------------------------------------------------------------------
|
||||
|
||||
def resolve(self, m: Module, e: Node) -> Res:
|
||||
if isinstance(e, ast.Constant):
|
||||
if e.value is None:
|
||||
return Res(none=True)
|
||||
if isinstance(e.value, str):
|
||||
return Res({e.value})
|
||||
if isinstance(e, ast.IfExp):
|
||||
return self.resolve(m, e.body) | self.resolve(m, e.orelse)
|
||||
if isinstance(e, ast.Name):
|
||||
return self.resolve_name(m, e)
|
||||
if isinstance(e, ast.Attribute):
|
||||
return self.resolve_attr(m, e)
|
||||
if isinstance(e, ast.Call):
|
||||
found = self.lookup_expr(m, e.func)
|
||||
if found and isinstance(found[0], Module) and isinstance(found[1], FUNCS):
|
||||
v = self.ann_values(found[0], found[1].returns)
|
||||
if v is not None:
|
||||
return Res(v)
|
||||
if isinstance(e, ast.Subscript):
|
||||
return self.resolve_lookup(m, e)
|
||||
return Res(unread=[self.at(m, e)])
|
||||
|
||||
def resolve_name(self, m: Module, e: ast.Name) -> Res:
|
||||
if self.caught_as(m, e, e.id) is not None:
|
||||
return Res(unread=[self.at(m, e)]) # the exception itself, not a reason
|
||||
fn = self.binder(m, e)
|
||||
if fn is None:
|
||||
return Res(unread=[self.at(m, e)])
|
||||
if not isinstance(fn, FUNCS):
|
||||
return Res(unread=[f"{self.at(m, e)} (bound by a lambda or a comprehension)"])
|
||||
if any(isinstance(n, (ast.Nonlocal, ast.Global)) and e.id in n.names for n in ast.walk(fn)):
|
||||
return Res(unread=[f"{self.at(m, e)} (named by a nonlocal or a global)"])
|
||||
a = fn.args
|
||||
params = {p.arg: p for p in [*a.posonlyargs, *a.args, *a.kwonlyargs]}
|
||||
assigned = self.assignments(fn, e.id)
|
||||
if e.id in params:
|
||||
if assigned:
|
||||
return Res(unread=[f"{self.at(m, e)} (a parameter its body binds again)"])
|
||||
return self.resolve_param(m, fn, params[e.id], e)
|
||||
out = Res(unread=[] if assigned else [self.at(m, e)]) # `*args`, `**kwargs`
|
||||
for s in assigned:
|
||||
if not (isinstance(s, ast.Assign) and all(isinstance(t, ast.Name) for t in s.targets)):
|
||||
out = out | Res(unread=[self.at(m, e)])
|
||||
elif id(s) in self._resolving:
|
||||
out = out | Res(unread=[f"{self.at(m, e)} (reached again while it is resolved)"])
|
||||
else:
|
||||
self._resolving.add(id(s))
|
||||
out = out | self.resolve(m, s.value)
|
||||
self._resolving.discard(id(s))
|
||||
return out
|
||||
|
||||
def resolve_param(self, m: Module, fn: Node, param: ast.arg, e: Node) -> Res:
|
||||
v = self.ann_values(m, param.annotation)
|
||||
if v is not None:
|
||||
return Res(v)
|
||||
if param.annotation is not None and getattr(param.annotation, "id", None) != "str":
|
||||
return Res(unread=[self.at(m, e)])
|
||||
return self.follow(m, fn, param)
|
||||
|
||||
def follow(self, m: Module, fn: Node, param: ast.arg) -> Res:
|
||||
"""A `str` parameter of a helper: what every call of the helper passes it."""
|
||||
assert isinstance(fn, FUNCS)
|
||||
key = (m.name, fn.name, param.arg)
|
||||
if key in self._followed:
|
||||
return self._followed[key]
|
||||
where = f"{m.path}:{fn.lineno}: {fn.name}({param.arg})"
|
||||
if key in self._following:
|
||||
return Res(unread=[f"{where} reached again while it is followed"])
|
||||
if m.top.get(fn.name) is not fn or fn.name in m.ambiguous:
|
||||
return Res(unread=[f"{where} is not a module-level helper"])
|
||||
self._following.add(key)
|
||||
positional = [a.arg for a in [*fn.args.posonlyargs, *fn.args.args]]
|
||||
by_name = param not in fn.args.posonlyargs
|
||||
out = Res()
|
||||
for cm, node, call, resolved in self.references(m, fn):
|
||||
if not resolved:
|
||||
out = out | Res(
|
||||
unread=[f"{self.at(cm, node)} (a reference the scan cannot resolve)"]
|
||||
)
|
||||
continue
|
||||
if call is None:
|
||||
out = out | Res(unread=[f"{self.at(cm, node)} (the helper used as a value)"])
|
||||
continue
|
||||
kw = {k.arg: k.value for k in call.keywords}
|
||||
given = param.arg in positional and positional.index(param.arg) < len(call.args)
|
||||
if by_name and param.arg in kw:
|
||||
out = out | self.resolve(cm, kw[param.arg])
|
||||
elif any(isinstance(a, ast.Starred) for a in call.args):
|
||||
out = out | Res(unread=[self.at(cm, call)])
|
||||
elif given:
|
||||
out = out | self.resolve(cm, call.args[positional.index(param.arg)])
|
||||
else:
|
||||
out = out | Res(unread=[f"{self.at(cm, call)} (does not pass {param.arg})"])
|
||||
self._followed[key] = out # once followed, its key answers from here: never reached again
|
||||
return out
|
||||
|
||||
def resolve_attr(self, m: Module, e: ast.Attribute) -> Res:
|
||||
base = e.value
|
||||
if isinstance(base, ast.Name):
|
||||
typed = self.typed(m, e, base.id)
|
||||
if typed is not None:
|
||||
out = Res()
|
||||
for cm, cls in typed:
|
||||
v = self.attr_values(cm, cls, e.attr)
|
||||
out = out | (
|
||||
Res(v)
|
||||
if v is not None
|
||||
else Res(
|
||||
unread=[f"{self.at(m, e)} ({cls.name}.{e.attr} has no Literal type)"]
|
||||
)
|
||||
)
|
||||
return out
|
||||
found = self.lookup_expr(m, e)
|
||||
if (
|
||||
found
|
||||
and isinstance(found[0], Module)
|
||||
and isinstance(found[1], (ast.Assign, ast.AnnAssign))
|
||||
and isinstance(found[1].value, ast.Constant)
|
||||
):
|
||||
if self.changed(found[0], found[1], e.attr, only_read=False):
|
||||
return Res(unread=[f"{self.at(m, e)} (bound again or deleted elsewhere)"])
|
||||
return self.resolve(found[0], found[1].value)
|
||||
return Res(unread=[self.at(m, e)])
|
||||
|
||||
def resolve_lookup(self, m: Module, e: ast.Subscript) -> Res:
|
||||
found = self.lookup_expr(m, e.value)
|
||||
if not (
|
||||
found
|
||||
and isinstance(found[0], Module)
|
||||
and isinstance(found[1], (ast.Assign, ast.AnnAssign))
|
||||
):
|
||||
return Res(unread=[self.at(m, e)])
|
||||
table = found[1].value
|
||||
if not isinstance(table, ast.Dict) or not all(
|
||||
isinstance(k, ast.Constant) and isinstance(k.value, str) for k in table.keys
|
||||
):
|
||||
return Res(unread=[self.at(m, e)])
|
||||
name = e.value.attr if isinstance(e.value, ast.Attribute) else getattr(e.value, "id", "")
|
||||
if self.changed(found[0], found[1], name, only_read=True):
|
||||
return Res(unread=[f"{self.at(m, e)} (the map is changed after it is built)"])
|
||||
out = Res()
|
||||
for v in table.values:
|
||||
out = out | self.resolve(found[0], v)
|
||||
index = self.resolve(m, e.slice)
|
||||
keys = {k.value for k in table.keys if isinstance(k, ast.Constant)}
|
||||
if index.unread or index.none or index.values != keys:
|
||||
self.errors.append(
|
||||
f"{self.at(m, e)}: the map's keys {sorted(keys)} are not what its index can be "
|
||||
f"{sorted(index.values)}{' + unread ' + str(index.unread) if index.unread else ''}"
|
||||
)
|
||||
return out
|
||||
|
||||
# --- sites ---------------------------------------------------------------------------------------
|
||||
|
||||
def is_detail(self, node: Node | None) -> bool:
|
||||
if isinstance(node, ast.IfExp):
|
||||
return self.is_detail(node.body) and self.is_detail(node.orelse)
|
||||
return _code_dict(node)
|
||||
|
||||
def factory(self, m: Module, fn: Node) -> bool:
|
||||
"""A module-level function each return of which is a detail, and each reference to which is a
|
||||
call sent as one (`quota_detail`)."""
|
||||
assert isinstance(fn, FUNCS)
|
||||
if id(fn) in self._factory:
|
||||
return self._factory[id(fn)]
|
||||
returns = [n for n in _own_scope(fn) if isinstance(n, ast.Return)]
|
||||
ok = (
|
||||
m.top.get(fn.name) is fn # a name bound twice: its calls are unresolved below
|
||||
and bool(returns)
|
||||
and all(self.is_detail(r.value) for r in returns)
|
||||
)
|
||||
for cm, _, call, resolved in self.references(m, fn) if ok else []:
|
||||
ok = resolved and id(call) in cm.slots # a bare reference is no call: never sent
|
||||
if not ok:
|
||||
break
|
||||
self._factory[id(fn)] = ok
|
||||
return ok
|
||||
|
||||
def placed(self, m: Module, d: Node) -> bool:
|
||||
"""`d` is sent as a detail: from a sender's slot, through the arms of `a if c else b`, or as what a
|
||||
factory returns."""
|
||||
holder, p = d, m.parents.get(d)
|
||||
while isinstance(p, ast.IfExp) and holder is not p.test:
|
||||
holder, p = p, m.parents.get(p)
|
||||
if id(holder) in m.slots:
|
||||
return True
|
||||
fn = self.enclosing(m, p, FUNCS) if isinstance(p, ast.Return) else None
|
||||
return fn is not None and self.factory(m, fn)
|
||||
|
||||
def sayable(self, m: Module, d: Node | None) -> bool:
|
||||
"""What a slot holds is a detail the client can say: a code dict, or a factory's call."""
|
||||
if isinstance(d, ast.IfExp):
|
||||
return self.sayable(m, d.body) and self.sayable(m, d.orelse)
|
||||
if _code_dict(d):
|
||||
return True
|
||||
found = self.lookup_expr(m, d.func) if isinstance(d, ast.Call) else None
|
||||
return bool(
|
||||
found
|
||||
and isinstance(found[0], Module)
|
||||
and isinstance(found[1], FUNCS)
|
||||
and self.factory(found[0], found[1])
|
||||
)
|
||||
|
||||
def entries(self, d: Node) -> list[tuple[str | None, Node]]:
|
||||
"""A dict's (key, value) pairs: None for a `**` spread, "?" for a key that is not a literal."""
|
||||
if isinstance(d, ast.Dict):
|
||||
return [
|
||||
(
|
||||
k.value
|
||||
if isinstance(k, ast.Constant) and isinstance(k.value, str)
|
||||
else None
|
||||
if k is None
|
||||
else "?",
|
||||
v,
|
||||
)
|
||||
for k, v in zip(d.keys, d.values, strict=True)
|
||||
]
|
||||
assert isinstance(d, ast.Call)
|
||||
return [(None, a) for a in d.args] + [(k.arg, k.value) for k in d.keywords]
|
||||
|
||||
def site(self, m: Module, d: Node, out: dict) -> None:
|
||||
items = self.entries(d)
|
||||
code = Res()
|
||||
for v in (v for k, v in items if k == "code"):
|
||||
code = code | self.resolve(m, v)
|
||||
problems = [
|
||||
f"{self.at(m, v)} (a spread in dict())"
|
||||
for k, v in items
|
||||
if k is None and isinstance(d, ast.Call)
|
||||
] + [
|
||||
f"{self.at(m, v)} (a spread or a computed key the code and the reason do not follow)"
|
||||
for i, (k, v) in enumerate(items)
|
||||
if k in (None, "?") and not {"code", "reason"} <= {k2 for k2, _ in items[i + 1 :]}
|
||||
]
|
||||
if not self.placed(m, d):
|
||||
problems.append(f"{self.at(m, d)} (a detail put where the scan cannot follow it)")
|
||||
if code.unread or code.none or len(code.values) != 1 or problems:
|
||||
out["unreadCodes"] += problems or [self.at(m, d)]
|
||||
return
|
||||
entry = out["codes"].setdefault(
|
||||
next(iter(code.values)), {"reasons": set(), "unread": [], "sites": 0, "reasonless": []}
|
||||
)
|
||||
entry["sites"] += 1
|
||||
res, has_reason = Res(), False
|
||||
for k, v in items:
|
||||
if k == "reason":
|
||||
res, has_reason = res | self.resolve(m, v), True
|
||||
if not has_reason or res.none:
|
||||
entry["reasonless"].append(f"{m.path}:{d.lineno}")
|
||||
entry["reasons"] |= res.values
|
||||
entry["unread"] += res.unread
|
||||
|
||||
def other_details(self) -> list[tuple[str, int, str | None]]:
|
||||
"""Each slot that sends what the client cannot say — nothing, or anything but a code dict or a
|
||||
factory's call: (its file, its line, the function it is sent from)."""
|
||||
out = []
|
||||
for m in self.mods.values():
|
||||
for call, d in detail_slots(m.tree):
|
||||
if self.sayable(m, d):
|
||||
continue
|
||||
fn = self.enclosing(m, call, FUNCS)
|
||||
out.append((m.path, call.lineno, getattr(fn, "name", None)))
|
||||
return sorted(out)
|
||||
|
||||
def run(self) -> dict:
|
||||
out: dict = {
|
||||
"codes": {},
|
||||
"unreadCodes": [],
|
||||
"errors": self.errors,
|
||||
"excluded": {},
|
||||
"aliases": {},
|
||||
"notRefusals": sorted(NOT_REFUSALS),
|
||||
}
|
||||
for m in self.mods.values():
|
||||
for name, stmt in m.top.items():
|
||||
if (
|
||||
isinstance(stmt, ast.Assign)
|
||||
and isinstance(stmt.value, ast.Subscript)
|
||||
and (v := self.ann_values(m, stmt.value))
|
||||
):
|
||||
out["aliases"].setdefault(m.path, {})[name] = sorted(v)
|
||||
for node in ast.walk(m.tree):
|
||||
if not _code_dict(node):
|
||||
continue
|
||||
if m.path in NOT_REFUSALS:
|
||||
out["excluded"][m.path] = out["excluded"].get(m.path, 0) + 1
|
||||
continue
|
||||
self.site(m, node, out)
|
||||
for entry in out["codes"].values():
|
||||
entry["reasons"] = sorted(entry["reasons"])
|
||||
return out
|
||||
|
||||
|
||||
def scan(sources: dict[str, str]) -> dict:
|
||||
return Scan(sources).run()
|
||||
|
||||
|
||||
def app_sources(app: pathlib.Path = APP) -> dict[str, str]:
|
||||
return {
|
||||
p.relative_to(app).as_posix(): p.read_text(encoding="utf-8")
|
||||
for p in sorted(app.rglob("*.py"))
|
||||
if "__pycache__" not in p.parts
|
||||
}
|
||||
|
||||
|
||||
def scan_app(app: pathlib.Path = APP) -> dict:
|
||||
return scan(app_sources(app))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
json.dump(
|
||||
scan_app(pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else APP),
|
||||
sys.stdout,
|
||||
indent=1,
|
||||
sort_keys=True,
|
||||
)
|
||||
@@ -125,3 +125,109 @@ def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_u
|
||||
r = c.post("/api/playlists/sync-youtube")
|
||||
assert r.status_code == 429
|
||||
assert r.json()["detail"] == {"code": "sync_cooldown"}
|
||||
|
||||
|
||||
def _pushable_list(db, make_user, monkeypatch, plroutes):
|
||||
"""A local (never exported) list and a user with the write scope; the merge itself is stubbed —
|
||||
the limiter is under test, not the YouTube round trip."""
|
||||
from app import auth
|
||||
from app.reader.models import Playlist
|
||||
|
||||
user = make_user()
|
||||
pl = Playlist(user_id=user.id, name="Local", source="local")
|
||||
db.add(pl)
|
||||
db.commit()
|
||||
monkeypatch.setattr(auth, "has_write_scope", lambda u: True)
|
||||
monkeypatch.setattr(plroutes, "plan_merge", lambda d, u, p, live=None: {"units_estimate": 0})
|
||||
monkeypatch.setattr(plroutes, "merge_playlist", lambda d, u, p, live=None: {"added": 0})
|
||||
return user, pl
|
||||
|
||||
|
||||
def test_the_push_side_goes_on_cooldown_after_its_allowance(db, as_user, make_user, monkeypatch):
|
||||
# SEC-18: push-plan reads the live list and push writes it, both against the instance-wide
|
||||
# YouTube budget. A press in the UI is one plan + one push, so the allowance is generous — but a
|
||||
# scripted loop stops at it, with a reason the UI can say.
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
c = as_user(user)
|
||||
allowance = plroutes.PUSH_ALLOWANCE
|
||||
codes = [c.get(f"/api/playlists/{pl.id}/push-plan").status_code for _ in range(allowance)]
|
||||
assert codes == [200] * allowance
|
||||
r = c.get(f"/api/playlists/{pl.id}/push-plan")
|
||||
assert r.status_code == 429
|
||||
assert r.json()["detail"] == {"code": "playlist", "reason": "push_cooldown"}
|
||||
|
||||
|
||||
def test_plan_and_push_draw_on_one_allowance(db, as_user, make_user, monkeypatch):
|
||||
# The asymmetric fix again, avoided up front: a limit on one of the two routes leaves the same
|
||||
# drain open next door.
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
c = as_user(user)
|
||||
for i in range(plroutes.PUSH_ALLOWANCE):
|
||||
if i % 2:
|
||||
r = c.get(f"/api/playlists/{pl.id}/push-plan")
|
||||
else:
|
||||
r = c.post(f"/api/playlists/{pl.id}/push")
|
||||
assert r.status_code == 200, (i, r.json())
|
||||
r = c.post(f"/api/playlists/{pl.id}/push")
|
||||
assert r.status_code == 429
|
||||
assert r.json()["detail"] == {"code": "playlist", "reason": "push_cooldown"}
|
||||
|
||||
|
||||
def test_a_refused_push_spends_no_allowance(db, as_user, make_user, monkeypatch):
|
||||
# The allowance counts pushes that run: Watch later (never pushable) and a list another sync
|
||||
# holds are refused before any YouTube call, so they must not leave the next real press at 429.
|
||||
from contextlib import contextmanager
|
||||
|
||||
from app.reader.models import Playlist
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
wl = Playlist(user_id=user.id, name="Watch later", kind="watch_later")
|
||||
db.add(wl)
|
||||
db.commit()
|
||||
c = as_user(user)
|
||||
over = plroutes.PUSH_ALLOWANCE + 1
|
||||
assert {c.get(f"/api/playlists/{wl.id}/push-plan").status_code for _ in range(over)} == {400}
|
||||
assert {c.post(f"/api/playlists/{wl.id}/push").status_code for _ in range(over)} == {400}
|
||||
|
||||
@contextmanager
|
||||
def busy(_db, _id):
|
||||
yield False
|
||||
|
||||
with monkeypatch.context() as m:
|
||||
m.setattr(plroutes, "sync_claim", busy)
|
||||
assert {c.post(f"/api/playlists/{pl.id}/push").status_code for _ in range(over)} == {409}
|
||||
assert c.post(f"/api/playlists/{pl.id}/push").status_code == 200
|
||||
|
||||
|
||||
def test_a_mirrored_youtube_list_is_pushable(db, as_user, make_user, monkeypatch):
|
||||
# A mirror of the user's own YouTube list takes local edits (`dirty`) and syncs them back; only
|
||||
# Watch later is refused.
|
||||
from app.reader.models import Playlist
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user, _ = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
mirror = Playlist(
|
||||
user_id=user.id, name="Mine", source="youtube", yt_playlist_id="PLx", dirty=True
|
||||
)
|
||||
db.add(mirror)
|
||||
db.commit()
|
||||
assert as_user(user).get(f"/api/playlists/{mirror.id}/push-plan").status_code == 200
|
||||
|
||||
|
||||
def test_the_push_cooldown_is_per_user(db, as_user, make_user, monkeypatch):
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
a, pa = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
b, pb = _pushable_list(db, make_user, monkeypatch, plroutes)
|
||||
ca = as_user(a)
|
||||
codes = [
|
||||
ca.get(f"/api/playlists/{pa.id}/push-plan").status_code
|
||||
for _ in range(plroutes.PUSH_ALLOWANCE + 1)
|
||||
]
|
||||
assert codes == [200] * plroutes.PUSH_ALLOWANCE + [429]
|
||||
assert as_user(b).get(f"/api/playlists/{pb.id}/push-plan").status_code == 200
|
||||
|
||||
@@ -72,3 +72,23 @@ def test_the_sync_status_counts_the_callers_subscriptions(db, make_user, as_user
|
||||
assert (body["subscriptions"], body["channels_total"]) == (1, 1)
|
||||
mine = as_user(user).get("/api/sync/my-status").json()
|
||||
assert (mine["channels_total"], mine["channels_recent_pending"]) == (1, 1)
|
||||
|
||||
|
||||
def test_a_channel_lists_when_it_entered_this_users_list_not_when_the_channel_did(
|
||||
db, make_user, as_user
|
||||
):
|
||||
from datetime import datetime, timezone
|
||||
|
||||
user = make_user()
|
||||
db.add(Channel(id="UCa", title="A", created_at=datetime(2026, 6, 10, tzinfo=timezone.utc)))
|
||||
db.flush()
|
||||
db.add(
|
||||
Subscription(
|
||||
user_id=user.id,
|
||||
channel_id="UCa",
|
||||
created_at=datetime(2026, 10, 12, 8, 30, tzinfo=timezone.utc),
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
listed = as_user(user).get("/api/channels").json()
|
||||
assert [(c["id"], c["added_at"]) for c in listed] == [("UCa", "2026-10-12T08:30:00+00:00")]
|
||||
|
||||
@@ -531,6 +531,9 @@ def test_a_youtube_clip_hands_the_range_to_yt_dlp(monkeypatch, tmp_path):
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *_a):
|
||||
return False
|
||||
|
||||
@@ -707,6 +710,160 @@ def test_a_clip_larger_than_what_its_owner_has_left_is_refused_and_not_placed(
|
||||
)
|
||||
|
||||
|
||||
# --- a GIF clip ------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_gif_clip_spec_names_its_size_and_is_another_clip_than_the_video():
|
||||
mp4 = clipmod.normalize("youtube", "abcdefghijk", 10, 20)
|
||||
gif = clipmod.normalize("youtube", "abcdefghijk", 10, 20, gif_side=320)
|
||||
assert gif == {**mp4, "gif": {"side": 320}}
|
||||
# A video clip's spec (and so its cache key) is the one it always was.
|
||||
assert "gif" not in mp4
|
||||
assert clipmod.sig(gif) != clipmod.sig(mp4)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("kw", "reason", "params"),
|
||||
[
|
||||
({"end_s": 40.001, "gif_side": 480}, "gif_too_long", {"limit_s": 30}),
|
||||
({"end_s": 20, "gif_side": 500}, "bad_gif", {}),
|
||||
({"end_s": 20, "gif_side": True}, "bad_gif", {}),
|
||||
],
|
||||
)
|
||||
def test_a_gif_clip_it_cannot_make_is_refused_with_its_reason(kw, reason, params):
|
||||
with pytest.raises(clipmod.ClipError) as e:
|
||||
clipmod.normalize("youtube", "abcdefghijk", 10, **kw)
|
||||
assert (e.value.reason, e.value.params) == (reason, params)
|
||||
|
||||
|
||||
def test_the_thirty_second_gif_cap_itself_is_allowed():
|
||||
assert clipmod.duration(clipmod.normalize("plex", "42", 0, 30, gif_side=480)) == 30
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("start", "end", "kw"),
|
||||
[
|
||||
# Each a range exactly at a cap whose float difference lands a hair past it.
|
||||
(30.529, 60.529, {"gif_side": 480}),
|
||||
(424.005, 1024.005, {}),
|
||||
(0.001, 1.001, {}),
|
||||
],
|
||||
)
|
||||
def test_a_range_exactly_at_a_cap_is_judged_in_milliseconds(start, end, kw):
|
||||
assert end - start not in (1, 30, 600) # the float difference is off the cap itself
|
||||
assert round((end - start) * 1000) in (1000, 30000, 600000)
|
||||
clipmod.normalize("youtube", "abcdefghijk", start, end, **kw)
|
||||
|
||||
|
||||
def test_a_gif_clip_keeps_no_audio_track_so_the_track_does_not_make_another_gif():
|
||||
first = clipmod.normalize("plex", "42", 10, 20, audio_track=0, gif_side=480)
|
||||
other = clipmod.normalize("plex", "42", 10, 20, audio_track=1, gif_side=480)
|
||||
assert "audio_track" not in first
|
||||
assert clipmod.sig(first) == clipmod.sig(other)
|
||||
# A video clip keeps the track it was heard on.
|
||||
assert clipmod.normalize("plex", "42", 10, 20, audio_track=1)["audio_track"] == 1
|
||||
|
||||
|
||||
def test_a_gif_clip_is_queued_as_a_file_and_too_long_says_its_own_cap(
|
||||
db, make_user, as_user, yt_video
|
||||
):
|
||||
c = as_user(make_user())
|
||||
r = _clip(c, source="youtube", ref=yt_video, gif_side=480)
|
||||
assert r.status_code == 200, r.text
|
||||
job = r.json()
|
||||
assert job["edit_spec"]["gif"] == {"side": 480}
|
||||
# A GIF is a picture: no player, no editor; it previews as an image.
|
||||
assert db.get(MediaAsset, job["asset"]["id"]).media_kind == "file"
|
||||
r = _clip(c, source="youtube", ref=yt_video, gif_side=480, end_s=45)
|
||||
assert r.status_code == 422
|
||||
assert r.json()["detail"] == {"code": "clip", "reason": "gif_too_long", "limit_s": 30}
|
||||
|
||||
|
||||
def _gif_bytes(w: int, h: int, n: int = 0) -> bytes:
|
||||
return b"GIF89a" + w.to_bytes(2, "little") + h.to_bytes(2, "little") + b"\x00" * (1 + n)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gif_worker(worker, monkeypatch):
|
||||
"""The worker stand-in, writing a GIF header where a GIF is asked for; the still is a sidecar."""
|
||||
seen = worker
|
||||
seen["gif_extra"] = 0
|
||||
|
||||
def ffmpeg(cmd, *_a, **_k):
|
||||
seen["cmd"] = cmd
|
||||
out = Path(cmd[-1])
|
||||
out.write_bytes(_gif_bytes(270, 480, seen["gif_extra"]) if out.suffix == ".gif" else b"x")
|
||||
|
||||
def still(path):
|
||||
dest = path.with_suffix(".jpg")
|
||||
dest.write_bytes(b"\xff\xd8\xff")
|
||||
return dest
|
||||
|
||||
def phone_safe(*_a, **_k):
|
||||
raise AssertionError("a GIF is not a video: no phone-safe pass")
|
||||
|
||||
monkeypatch.setattr(pipeline.editmod, "run_ffmpeg", ffmpeg)
|
||||
monkeypatch.setattr(pipeline.gifmod, "ensure_still", still)
|
||||
monkeypatch.setattr(pipeline, "_ensure_phone_safe", phone_safe)
|
||||
monkeypatch.setattr(probemod, "probe_media", lambda _p: (None, None, None, None, None))
|
||||
return seen
|
||||
|
||||
|
||||
def test_a_gif_clip_is_cut_through_the_gif_command_and_stored_as_an_image_file(
|
||||
db, make_user, as_user, plex_episode, gif_worker
|
||||
):
|
||||
gif_worker["probe"] = {
|
||||
"streams": [{"codec_type": "video", "color_transfer": "smpte2084"}, {"codec_type": "audio"}]
|
||||
}
|
||||
job = _clip(as_user(make_user()), source="plex", ref="777", gif_side=480, end_s=25).json()
|
||||
job, asset = gif_worker["run"](job)
|
||||
assert (job.status, asset.status, asset.container, asset.media_kind) == (
|
||||
"done",
|
||||
"ready",
|
||||
"gif",
|
||||
"file",
|
||||
)
|
||||
assert asset.rel_path.endswith(".gif")
|
||||
assert (asset.width, asset.height) == (270, 480)
|
||||
assert asset.poster_path == str(Path(asset.rel_path).with_suffix(".jpg"))
|
||||
cmd = gif_worker["cmd"]
|
||||
graph = cmd[cmd.index("-filter_complex") + 1]
|
||||
# The HDR source is tonemapped before the palette is made from it; no sound rides along.
|
||||
assert graph.index("tonemap") < graph.index("palettegen")
|
||||
assert "-an" in cmd and "-map" not in cmd
|
||||
assert cmd[cmd.index("-t") + 1] == "15.000"
|
||||
|
||||
|
||||
def test_a_gif_clip_is_never_refused_for_an_audio_track(
|
||||
db, make_user, as_user, plex_episode, gif_worker
|
||||
):
|
||||
# A GIF queued with a track (as one made before GIFs dropped it was): the file has one track,
|
||||
# the spec names a fourth, and a GIF, which has no sound, is made all the same.
|
||||
job = _clip(as_user(make_user()), source="plex", ref="777", gif_side=480, end_s=25).json()
|
||||
row = db.get(DownloadJob, job["id"])
|
||||
row.edit_spec = {**row.edit_spec, "audio_track": 3}
|
||||
db.commit()
|
||||
job, asset = gif_worker["run"](job)
|
||||
assert (job.status, asset.status) == ("done", "ready")
|
||||
|
||||
|
||||
def test_a_gif_over_its_size_cap_is_not_kept_and_says_why(
|
||||
db, make_user, as_user, plex_episode, gif_worker, monkeypatch
|
||||
):
|
||||
from app.downloads import gif as gifmod
|
||||
|
||||
monkeypatch.setattr(gifmod, "MAX_BYTES", 20)
|
||||
gif_worker["gif_extra"] = 50
|
||||
job = _clip(as_user(make_user()), source="plex", ref="777", gif_side=640).json()
|
||||
job, asset = gif_worker["run"](job)
|
||||
assert (job.status, job.error_code, job.error, asset.rel_path) == (
|
||||
"error",
|
||||
"gif_too_big",
|
||||
dlerrors.GIF_TOO_BIG_MESSAGE,
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
def test_a_youtube_clip_range_yt_dlp_refuses_for_its_cap_says_it_is_too_big(
|
||||
db, make_user, as_user, yt_video, worker, monkeypatch
|
||||
):
|
||||
|
||||
@@ -75,6 +75,8 @@ def test_our_own_worker_refusals_classify_as_themselves():
|
||||
assert classify(dlerrors.CLIP_TOO_BIG_MESSAGE) == "quota_exceeded"
|
||||
assert classify(dlerrors.CONVERTED_TOO_BIG_MESSAGE) == "quota_exceeded"
|
||||
assert classify(dlerrors.DOWNLOAD_TOO_BIG_MESSAGE) == "quota_exceeded"
|
||||
# A GIF over its own cap is not the storage: its own category, its own sentence.
|
||||
assert classify(dlerrors.GIF_TOO_BIG_MESSAGE) == "gif_too_big"
|
||||
|
||||
|
||||
def test_the_redacted_network_text_classifies_back_to_network():
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
"""The editor's crop lands on the part of the picture the frame was drawn on, also when a video's
|
||||
pixels are not square. The browser reports such a video by its shown size (320 stored, 640 shown),
|
||||
and ffmpeg's `crop` counts stored pixels: measured on a real clip, a frame over the middle half kept
|
||||
half the frame's red surroundings. So a crop carries the size it was measured on.
|
||||
|
||||
Real ffmpeg: the pixels of the output are the oracle.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import edit, pipeline, service, storage
|
||||
from app.downloads import upload as uploadmod
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
|
||||
pytestmark = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
|
||||
|
||||
|
||||
def _source(path: Path, size: str, box: str, sar: int, rotate: int = 0) -> None:
|
||||
"""Red, with a green box: a crop of the box holds no red. `rotate` tags the file to be shown
|
||||
turned by that many degrees counter-clockwise, as a phone records a portrait video."""
|
||||
tag = ["-display_rotation:v:0", str(rotate)] if rotate else []
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", f"color=c=red:s={size}:d=2:r=10",
|
||||
"-f", "lavfi", "-i", "sine=duration=2",
|
||||
"-vf", f"drawbox={box}:c=lime:t=fill,setsar={sar}",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", str(path),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
if tag:
|
||||
tagged = path.with_name("tagged-" + path.name)
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-loglevel", "error", *tag, "-i", str(path), "-c", "copy", str(tagged)],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
tagged.replace(path)
|
||||
|
||||
|
||||
def _red_and_green(path: Path) -> tuple[int, int]:
|
||||
raw = subprocess.run(
|
||||
["ffmpeg", "-loglevel", "error", "-i", str(path), "-frames:v", "1",
|
||||
"-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
).stdout # fmt: skip
|
||||
red = green = 0
|
||||
for i in range(0, len(raw) - 2, 3):
|
||||
if raw[i] > 180 and raw[i + 1] < 80:
|
||||
red += 1
|
||||
elif raw[i + 1] > 180 and raw[i] < 80:
|
||||
green += 1
|
||||
return red, green
|
||||
|
||||
|
||||
def _cut_asset(db, make_user, src: Path, spec: dict, then=lambda: None) -> MediaAsset:
|
||||
user = make_user()
|
||||
source = uploadmod.ingest_upload(
|
||||
db, user.id, filename="clip.mp4", src_path=src, size=src.stat().st_size
|
||||
)
|
||||
assert source.status == "done"
|
||||
then()
|
||||
job = service.enqueue_edit(db, user.id, source, spec)
|
||||
asset = db.get(MediaAsset, job.asset_id)
|
||||
job.status, asset.status = "running", "downloading" # the worker's claim
|
||||
db.commit()
|
||||
|
||||
pipeline._process_edit(job.id, asset.id)
|
||||
|
||||
db.expire_all()
|
||||
job, asset = db.get(DownloadJob, job.id), db.get(MediaAsset, job.asset_id)
|
||||
assert (job.status, job.error, asset.status) == ("done", None, "ready")
|
||||
return asset
|
||||
|
||||
|
||||
def _cut(db, make_user, tmp_path, src: Path, crop: dict, **spec) -> Path:
|
||||
asset = _cut_asset(db, make_user, src, {"crop": crop, **spec})
|
||||
return storage.abs_path(str(tmp_path), asset.rel_path)
|
||||
|
||||
|
||||
def test_a_crop_of_a_video_with_wide_pixels_keeps_what_the_frame_showed(
|
||||
db, make_user, tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
|
||||
|
||||
# What the editor sends for the middle half of this video: the browser shows it at 640 x 240.
|
||||
final = _cut(
|
||||
db, make_user, tmp_path, src,
|
||||
{"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240},
|
||||
) # fmt: skip
|
||||
|
||||
red, green = _red_and_green(final)
|
||||
assert (red, green) == (0, 160 * 120)
|
||||
|
||||
|
||||
def test_a_crop_of_square_pixels_is_cut_where_it_was_drawn(db, make_user, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "480x640", "x=120:y=160:w=240:h=320", sar=1)
|
||||
|
||||
final = _cut(
|
||||
db, make_user, tmp_path, src,
|
||||
{"x": 120, "y": 160, "w": 240, "h": 320, "frame_w": 480, "frame_h": 640},
|
||||
) # fmt: skip
|
||||
|
||||
assert _red_and_green(final) == (0, 240 * 320)
|
||||
|
||||
|
||||
def test_a_crop_stored_before_it_carried_its_frame_is_cut_as_it_always_was():
|
||||
spec = edit.normalize_edit_spec({"crop": {"x": 4, "y": 6, "w": 100, "h": 50}})
|
||||
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
|
||||
cmd = edit.build_edit_cmd(Path("in.mp4"), Path("out.mp4"), spec, "mp4")
|
||||
assert cmd[cmd.index("-vf") + 1].startswith("crop=100:50:4:6,")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("frame", [{"frame_w": 0, "frame_h": 240}, {"frame_w": "x", "frame_h": 2}])
|
||||
def test_a_frame_that_cannot_be_read_is_not_divided_by(frame):
|
||||
spec = edit.normalize_edit_spec({"crop": {"x": 4, "y": 6, "w": 100, "h": 50, **frame}})
|
||||
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
|
||||
|
||||
|
||||
def test_a_crop_of_a_join_keeps_what_the_frame_showed(db, make_user, tmp_path, monkeypatch):
|
||||
"""A cut-list joins through `filter_complex`, a crop path of its own."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
|
||||
|
||||
final = _cut(
|
||||
db, make_user, tmp_path, src,
|
||||
{"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240},
|
||||
segments=[{"start_s": 0, "end_s": 0.5}, {"start_s": 1, "end_s": 1.5}], accurate=True,
|
||||
) # fmt: skip
|
||||
|
||||
assert _red_and_green(final) == (0, 160 * 120)
|
||||
|
||||
|
||||
def test_a_crop_of_a_rotated_video_is_cut_where_it_was_drawn(db, make_user, tmp_path, monkeypatch):
|
||||
"""The browser shows a rotated file upright, so the frame was drawn on the upright picture."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
# Stored 320 x 240, green on its left half; shown turned a quarter, 240 x 320.
|
||||
_source(src, "320x240", "x=0:y=0:w=160:h=240", sar=1, rotate=90)
|
||||
|
||||
# Turned counter-clockwise, the stored left half is the shown bottom half.
|
||||
final = _cut(
|
||||
db, make_user, tmp_path, src,
|
||||
{"x": 0, "y": 160, "w": 240, "h": 160, "frame_w": 240, "frame_h": 320},
|
||||
) # fmt: skip
|
||||
|
||||
assert _red_and_green(final) == (0, 240 * 160)
|
||||
|
||||
|
||||
def test_a_clip_ffprobe_cannot_size_gets_the_size_its_crop_cut(
|
||||
db, make_user, tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
|
||||
|
||||
def probe_fails():
|
||||
monkeypatch.setattr(pipeline.probe, "probe_media", lambda _p: (None,) * 5)
|
||||
|
||||
asset = _cut_asset(
|
||||
db, make_user, src,
|
||||
{"crop": {"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240}},
|
||||
then=probe_fails,
|
||||
) # fmt: skip
|
||||
|
||||
# Half the shown width of a 320-wide file: 160 stored, not the 320 the frame counted.
|
||||
assert (asset.width, asset.height) == (160, 120)
|
||||
|
||||
|
||||
def test_a_rotated_clip_ffprobe_cannot_size_gets_the_upright_size_its_crop_cut(
|
||||
db, make_user, tmp_path, monkeypatch
|
||||
):
|
||||
"""The frame was drawn on the upright picture, which is the stored one turned: so is its cut."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
src = tmp_path / "in.mp4"
|
||||
_source(src, "320x240", "x=0:y=0:w=160:h=240", sar=1, rotate=90)
|
||||
|
||||
def probe_fails():
|
||||
monkeypatch.setattr(pipeline.probe, "probe_media", lambda _p: (None,) * 5)
|
||||
|
||||
asset = _cut_asset(
|
||||
db, make_user, src,
|
||||
{"crop": {"x": 0, "y": 160, "w": 240, "h": 160, "frame_w": 240, "frame_h": 320}},
|
||||
then=probe_fails,
|
||||
) # fmt: skip
|
||||
|
||||
# What ffmpeg cut (the rotated test above): 240 x 160, not the stored 320 x 120.
|
||||
assert (asset.width, asset.height) == (240, 160)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key", ["frame_w", "x"])
|
||||
def test_an_infinite_number_in_a_crop_is_dropped_not_a_crash(key):
|
||||
crop = {"x": 4, "y": 6, "w": 100, "h": 50, "frame_w": 640, "frame_h": 240}
|
||||
spec = edit.normalize_edit_spec({"crop": {**crop, key: float("inf")}})
|
||||
if key == "frame_w":
|
||||
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
|
||||
else:
|
||||
assert "crop" not in spec
|
||||
@@ -0,0 +1,197 @@
|
||||
"""The editor's GIF output: one marked range of a library video made into a GIF (downloads/gif), held
|
||||
to the GIF's limits at enqueue, and made by the worker with real ffmpeg."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import clip as clipmod
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import pipeline, service
|
||||
from app.downloads import probe as probemod
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import upload as uploadmod
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
|
||||
needs_ffmpeg = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
|
||||
|
||||
|
||||
def test_a_gif_edit_spec_keeps_its_size_and_is_another_edit_than_the_video():
|
||||
video = editmod.normalize_edit_spec({"trim": {"start_s": 1, "end_s": 5}})
|
||||
gif = editmod.normalize_edit_spec({"trim": {"start_s": 1, "end_s": 5}, "gif": {"side": 320}})
|
||||
assert gif["gif"] == {"side": 320} and "gif" not in video
|
||||
assert editmod.edit_sig(gif) != editmod.edit_sig(video)
|
||||
# A GIF is always made, never stream-copied.
|
||||
assert editmod.needs_reencode(gif)
|
||||
|
||||
|
||||
def _source(db, user, tmp_path, seconds=40):
|
||||
src = tmp_path / "in.mp4"
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", f"testsrc2=size=640x360:rate=10:duration={seconds}",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", str(src),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
return uploadmod.ingest_upload(
|
||||
db, user.id, filename="in.mp4", src_path=src, size=src.stat().st_size
|
||||
)
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
@pytest.mark.parametrize(
|
||||
("spec", "reason"),
|
||||
[
|
||||
# A GIF is one range with an end: no join, no open end.
|
||||
({"segments": [{"start_s": 0, "end_s": 2}, {"start_s": 4, "end_s": 6}]}, "gif_range"),
|
||||
({"trim": {"start_s": 3}}, "gif_range"),
|
||||
({"trim": {"start_s": 0, "end_s": 30.5}}, "gif_too_long"),
|
||||
({"trim": {"start_s": 0, "end_s": 5}, "gif_side": 999}, "bad_gif"),
|
||||
],
|
||||
)
|
||||
def test_a_gif_edit_it_cannot_make_is_refused_with_its_reason(
|
||||
db, make_user, tmp_path, monkeypatch, spec, reason
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
source = _source(db, u, tmp_path)
|
||||
side = spec.pop("gif_side", 480)
|
||||
with pytest.raises(service.EditError) as e:
|
||||
service.enqueue_edit(db, u.id, source, {**spec, "gif": {"side": side}})
|
||||
assert e.value.reason == reason
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_the_edit_route_says_a_gif_too_long_with_its_cap(
|
||||
db, make_user, as_user, tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
c = as_user(u)
|
||||
source = _source(db, u, tmp_path)
|
||||
r = c.post(
|
||||
"/api/downloads/edit",
|
||||
json={
|
||||
"source_job_id": source.id,
|
||||
"edit_spec": {"trim": {"start_s": 0, "end_s": 30.5}, "gif": {"side": 480}},
|
||||
},
|
||||
)
|
||||
assert r.status_code == 400, r.text
|
||||
# The cap goes along: the refusal's sentence says it ("at most 30 seconds").
|
||||
assert r.json()["detail"] == {"code": "edit", "reason": "gif_too_long", "limit_s": 30}
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_gif_edit_of_exactly_thirty_seconds_is_allowed(db, make_user, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
source = _source(db, u, tmp_path)
|
||||
# Ends kept to the millisecond whose float difference lands a hair over 30 s.
|
||||
assert 32.02 - 2.02 > gifmod.MAX_S
|
||||
job = service.enqueue_edit(
|
||||
db, u.id, source, {"trim": {"start_s": 2.02, "end_s": 32.02}, "gif": {"side": 480}}
|
||||
)
|
||||
assert job.edit_spec["gif"] == {"side": 480}
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_gif_is_made_of_a_video_never_of_audio(db, make_user, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
src = tmp_path / "in.m4a"
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "sine=frequency=440:duration=10", "-c:a", "aac", str(src),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
source = uploadmod.ingest_upload(
|
||||
db, u.id, filename="in.m4a", src_path=src, size=src.stat().st_size
|
||||
)
|
||||
assert db.get(MediaAsset, source.asset_id).media_kind == "audio"
|
||||
spec = {"trim": {"start_s": 1, "end_s": 5}}
|
||||
with pytest.raises(service.EditError) as e:
|
||||
service.enqueue_edit(db, u.id, source, {**spec, "gif": {"side": 480}})
|
||||
assert e.value.reason == "gif_not_video"
|
||||
# The same range as audio is an edit as it always was.
|
||||
assert service.enqueue_edit(db, u.id, source, spec).edit_spec["trim"]["end_s"] == 5
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_an_hdr_source_is_tonemapped_before_its_gif_palette(db, make_user, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
src = tmp_path / "hlg.mp4"
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "testsrc2=size=640x360:rate=10:duration=4",
|
||||
"-vf", "setparams=color_trc=arib-std-b67:color_primaries=bt2020:colorspace=bt2020nc",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", str(src),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
source = uploadmod.ingest_upload(
|
||||
db, u.id, filename="hlg.mp4", src_path=src, size=src.stat().st_size
|
||||
)
|
||||
# The upload keeps its HLG tag, as an HDR phone video's does.
|
||||
placed = Path(settings.files_root) / db.get(MediaAsset, source.asset_id).rel_path
|
||||
assert clipmod.is_hdr(probemod.probe_json(placed) or {})
|
||||
job = service.enqueue_edit(
|
||||
db, u.id, source, {"trim": {"start_s": 1, "end_s": 3}, "gif": {"side": 320}}
|
||||
)
|
||||
asset = db.get(MediaAsset, job.asset_id)
|
||||
job.status, asset.status = "running", "downloading"
|
||||
db.commit()
|
||||
seen: dict = {}
|
||||
build = gifmod.build_cmd
|
||||
|
||||
def spy(*a, **k):
|
||||
seen["pre"] = k["pre"]
|
||||
return build(*a, **k)
|
||||
|
||||
monkeypatch.setattr(pipeline.gifmod, "build_cmd", spy)
|
||||
pipeline._process_edit(job.id, asset.id)
|
||||
|
||||
assert "tonemap" in seen["pre"]
|
||||
db.expire_all()
|
||||
assert db.get(DownloadJob, job.id).status == "done"
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_gif_edit_is_made_by_the_worker_and_kept_as_an_image_file(
|
||||
db, make_user, tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "root"))
|
||||
u = make_user()
|
||||
source = _source(db, u, tmp_path, seconds=6)
|
||||
spec = {"trim": {"start_s": 1, "end_s": 3}, "crop": {"x": 0, "y": 0, "w": 320, "h": 360}}
|
||||
job = service.enqueue_edit(db, u.id, source, {**spec, "gif": {"side": 320}})
|
||||
asset = db.get(MediaAsset, job.asset_id)
|
||||
assert asset.media_kind == "file"
|
||||
job.status, asset.status = "running", "downloading" # the worker's claim
|
||||
db.commit()
|
||||
|
||||
pipeline._process_edit(job.id, asset.id)
|
||||
|
||||
db.expire_all()
|
||||
job, asset = db.get(DownloadJob, job.id), db.get(MediaAsset, job.asset_id)
|
||||
assert (job.status, asset.status, asset.container, asset.media_kind) == (
|
||||
"done",
|
||||
"ready",
|
||||
"gif",
|
||||
"file",
|
||||
)
|
||||
placed = tmp_path / "root" / asset.rel_path
|
||||
assert placed.read_bytes()[:6] == b"GIF89a"
|
||||
# The crop's 320x360 has its long side (360) fitted into 320.
|
||||
assert (asset.width, asset.height) == gifmod.size(placed) == (284, 320)
|
||||
assert asset.poster_path and (tmp_path / "root" / asset.poster_path).read_bytes()[:2] == (
|
||||
b"\xff\xd8"
|
||||
)
|
||||
@@ -0,0 +1,412 @@
|
||||
"""SEC-17: the egress gate judges the address a guarded download actually dials.
|
||||
|
||||
Two loopback addresses stand in for the two sides: 127.0.0.2 plays a public host, 127.0.0.1 our own
|
||||
network. The gate under test refuses exactly 127.0.0.1, so every assertion is about the judgement and
|
||||
the wiring, with real sockets, a real yt-dlp and a real ffmpeg — no mocked transport.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import http.server
|
||||
import ipaddress
|
||||
import socket
|
||||
import socketserver
|
||||
import struct
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yt_dlp
|
||||
from yt_dlp.utils import download_range_func
|
||||
|
||||
from app.downloads import adapters, egress, errors, formats
|
||||
|
||||
PUBLIC = "127.0.0.2"
|
||||
INTERNAL = "127.0.0.1"
|
||||
|
||||
|
||||
def _forbid_internal(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
return str(ip) == INTERNAL
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gate():
|
||||
g = egress.start_gate(forbidden=_forbid_internal)
|
||||
yield g
|
||||
g.shutdown()
|
||||
g.server_close()
|
||||
|
||||
|
||||
class _Listener:
|
||||
"""A TCP listener that only counts who connected — the internal service under attack."""
|
||||
|
||||
def __init__(self, host: str = INTERNAL, port: int = 0):
|
||||
self.hits = 0
|
||||
self.sock = socket.socket()
|
||||
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
self.sock.bind((host, port))
|
||||
self.sock.listen()
|
||||
self.port = self.sock.getsockname()[1]
|
||||
threading.Thread(target=self._serve, daemon=True).start()
|
||||
|
||||
def _serve(self):
|
||||
while True:
|
||||
try:
|
||||
conn, _ = self.sock.accept()
|
||||
except OSError:
|
||||
return
|
||||
self.hits += 1
|
||||
with contextlib.suppress(OSError):
|
||||
conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
|
||||
conn.close()
|
||||
|
||||
def close(self):
|
||||
self.sock.close()
|
||||
|
||||
|
||||
def _public_server(routes: dict, port: int = 0, directory: Path | None = None):
|
||||
"""An HTTP server on the 'public' address: `routes` maps a path to (status, headers); anything
|
||||
else is served from `directory`."""
|
||||
|
||||
class H(http.server.SimpleHTTPRequestHandler):
|
||||
def __init__(self, *a, **k):
|
||||
super().__init__(*a, directory=str(directory or "."), **k)
|
||||
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
if self.path in routes:
|
||||
status, headers, body = routes[self.path]
|
||||
self.send_response(status)
|
||||
for k, v in headers.items():
|
||||
self.send_header(k, v)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
return
|
||||
super().do_GET()
|
||||
|
||||
srv = socketserver.ThreadingTCPServer((PUBLIC, port), H)
|
||||
srv.daemon_threads = True
|
||||
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||
return srv
|
||||
|
||||
|
||||
def _guarded_opts(tmp_path: Path, gate_url: str, **extra) -> dict:
|
||||
opts = formats.build_ydl_opts({}, str(tmp_path / "%(id)s.%(ext)s"), lambda d: None, egress=True)
|
||||
opts.update(retries=0, fragment_retries=0, extractor_retries=0, socket_timeout=5)
|
||||
opts["proxy"] = gate_url # the test's gate (it lets 127.0.0.2 through), not the worker's
|
||||
opts.update(extra)
|
||||
return opts
|
||||
|
||||
|
||||
def _download_error(opts: dict, url: str) -> str:
|
||||
with pytest.raises(yt_dlp.utils.DownloadError) as exc, egress.open_ydl(opts) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
return str(exc.value)
|
||||
|
||||
|
||||
# --- the judgement ---------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_name_with_any_forbidden_answer_is_refused():
|
||||
def resolve(answers):
|
||||
return lambda host, port, *a: [(2, 1, 6, "", (ip, port)) for ip in answers]
|
||||
|
||||
assert egress.vetted_address(
|
||||
"x", 80, resolve=resolve([PUBLIC]), forbidden=_forbid_internal
|
||||
) == (
|
||||
PUBLIC,
|
||||
80,
|
||||
)
|
||||
assert (
|
||||
egress.vetted_address(
|
||||
"x", 80, resolve=resolve([PUBLIC, INTERNAL]), forbidden=_forbid_internal
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
def no_answer(*a):
|
||||
raise socket.gaierror("no such name")
|
||||
|
||||
assert egress.vetted_address("x", 80, resolve=no_answer, forbidden=_forbid_internal) is None
|
||||
|
||||
|
||||
def test_the_real_judgement_refuses_our_own_networks():
|
||||
# The worker's gate uses urlguard's judgement unchanged — the same one the enqueue check uses.
|
||||
for addr in ("127.0.0.1", "10.1.2.3", "172.18.0.5", "192.168.1.1", "169.254.169.254"):
|
||||
assert egress.vetted_address(addr, 80, forbidden=egress.urlguard.is_forbidden) is None, addr
|
||||
assert egress.vetted_address("93.184.215.14", 80, forbidden=egress.urlguard.is_forbidden)
|
||||
|
||||
|
||||
def test_the_gate_dials_the_address_it_vetted_not_a_second_answer(gate):
|
||||
# DNS rebinding: a name that answers public for the check and private for the fetch. The gate
|
||||
# resolves once per connection and dials that answer.
|
||||
public = _public_server({"/": (200, {}, b"public")})
|
||||
internal = _Listener(INTERNAL, public.server_address[1])
|
||||
calls = []
|
||||
|
||||
def rebinding(host, port, *a):
|
||||
calls.append(host)
|
||||
ip = PUBLIC if len(calls) == 1 else INTERNAL
|
||||
return [(2, 1, 6, "", (ip, port))]
|
||||
|
||||
gate.resolve = rebinding
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", gate.server_address[1])) as s:
|
||||
s.sendall(
|
||||
f"GET http://rebind.test:{public.server_address[1]}/ HTTP/1.1\r\n"
|
||||
f"Host: rebind.test\r\n\r\n".encode()
|
||||
)
|
||||
answer = b"".join(iter(lambda: s.recv(4096), b""))
|
||||
assert answer.endswith(b"public")
|
||||
assert calls == ["rebind.test"]
|
||||
assert internal.hits == 0
|
||||
finally:
|
||||
public.shutdown()
|
||||
internal.close()
|
||||
|
||||
|
||||
def test_a_tunnel_into_the_network_is_refused_before_it_is_dialled(gate):
|
||||
internal = _Listener()
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", gate.server_address[1])) as s:
|
||||
s.sendall(f"CONNECT {INTERNAL}:{internal.port} HTTP/1.1\r\n\r\n".encode())
|
||||
answer = s.recv(4096)
|
||||
assert answer.startswith(b"HTTP/1.1 403")
|
||||
assert internal.hits == 0
|
||||
finally:
|
||||
internal.close()
|
||||
|
||||
|
||||
def test_a_client_that_hangs_up_mid_transfer_is_not_an_error(gate, monkeypatch):
|
||||
# yt-dlp closes a connection once it has read what it needed; seen live as a traceback the gate
|
||||
# printed for every such download.
|
||||
errors_seen = []
|
||||
monkeypatch.setattr(gate, "handle_error", lambda *a: errors_seen.append(a))
|
||||
public = _public_server({"/big": (200, {}, b"x" * 4_000_000)})
|
||||
try:
|
||||
s = socket.create_connection(("127.0.0.1", gate.server_address[1]))
|
||||
s.sendall(f"GET http://{PUBLIC}:{public.server_address[1]}/big HTTP/1.1\r\n\r\n".encode())
|
||||
s.recv(1024)
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
|
||||
s.close() # a reset, not a polite close
|
||||
time.sleep(0.5)
|
||||
assert errors_seen == []
|
||||
finally:
|
||||
public.shutdown()
|
||||
|
||||
|
||||
# --- a real yt-dlp through it ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_redirect_into_the_network_reaches_it_unguarded_and_not_through_the_gate(tmp_path, gate):
|
||||
internal = _Listener()
|
||||
public = _public_server(
|
||||
{"/v.mp4": (302, {"Location": f"http://{INTERNAL}:{internal.port}/v.mp4"}, b"")}
|
||||
)
|
||||
url = f"http://{PUBLIC}:{public.server_address[1]}/v.mp4"
|
||||
try:
|
||||
# The hole, as it was: yt-dlp follows the 3xx into our network.
|
||||
bare = {"outtmpl": str(tmp_path / "bare" / "%(id)s.%(ext)s"), "quiet": True, "retries": 0}
|
||||
with contextlib.suppress(yt_dlp.utils.DownloadError), yt_dlp.YoutubeDL(bare) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
assert internal.hits >= 1
|
||||
before = internal.hits
|
||||
|
||||
_download_error(_guarded_opts(tmp_path, gate.url), url)
|
||||
assert internal.hits == before
|
||||
finally:
|
||||
public.shutdown()
|
||||
internal.close()
|
||||
|
||||
|
||||
def test_a_live_and_a_dead_internal_port_leave_the_same_text(tmp_path, gate):
|
||||
live = _Listener()
|
||||
dead = _Listener()
|
||||
dead_port = dead.port
|
||||
dead.close()
|
||||
public = _public_server(
|
||||
{
|
||||
"/live.mp4": (302, {"Location": f"http://{INTERNAL}:{live.port}/x"}, b""),
|
||||
"/dead.mp4": (302, {"Location": f"http://{INTERNAL}:{dead_port}/x"}, b""),
|
||||
}
|
||||
)
|
||||
base = f"http://{PUBLIC}:{public.server_address[1]}"
|
||||
try:
|
||||
stored = []
|
||||
for name in ("live", "dead"):
|
||||
msg = _download_error(_guarded_opts(tmp_path / name, gate.url), f"{base}/{name}.mp4")
|
||||
stored.append(errors.user_error_text("url", msg, errors.classify(msg)))
|
||||
assert stored[0] == stored[1]
|
||||
assert live.hits == 0
|
||||
finally:
|
||||
public.shutdown()
|
||||
live.close()
|
||||
|
||||
|
||||
def test_a_format_fetched_past_the_gate_is_never_chosen(tmp_path):
|
||||
# ffmpeg speaks rtmp, rtsp, tcp… directly, where no HTTP proxy is asked. A guarded download keeps
|
||||
# only the formats fetched over http(s).
|
||||
opts = _guarded_opts(tmp_path, "http://127.0.0.1:9")
|
||||
info = {
|
||||
"id": "x",
|
||||
"title": "x",
|
||||
"extractor": "generic",
|
||||
"extractor_key": "Generic",
|
||||
"webpage_url": f"http://{PUBLIC}/page",
|
||||
"formats": [
|
||||
{"format_id": "rtmp", "url": f"rtmp://{INTERNAL}/live", "ext": "flv", "height": 1080},
|
||||
{"format_id": "web", "url": f"http://{PUBLIC}/a.mp4", "ext": "mp4", "height": 360},
|
||||
],
|
||||
}
|
||||
with egress.open_ydl(opts) as ydl:
|
||||
chosen = ydl.process_ie_result(dict(info), download=False)
|
||||
assert chosen["format_id"] == "web"
|
||||
|
||||
only_rtmp = dict(info, formats=[info["formats"][0]])
|
||||
with (
|
||||
pytest.raises(yt_dlp.utils.YoutubeDLError, match="Requested format is not available"),
|
||||
egress.open_ydl(opts) as ydl,
|
||||
):
|
||||
ydl.process_ie_result(only_rtmp, download=False)
|
||||
|
||||
|
||||
def test_a_format_yt_dlp_fetches_itself_is_kept():
|
||||
# f4m, ism and the DASH generator fetch through yt-dlp's own urlopen; a bare m3u8 goes to the
|
||||
# native HLS fetcher under hls_prefer_native (set for every guarded download). All ask the gate.
|
||||
kept = [
|
||||
{"format_id": p, "url": f"https://{PUBLIC}/{p}", "protocol": p}
|
||||
for p in ("f4m", "ism", "m3u8", "m3u8_native", "http_dash_segments_generator", "https")
|
||||
]
|
||||
raw = {"format_id": "rtmp", "url": f"rtmp://{PUBLIC}/live", "protocol": "rtmp"}
|
||||
_, info = egress.HttpOnlyFormats().run({"formats": [*kept, raw]})
|
||||
assert [f["format_id"] for f in info["formats"]] == [f["format_id"] for f in kept]
|
||||
|
||||
|
||||
def test_a_single_url_result_past_the_gate_loses_its_url():
|
||||
_, info = egress.HttpOnlyFormats().run({"url": f"rtmp://{INTERNAL}/live"})
|
||||
assert info["url"] is None
|
||||
_, info = egress.HttpOnlyFormats().run({"url": f"https://{PUBLIC}/a.mp4"})
|
||||
assert info["url"] == f"https://{PUBLIC}/a.mp4"
|
||||
|
||||
|
||||
def test_the_format_filter_says_no_phase_of_its_own(tmp_path):
|
||||
# It is a post-processor only to run before the format choice; the job's postprocessor hook
|
||||
# must not see it (it painted every guarded job "processing" before a byte was fetched).
|
||||
calls: list[dict] = []
|
||||
opts = _guarded_opts(tmp_path, "http://127.0.0.1:9", postprocessor_hooks=[calls.append])
|
||||
with egress.open_ydl(opts) as ydl:
|
||||
ydl.run_all_pps("pre_process", {"id": "x", "formats": []})
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_an_allowed_tunnel_carries_bytes_both_ways(gate):
|
||||
# The CONNECT path a site served over https takes: a 200, then the bytes the client sent after
|
||||
# the head (in the same packet here) reach the far end, and its answer comes back.
|
||||
far = socket.socket()
|
||||
far.bind((PUBLIC, 0))
|
||||
far.listen()
|
||||
|
||||
def echo():
|
||||
conn, _ = far.accept()
|
||||
with conn:
|
||||
conn.sendall(b"PONG:" + conn.recv(64))
|
||||
|
||||
threading.Thread(target=echo, daemon=True).start()
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", gate.server_address[1]), timeout=5) as s:
|
||||
s.sendall(f"CONNECT {PUBLIC}:{far.getsockname()[1]} HTTP/1.1\r\n\r\nPING".encode())
|
||||
got = b""
|
||||
while b"PONG:PING" not in got:
|
||||
chunk = s.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
got += chunk
|
||||
assert got.startswith(b"HTTP/1.1 200")
|
||||
assert got.endswith(b"PONG:PING")
|
||||
finally:
|
||||
far.close()
|
||||
|
||||
|
||||
def test_the_workers_own_gate_refuses_our_network():
|
||||
# Every other gate here is built with a test rule; this is the one `proxy_url()` starts.
|
||||
internal = _Listener()
|
||||
try:
|
||||
port = int(egress.proxy_url().rsplit(":", 1)[1])
|
||||
with socket.create_connection(("127.0.0.1", port), timeout=5) as s:
|
||||
s.sendall(f"CONNECT {INTERNAL}:{internal.port} HTTP/1.1\r\n\r\n".encode())
|
||||
assert s.recv(4096).startswith(b"HTTP/1.1 403")
|
||||
assert internal.hits == 0
|
||||
finally:
|
||||
internal.close()
|
||||
|
||||
|
||||
def _clip_hits(tmp_path: Path, segment: str, proxy: str) -> int:
|
||||
"""How many connections the internal listener takes while a clip (`download_ranges`, which
|
||||
yt-dlp hands to ffmpeg even for an HLS format) is cut from a playlist whose one segment is
|
||||
`segment` — formatted with the listener's address."""
|
||||
internal = _Listener()
|
||||
playlist = (
|
||||
"#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-TARGETDURATION:2\n#EXTINF:2.0,\n"
|
||||
+ segment.format(host=INTERNAL, port=internal.port)
|
||||
+ "\n#EXT-X-ENDLIST\n"
|
||||
).encode()
|
||||
public = _public_server(
|
||||
{"/p.m3u8": (200, {"Content-Type": "application/vnd.apple.mpegurl"}, playlist)}
|
||||
)
|
||||
try:
|
||||
opts = _guarded_opts(tmp_path, proxy, download_ranges=download_range_func([], [(0, 1)]))
|
||||
with contextlib.suppress(yt_dlp.utils.DownloadError), egress.open_ydl(opts) as ydl:
|
||||
ydl.extract_info(f"http://{PUBLIC}:{public.server_address[1]}/p.m3u8", download=True)
|
||||
return internal.hits
|
||||
finally:
|
||||
public.shutdown()
|
||||
internal.close()
|
||||
|
||||
|
||||
def test_a_clip_cut_by_ffmpeg_reaches_a_segment_only_through_the_gate(tmp_path, gate):
|
||||
# Without the gate, ffmpeg fetches a hostile playlist's segment from inside our network — the
|
||||
# control that shows this path is real. Through the gate it is never dialled.
|
||||
segment = "http://{host}:{port}/s.ts"
|
||||
assert _clip_hits(tmp_path / "bare", segment, proxy="") >= 1
|
||||
assert _clip_hits(tmp_path / "gated", segment, proxy=gate.url) == 0
|
||||
|
||||
|
||||
def test_ffmpeg_itself_refuses_a_raw_tcp_segment(tmp_path):
|
||||
# A boundary, not our code: the HTTP proxy cannot see a tcp:// segment, and measured, ffmpeg's
|
||||
# HLS reader does not open one from an http playlist. This fails if an ffmpeg upgrade starts to.
|
||||
assert _clip_hits(tmp_path, "tcp://{host}:{port}/", proxy="") == 0
|
||||
|
||||
|
||||
# --- the wiring ------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_only_youtube_downloads_skip_the_gate():
|
||||
assert adapters.SourceAdapter.egress_guarded is True
|
||||
assert adapters.resolve_adapter("url", "https://www.facebook.com/x").egress_guarded is True
|
||||
assert adapters.resolve_adapter("url", "https://vimeo.com/1").egress_guarded is True
|
||||
assert adapters.resolve_adapter("youtube", "dQw4w9WgXcQ").egress_guarded is False
|
||||
|
||||
|
||||
def test_a_guarded_download_goes_through_the_gate_whatever_its_adapter_adds():
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, extra={"proxy": "", "hls_prefer_native": False}, egress=True
|
||||
)
|
||||
assert opts["proxy"] == egress.proxy_url()
|
||||
assert opts["hls_prefer_native"] is True
|
||||
assert opts[egress.GUARDED] is True
|
||||
assert "proxy" not in formats.build_ydl_opts({}, "/tmp/x", lambda d: None, egress=False)
|
||||
|
||||
|
||||
def test_every_yt_dlp_this_app_opens_goes_through_open_ydl():
|
||||
app = Path(egress.__file__).resolve().parents[1]
|
||||
offenders = [
|
||||
f"{p.relative_to(app)}:{n}"
|
||||
for p in sorted(app.rglob("*.py"))
|
||||
if p.name != "egress.py"
|
||||
for n, line in enumerate(p.read_text(encoding="utf-8").splitlines(), 1)
|
||||
if "YoutubeDL(" in line
|
||||
]
|
||||
assert offenders == []
|
||||
@@ -18,7 +18,7 @@ import yt_dlp
|
||||
from yt_dlp.networking.exceptions import TransportError
|
||||
from yt_dlp.networking.impersonate import ImpersonateTarget
|
||||
|
||||
from app.downloads import adapters, pipeline
|
||||
from app.downloads import adapters, egress, pipeline
|
||||
from app.downloads import errors as dlerrors
|
||||
|
||||
_URL = "https://www.facebook.com/reel/806379462330494/?fs=e&fs=e"
|
||||
@@ -73,6 +73,9 @@ def ydl(monkeypatch):
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
@@ -157,6 +160,16 @@ def test_a_cookie_group_post_downloads_the_posts_own_video(ydl, tmp_path):
|
||||
# Never the post link: that is where the feed video comes from.
|
||||
assert web["extracted"] == [watch, watch]
|
||||
assert [o.get("impersonate") for o in seen[1:]] == [None, _CHROME]
|
||||
# The post page read and both attempts all go through the egress gate.
|
||||
assert all(o.get(egress.GUARDED) and o["proxy"] == egress.proxy_url() for o in seen), seen
|
||||
|
||||
|
||||
def test_a_youtube_download_skips_the_gate(ydl, tmp_path):
|
||||
seen, outcomes, _ = ydl
|
||||
outcomes += [{"id": "dQw4w9WgXcQ"}]
|
||||
yt = adapters.resolve_adapter("youtube", "dQw4w9WgXcQ")
|
||||
pipeline._extract_with_fallback(1, 1, "https://youtu.be/dQw4w9WgXcQ", {}, tmp_path, yt)
|
||||
assert seen and not any(o.get(egress.GUARDED) or o.get("proxy") for o in seen)
|
||||
|
||||
|
||||
def test_a_group_post_without_a_single_video_fails_saying_so(ydl, tmp_path):
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
"""A Facebook download's title when the signed-in read gives only a placeholder.
|
||||
|
||||
Measured on prod (2026-11-01): every Facebook download since 2026-10-28 is titled "Facebook" — the
|
||||
signed-in read (the owner's cookie) now gives that placeholder and NO description, so the adapter had
|
||||
no post text to fall back on, and the download name copied the placeholder too. A signed-out read of
|
||||
the same reel (yt-dlp 2026.08.19, as on prod) still carries the post's text in its description.
|
||||
"""
|
||||
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.downloads import adapters, backfill, egress, pipeline
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
|
||||
_URL = "https://www.facebook.com/reel/1860682468422402/"
|
||||
# The page yt-dlp says it read for it (`webpage_url`); a group post's link resolves the same way.
|
||||
_PAGE = "https://www.facebook.com/watch/?v=1860682468422402"
|
||||
# The measured signed-out read of that reel.
|
||||
_PUBLIC = {
|
||||
"title": "163K reactions · 14K shares | sorry not helping #workhumor #officelife | Work dot exe",
|
||||
"description": "sorry not helping\n\n#workhumor #officelife",
|
||||
"uploader": "Work dot exe",
|
||||
}
|
||||
|
||||
|
||||
def _fake_ydl(monkeypatch, outcome):
|
||||
"""yt-dlp faked at its `YoutubeDL` boundary: records each read (its options and its `url`),
|
||||
answers `outcome`."""
|
||||
seen: list[dict] = []
|
||||
|
||||
class FakeYDL:
|
||||
def __init__(self, opts):
|
||||
self.opts = opts
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
def extract_info(self, url, download):
|
||||
seen.append({**self.opts, "url": url})
|
||||
return outcome
|
||||
|
||||
monkeypatch.setattr("yt_dlp.YoutubeDL", FakeYDL)
|
||||
return seen
|
||||
|
||||
|
||||
def test_a_placeholder_with_nothing_behind_it_takes_the_signed_out_pages_text(monkeypatch):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
fb = adapters.FacebookAdapter()
|
||||
|
||||
info = backfill.public_title_info({"title": "Facebook", "description": None}, _URL, fb, 7)
|
||||
|
||||
assert fb.title(info) == "sorry not helping"
|
||||
# One metadata-only read, and signed out: no cookie goes with it.
|
||||
assert len(seen) == 1
|
||||
assert seen[0]["skip_download"] is True
|
||||
assert not seen[0].get("cookiefile")
|
||||
assert seen[0][egress.GUARDED] is True and seen[0]["proxy"] == egress.proxy_url()
|
||||
|
||||
|
||||
def test_a_real_title_reads_nothing_more(monkeypatch):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
info = {"title": "Funny Dark Humor Dad Joke!", "description": None}
|
||||
|
||||
assert backfill.public_title_info(info, _URL, adapters.FacebookAdapter(), 7) is info
|
||||
assert seen == []
|
||||
|
||||
|
||||
def test_a_placeholder_with_the_posts_text_behind_it_reads_nothing_more(monkeypatch):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
info = {"title": "Video", "description": "Office jokes, part two"}
|
||||
|
||||
assert backfill.public_title_info(info, _URL, adapters.FacebookAdapter(), 7) is info
|
||||
assert seen == []
|
||||
|
||||
|
||||
def test_a_placeholder_the_signed_out_page_cannot_better_stays(monkeypatch):
|
||||
_fake_ydl(monkeypatch, {"title": "Facebook", "description": None})
|
||||
fb = adapters.FacebookAdapter()
|
||||
|
||||
info = backfill.public_title_info({"title": "Facebook", "description": None}, _URL, fb, 7)
|
||||
|
||||
assert fb.title(info) == "Facebook"
|
||||
|
||||
|
||||
def test_another_source_is_never_read_again(monkeypatch):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
info = {"title": "Video", "description": None}
|
||||
|
||||
assert backfill.public_title_info(info, _URL, adapters.SourceAdapter(), 7) is info
|
||||
assert seen == []
|
||||
|
||||
|
||||
def test_the_backfill_retitles_a_placeholder_download_and_the_names_that_repeated_it(
|
||||
db, make_user, monkeypatch
|
||||
):
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
owner, other = make_user(), make_user()
|
||||
asset = MediaAsset(
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
format_sig="fb",
|
||||
status="ready",
|
||||
rel_path="facebook/x.mp4",
|
||||
poster_path="facebook/x.jpg",
|
||||
uploader_url="https://www.facebook.com/1",
|
||||
title="Facebook",
|
||||
ref_count=2,
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
copied = DownloadJob(
|
||||
user_id=owner.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="done",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
display_name="Facebook",
|
||||
)
|
||||
renamed = DownloadJob(
|
||||
user_id=other.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="done",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
display_name="Office jokes",
|
||||
)
|
||||
db.add_all([copied, renamed])
|
||||
db.commit()
|
||||
|
||||
assert backfill.run()["retitled"] == 1
|
||||
assert len(seen) == 1
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(MediaAsset, asset.id).title == "sorry not helping"
|
||||
assert db.get(DownloadJob, copied.id).display_name == "sorry not helping"
|
||||
assert db.get(DownloadJob, renamed.id).display_name == "Office jokes"
|
||||
# Run again: nothing left to retitle, and no page read for it.
|
||||
assert backfill.run()["retitled"] == 0
|
||||
assert len(seen) == 1
|
||||
|
||||
|
||||
def test_a_name_given_while_the_backfill_runs_stays(db, make_user, monkeypatch):
|
||||
# The owner renames the download while the backfill reads the page: the owner's name wins, it is
|
||||
# not overwritten by what was read before it.
|
||||
_fake_ydl(monkeypatch, _PUBLIC)
|
||||
owner = make_user()
|
||||
asset = MediaAsset(
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
format_sig="fb",
|
||||
status="ready",
|
||||
rel_path="facebook/x.mp4",
|
||||
uploader_url="https://www.facebook.com/1",
|
||||
title="Facebook",
|
||||
ref_count=1,
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
job = DownloadJob(
|
||||
user_id=owner.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="done",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
display_name="Facebook",
|
||||
)
|
||||
db.add(job)
|
||||
db.commit()
|
||||
|
||||
read_page = backfill.public_title_info
|
||||
|
||||
def renamed_while_reading(*args):
|
||||
with SessionLocal() as other:
|
||||
other.execute(
|
||||
update(DownloadJob)
|
||||
.where(DownloadJob.id == job.id)
|
||||
.values(display_name="Office jokes")
|
||||
)
|
||||
other.commit()
|
||||
return read_page(*args)
|
||||
|
||||
monkeypatch.setattr(backfill, "public_title_info", renamed_while_reading)
|
||||
monkeypatch.setattr(backfill.editmod, "ensure_poster", lambda a, root: None)
|
||||
|
||||
assert backfill.run()["retitled"] == 1
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(DownloadJob, job.id).display_name == "Office jokes"
|
||||
|
||||
|
||||
def test_the_poster_step_holds_no_row_lock(db, make_user, monkeypatch):
|
||||
# It can cut a frame for 30 s. A delete locks the asset and then the job: with the job held here
|
||||
# and the asset still to write, that is a deadlock, and a rename waits out the whole cut.
|
||||
_fake_ydl(monkeypatch, _PUBLIC)
|
||||
owner = make_user()
|
||||
asset = MediaAsset(
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
format_sig="fb",
|
||||
status="ready",
|
||||
rel_path="facebook/x.mp4",
|
||||
uploader_url="https://www.facebook.com/1",
|
||||
title="Facebook",
|
||||
ref_count=1,
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
job = DownloadJob(
|
||||
user_id=owner.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="done",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
display_name="Facebook",
|
||||
)
|
||||
db.add(job)
|
||||
db.commit()
|
||||
asset_id, job_id = asset.id, job.id
|
||||
|
||||
def poster_finds_rows_free(a, root):
|
||||
with SessionLocal() as other:
|
||||
other.execute(
|
||||
select(MediaAsset).where(MediaAsset.id == asset_id).with_for_update(nowait=True)
|
||||
)
|
||||
other.execute(
|
||||
select(DownloadJob).where(DownloadJob.id == job_id).with_for_update(nowait=True)
|
||||
)
|
||||
other.rollback()
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(backfill.editmod, "ensure_poster", poster_finds_rows_free)
|
||||
|
||||
assert backfill.run()["retitled"] == 1
|
||||
|
||||
|
||||
def test_a_download_titled_facebook_is_named_from_the_signed_out_page(
|
||||
db, make_user, monkeypatch, tmp_path
|
||||
):
|
||||
# The whole download path with yt-dlp and the media tools faked: the signed-in read gives only
|
||||
# the placeholder, and the file and the download's name get the post's text.
|
||||
seen = _fake_ydl(monkeypatch, _PUBLIC)
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
|
||||
monkeypatch.setattr(pipeline, "_ensure_phone_safe", lambda _j, produced, *_a: produced)
|
||||
monkeypatch.setattr(
|
||||
pipeline.probe, "resolve_media_facts", lambda produced, info: ("h264", "aac", 720, 1280, 9)
|
||||
)
|
||||
monkeypatch.setattr(pipeline.editmod, "thumbnail_to_jpg", lambda *a: None)
|
||||
monkeypatch.setattr(pipeline.editmod, "ensure_poster", lambda *a: None)
|
||||
owner = make_user()
|
||||
asset = MediaAsset(
|
||||
source_kind="url", source_ref=_URL, format_sig="fb", status="downloading", ref_count=1
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
job = DownloadJob(
|
||||
user_id=owner.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref=_URL,
|
||||
status="running",
|
||||
queue_pos=0,
|
||||
profile_snapshot={},
|
||||
)
|
||||
db.add(job)
|
||||
db.commit()
|
||||
|
||||
def signed_in_read(job_id, asset_id, url, spec, staging, adapter, **kw):
|
||||
f = staging / "1860682468422402.mp4"
|
||||
f.write_bytes(b"v" * 100)
|
||||
return {
|
||||
"id": "1860682468422402",
|
||||
"title": "Facebook",
|
||||
"uploader": "Work dot exe",
|
||||
"webpage_url": _PAGE,
|
||||
"requested_downloads": [{"filepath": str(f)}],
|
||||
}
|
||||
|
||||
monkeypatch.setattr(pipeline, "_extract_with_fallback", signed_in_read)
|
||||
pipeline._download(job.id, asset.id, "url", _URL, {}, owner.id)
|
||||
|
||||
# The signed-out read goes to the page yt-dlp read, not the link queued: a group post's link
|
||||
# resolves to its video, and signed out the post itself is the login wall.
|
||||
assert [r["url"] for r in seen] == [_PAGE]
|
||||
db.expire_all()
|
||||
assert db.get(DownloadJob, job.id).status == "done"
|
||||
assert db.get(MediaAsset, asset.id).title == "sorry not helping"
|
||||
assert db.get(DownloadJob, job.id).display_name == "sorry not helping"
|
||||
@@ -359,6 +359,9 @@ def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkey
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
@@ -367,8 +370,8 @@ def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkey
|
||||
f"ERROR: [youtube] dQw4w9WgXcQ: Video unavailable ({target})"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
|
||||
assert backfill._extract_meta(url, adapters.YouTubeAdapter(), 11) is None
|
||||
monkeypatch.setattr("yt_dlp.YoutubeDL", _Refusing)
|
||||
assert backfill.extract_meta(url, adapters.YouTubeAdapter(), 11) is None
|
||||
|
||||
lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
|
||||
assert lines and all("dQw4w9WgXcQ" not in json.dumps(o) for o in lines)
|
||||
@@ -386,14 +389,17 @@ def test_the_metadata_backfill_takes_out_a_generic_name_whatever_it_holds(sink,
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
def extract_info(self, target, download):
|
||||
raise yt_dlp.utils.DownloadError("ERROR: [generic] Our Trip: Day 1: Unable to download")
|
||||
|
||||
monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
|
||||
assert backfill._extract_meta(url, adapters.SourceAdapter(), 11) is None
|
||||
monkeypatch.setattr("yt_dlp.YoutubeDL", _Refusing)
|
||||
assert backfill.extract_meta(url, adapters.SourceAdapter(), 11) is None
|
||||
|
||||
lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
|
||||
assert lines and all("Day 1" not in json.dumps(o) for o in lines)
|
||||
@@ -449,6 +455,9 @@ def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, t
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def add_post_processor(self, pp, when="post_process"):
|
||||
pass # the egress format filter; nothing here to filter
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
@@ -463,6 +472,7 @@ def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, t
|
||||
resolve_url=lambda target, opts: target,
|
||||
ydl_opts_extra=lambda clients: {},
|
||||
retriable=lambda text: True,
|
||||
egress_guarded=False,
|
||||
)
|
||||
monkeypatch.setattr(pipeline.yt_dlp, "YoutubeDL", _YDL)
|
||||
monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: None)
|
||||
|
||||
@@ -150,6 +150,7 @@ def test_a_password_folder_link_shows_nothing_until_unlocked(shelf, client):
|
||||
assert client.get(f"/api/public/watch/{token}").json() == {
|
||||
"needs_password": True,
|
||||
"kind": "folder",
|
||||
"video": False,
|
||||
}
|
||||
bare = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/file"
|
||||
assert client.get(bare).status_code == 403
|
||||
@@ -312,7 +313,7 @@ def test_a_password_file_link_says_it_opens_a_file(shelf, client):
|
||||
.json()
|
||||
)
|
||||
meta = client.get(f"/api/public/watch/{made['token']}").json()
|
||||
assert meta == {"needs_password": True, "kind": "file"}
|
||||
assert meta == {"needs_password": True, "kind": "file", "video": True}
|
||||
|
||||
|
||||
def test_a_file_whose_bytes_are_gone_is_not_offered(shelf, client, db):
|
||||
|
||||
@@ -15,16 +15,20 @@ from app.downloads import formats
|
||||
|
||||
|
||||
def _opts(spec):
|
||||
return formats.build_ydl_opts(spec, "/tmp/%(id)s.%(ext)s", lambda d: None)
|
||||
return formats.build_ydl_opts(spec, "/tmp/%(id)s.%(ext)s", lambda d: None, egress=False)
|
||||
|
||||
|
||||
def test_adapter_extra_opts_are_merged_in():
|
||||
# The SourceAdapter contributes source-specific opts (e.g. YouTube's extractor_args) via `extra`.
|
||||
extra = {"extractor_args": {"youtube": {"player_client": ["web"]}}}
|
||||
opts = formats.build_ydl_opts({}, "/tmp/%(id)s.%(ext)s", lambda d: None, extra=extra)
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "/tmp/%(id)s.%(ext)s", lambda d: None, extra=extra, egress=False
|
||||
)
|
||||
assert opts["extractor_args"]["youtube"]["player_client"] == ["web"]
|
||||
# No extra → no extractor_args (a generic source contributes nothing).
|
||||
assert "extractor_args" not in formats.build_ydl_opts({}, "/tmp/x", lambda d: None)
|
||||
assert "extractor_args" not in formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, egress=False
|
||||
)
|
||||
|
||||
|
||||
def test_mp4_default_selects_avc1_with_progressive_fallback():
|
||||
@@ -113,7 +117,7 @@ def test_a_playlist_url_can_only_yield_one_item():
|
||||
|
||||
|
||||
def test_the_remaining_quota_becomes_a_hard_size_ceiling():
|
||||
opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000)
|
||||
opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000, egress=False)
|
||||
assert opts["max_filesize"] == 5_000_000
|
||||
|
||||
|
||||
@@ -122,7 +126,7 @@ def test_no_budget_means_no_size_filter():
|
||||
# enqueue, not an impossible yt-dlp filter to express here.
|
||||
for budget in (None, 0, -1):
|
||||
assert "max_filesize" not in formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, max_bytes=budget
|
||||
{}, "/tmp/x", lambda d: None, max_bytes=budget, egress=False
|
||||
)
|
||||
|
||||
|
||||
@@ -130,7 +134,12 @@ def test_a_source_adapter_cannot_raise_the_size_ceiling():
|
||||
# The adapter merge happens after max_filesize is set, so this pins that no adapter option can
|
||||
# widen a storage cap — the mirror of the cookiefile rule, in the opposite direction.
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, extra={"max_filesize": 999_999_999}, max_bytes=5_000_000
|
||||
{},
|
||||
"/tmp/x",
|
||||
lambda d: None,
|
||||
extra={"max_filesize": 999_999_999},
|
||||
max_bytes=5_000_000,
|
||||
egress=False,
|
||||
)
|
||||
assert opts["max_filesize"] == 5_000_000
|
||||
|
||||
@@ -197,7 +206,7 @@ def test_yt_dlp_refuses_a_file_over_the_cap_without_raising_and_the_watch_sees_i
|
||||
for cap, refused in ((1_000, True), (10_000_000, False)):
|
||||
out = tmp_path / f"out-{cap}"
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, str(out / "%(id)s.%(ext)s"), lambda d: None, max_bytes=cap
|
||||
{}, str(out / "%(id)s.%(ext)s"), lambda d: None, max_bytes=cap, egress=False
|
||||
)
|
||||
with formats.watching_size_cap() as over, yt_dlp.YoutubeDL(opts) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
"""The GIF output (downloads/gif): its limits, the one ffmpeg command every GIF goes through, and what
|
||||
that command really makes — run with real ffmpeg, since the size, the palette and the frame are the
|
||||
encoder's, not the argv's."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.downloads import gif as gifmod
|
||||
from app.downloads import mediakind
|
||||
from app.downloads import probe as probemod
|
||||
from app.downloads.edit import INPUT_GUARD
|
||||
|
||||
needs_ffmpeg = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
|
||||
|
||||
|
||||
def test_a_side_is_one_of_the_offered_sizes_and_anything_else_is_refused():
|
||||
assert gifmod.SIDES == (320, 480, 640) and gifmod.DEFAULT_SIDE == 480
|
||||
for side in gifmod.SIDES:
|
||||
assert gifmod.normalize_side(side) == side
|
||||
for bad in (0, 100, 481, 1080, -480, True, "480", None, 480.5):
|
||||
with pytest.raises(ValueError):
|
||||
gifmod.normalize_side(bad)
|
||||
|
||||
|
||||
def test_the_limits_are_the_measured_ones():
|
||||
# 30 s at most; a link preview moves in Teams up to 15 MB (measured: 15 moved, 22 did not); a GIF
|
||||
# over 30 MB is not kept.
|
||||
assert gifmod.MAX_S == 30
|
||||
assert gifmod.PREVIEW_MAX_BYTES == 15 * 1024 * 1024
|
||||
assert gifmod.MAX_BYTES == 30 * 1024 * 1024
|
||||
assert gifmod.FPS == 12
|
||||
|
||||
|
||||
def _cmd(**kw):
|
||||
args = {"pre": "", "side": 480, **kw}
|
||||
return gifmod.build_cmd(Path("/m/in.mkv"), Path("/s/out.gif"), 61.5, 12.0, **args)
|
||||
|
||||
|
||||
def test_the_seek_is_on_the_input_and_the_library_read_is_guarded():
|
||||
cmd = _cmd()
|
||||
i = cmd.index("-i")
|
||||
assert cmd[cmd.index("-ss") + 1] == "61.500" and cmd.index("-ss") < i
|
||||
assert cmd[i + 1] == "/m/in.mkv"
|
||||
assert cmd[cmd.index("-t") + 1] == "12.000" and cmd.index("-t") > i
|
||||
g = cmd.index(INPUT_GUARD[0])
|
||||
assert cmd[g : g + len(INPUT_GUARD)] == INPUT_GUARD and g < i
|
||||
assert cmd[-1] == "/s/out.gif"
|
||||
|
||||
|
||||
def test_a_gif_has_no_sound_and_one_palette_of_its_own():
|
||||
cmd = _cmd()
|
||||
assert "-an" in cmd and "-sn" in cmd and "-dn" in cmd
|
||||
graph = cmd[cmd.index("-filter_complex") + 1]
|
||||
assert f"fps={gifmod.FPS}" in graph
|
||||
assert "palettegen=stats_mode=diff" in graph
|
||||
assert "paletteuse=dither=bayer:bayer_scale=5:diff_mode=rectangle" in graph
|
||||
assert "flags=lanczos" in graph
|
||||
|
||||
|
||||
def test_a_crop_or_a_tonemap_comes_before_the_scale_and_the_palette():
|
||||
cmd = _cmd(pre="crop=100:100:0:0,format=yuv420p")
|
||||
graph = cmd[cmd.index("-filter_complex") + 1]
|
||||
assert graph.index("crop=100:100:0:0") < graph.index("scale=") < graph.index("palettegen")
|
||||
|
||||
|
||||
def test_the_length_cap_counts_whole_milliseconds():
|
||||
# Two ends kept to the millisecond whose float difference lands a hair over 30 s.
|
||||
assert 60.529 - 30.529 > gifmod.MAX_S
|
||||
assert not gifmod.too_long(30.529, 60.529)
|
||||
assert not gifmod.too_long(0, 30)
|
||||
assert gifmod.too_long(0, 30.001)
|
||||
|
||||
|
||||
def test_the_pixels_are_squared_after_the_crop_and_before_the_size():
|
||||
graph = gifmod.filter_graph("crop=100:100:0:0", 480)
|
||||
assert graph.index("crop=") < graph.index("setsar=1") < graph.index("flags=lanczos")
|
||||
|
||||
|
||||
def _video(path: Path, size: str, seconds: int = 2, sar: str = "1") -> Path:
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", f"testsrc2=size={size}:rate=25:duration={seconds}",
|
||||
"-vf", f"setsar={sar}",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", str(path),
|
||||
],
|
||||
check=True,
|
||||
) # fmt: skip
|
||||
return path
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
@pytest.mark.parametrize(
|
||||
("size", "sar", "expected"),
|
||||
[
|
||||
# A DVD stored 720x480 and shown at 16:9: a GIF shows its stored pixels as they are, so
|
||||
# they are squared first, or the picture comes out 480x320 (3:2), squeezed.
|
||||
("720x480", "32/27", (480, 270)),
|
||||
# HDV: 1440x1080 shown at 16:9.
|
||||
("1440x1080", "4/3", (480, 270)),
|
||||
],
|
||||
)
|
||||
def test_a_source_with_wide_pixels_keeps_its_shown_shape(tmp_path, size, sar, expected):
|
||||
src = _video(tmp_path / "in.mp4", size, sar=sar)
|
||||
out = tmp_path / "out.gif"
|
||||
subprocess.run(gifmod.build_cmd(src, out, 0.0, 1.0, pre="", side=480), check=True)
|
||||
assert gifmod.size(out) == expected
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
@pytest.mark.parametrize(
|
||||
("size", "side", "expected"),
|
||||
[
|
||||
# The long side is capped, so a phone's portrait video is as small as a landscape one.
|
||||
("1280x720", 480, (480, 270)),
|
||||
("720x1280", 480, (270, 480)),
|
||||
("1280x720", 320, (320, 180)),
|
||||
# A picture already smaller than the size is never blown up.
|
||||
("160x120", 640, (160, 120)),
|
||||
],
|
||||
)
|
||||
def test_the_long_side_is_capped_and_a_small_picture_is_kept(tmp_path, size, side, expected):
|
||||
src = _video(tmp_path / "in.mp4", size)
|
||||
out = tmp_path / "out.gif"
|
||||
subprocess.run(gifmod.build_cmd(src, out, 0.5, 1.0, pre="", side=side), check=True)
|
||||
assert out.read_bytes()[:6] == b"GIF89a"
|
||||
assert gifmod.size(out) == expected
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_made_gif_is_a_library_file_that_previews_as_an_image(tmp_path):
|
||||
src = _video(tmp_path / "in.mp4", "320x240")
|
||||
out = tmp_path / "out.gif"
|
||||
subprocess.run(gifmod.build_cmd(src, out, 0.0, 1.0, pre="", side=320), check=True)
|
||||
vcodec, acodec, *_ = probemod.probe_media(out)
|
||||
assert mediakind.derive(vcodec, acodec) == mediakind.FILE
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_a_gif_gets_a_jpeg_still_for_link_previews(tmp_path):
|
||||
src = _video(tmp_path / "in.mp4", "320x240")
|
||||
out = tmp_path / "out.gif"
|
||||
subprocess.run(gifmod.build_cmd(src, out, 0.0, 1.0, pre="", side=320), check=True)
|
||||
still = gifmod.ensure_still(out)
|
||||
assert still == out.with_suffix(".jpg")
|
||||
assert still.read_bytes()[:3] == b"\xff\xd8\xff"
|
||||
|
||||
|
||||
@needs_ffmpeg
|
||||
def test_the_still_is_cut_only_from_a_real_gif(tmp_path):
|
||||
# The still's read is held to the GIF demuxer: a "gif" that is something else gives no still.
|
||||
fake = tmp_path / "out.gif"
|
||||
fake.write_bytes(b"ffconcat version 1.0\nfile '/etc/passwd'\n")
|
||||
assert gifmod.ensure_still(fake) is None
|
||||
|
||||
|
||||
def test_the_size_is_read_from_the_gif_header_and_nothing_else_has_one(tmp_path):
|
||||
g = tmp_path / "a.gif"
|
||||
g.write_bytes(b"GIF89a" + (300).to_bytes(2, "little") + (170).to_bytes(2, "little") + b"\x00")
|
||||
assert gifmod.size(g) == (300, 170)
|
||||
other = tmp_path / "b.gif"
|
||||
other.write_bytes(b"\x89PNG\r\n\x1a\n")
|
||||
assert gifmod.size(other) is None
|
||||
assert gifmod.size(tmp_path / "missing.gif") is None
|
||||
@@ -0,0 +1,137 @@
|
||||
"""A shared GIF (or any shared picture): served inline as itself on the public page, and its link
|
||||
preview card. Teams moves a GIF `og:image` up to 15 MB; Messenger never showed one and showed a JPEG
|
||||
(measured 2026-11-09/11), so Facebook's crawler and a GIF too big to move get the still. DB-lane."""
|
||||
|
||||
import pytest
|
||||
|
||||
from app import main
|
||||
from app.config import settings
|
||||
from app.downloads import og
|
||||
from app.downloads import probe as probemod
|
||||
from app.downloads.models import DownloadJob, MediaAsset
|
||||
|
||||
_PAGE = """<!doctype html><html><head>
|
||||
<!--OG:START--><meta property="og:title" content="Siftlode" /><!--OG:END-->
|
||||
</head><body><script src="/assets/watch.js"></script></body></html>"""
|
||||
|
||||
_GIF_PROBE = {
|
||||
"streams": [{"codec_type": "video", "codec_name": "gif", "width": 300, "height": 170}],
|
||||
"format": {"format_name": "gif"},
|
||||
}
|
||||
_GIF = b"GIF89a" + (300).to_bytes(2, "little") + (170).to_bytes(2, "little") + b"\x00" * 16
|
||||
|
||||
_TEAMS = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) SkypeUriPreview Preview/0.5 skype-url-preview@microsoft.com"
|
||||
_FACEBOOK = "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pages(tmp_path, monkeypatch):
|
||||
watch = tmp_path / "watch.html"
|
||||
watch.write_text(_PAGE, encoding="utf-8")
|
||||
monkeypatch.setattr(main, "WATCH_HTML", watch)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def shared(as_user, make_user, tmp_path, monkeypatch, upload_headers):
|
||||
"""Uploads `content` as `name` (ffprobe reading it as `data`) and shares it."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "media"))
|
||||
owner = as_user(make_user())
|
||||
|
||||
def share(name: str, content: bytes, data: dict | None, **link: object) -> dict:
|
||||
monkeypatch.setattr(probemod, "probe_json", lambda _p: data)
|
||||
job = owner.post(
|
||||
"/api/files/upload", content=content, headers=upload_headers(name, display_name=name)
|
||||
)
|
||||
assert job.status_code == 200, job.text
|
||||
made = owner.post(f"/api/downloads/{job.json()['id']}/links", json=link)
|
||||
assert made.status_code == 200, made.text
|
||||
return {**made.json(), "job_id": job.json()["id"]}
|
||||
|
||||
return share
|
||||
|
||||
|
||||
def _asset(db, made) -> MediaAsset:
|
||||
return db.get(MediaAsset, db.get(DownloadJob, made["job_id"]).asset_id)
|
||||
|
||||
|
||||
def test_a_shared_gif_is_served_inline_as_a_gif_and_can_run_nothing(shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE, allow_download=False)
|
||||
r = client.get(f"/api/public/watch/{made['token']}/file")
|
||||
assert r.status_code == 200
|
||||
assert r.headers["content-type"] == "image/gif"
|
||||
assert r.headers["content-disposition"].startswith("inline")
|
||||
# A "gif" whose bytes are a page must not run on this login-free origin.
|
||||
assert r.headers["x-content-type-options"] == "nosniff"
|
||||
assert "sandbox" in r.headers["content-security-policy"]
|
||||
assert r.content == _GIF
|
||||
|
||||
|
||||
def test_a_gif_link_that_allows_download_hands_it_over_as_a_file(shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE, allow_download=True)
|
||||
r = client.get(f"/api/public/watch/{made['token']}/file")
|
||||
assert r.headers["content-type"] == "image/gif"
|
||||
assert r.headers["content-disposition"].startswith("attachment")
|
||||
|
||||
|
||||
def test_any_other_file_is_still_only_a_download(shared, client):
|
||||
made = shared("notes.html", b"<script>alert(1)</script>", None)
|
||||
r = client.get(f"/api/public/watch/{made['token']}/file")
|
||||
assert r.headers["content-type"] == "application/octet-stream"
|
||||
assert r.headers["content-disposition"].startswith("attachment")
|
||||
|
||||
|
||||
def test_the_page_is_told_a_picture_is_a_picture(shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
meta = client.get(f"/api/public/watch/{made['token']}").json()
|
||||
assert meta["preview"] == "image"
|
||||
other = shared("backup.tar.gz", b"x", None)
|
||||
assert client.get(f"/api/public/watch/{other['token']}").json()["preview"] is None
|
||||
|
||||
|
||||
def _og(text: str, prop: str) -> str | None:
|
||||
marker = f'property="{prop}" content="'
|
||||
if marker not in text:
|
||||
return None
|
||||
return text.split(marker, 1)[1].split('"', 1)[0]
|
||||
|
||||
|
||||
def test_a_gif_link_previews_as_the_moving_gif_itself(pages, shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
page = client.get(f"/file/{made['token']}", headers={"User-Agent": _TEAMS}).text
|
||||
assert _og(page, "og:image") == (f"{settings.app_base}/api/public/watch/{made['token']}/file")
|
||||
assert _og(page, "og:image:type") == "image/gif"
|
||||
# Its size from its own header: a card wants it up front.
|
||||
assert (_og(page, "og:image:width"), _og(page, "og:image:height")) == ("300", "170")
|
||||
|
||||
|
||||
def test_facebooks_crawler_gets_the_still_of_a_gif(pages, shared, client, db):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
asset = _asset(db, made)
|
||||
asset.poster_path = "wave.jpg"
|
||||
db.commit()
|
||||
page = client.get(f"/file/{made['token']}", headers={"User-Agent": _FACEBOOK}).text
|
||||
assert _og(page, "og:image") == (
|
||||
f"{settings.app_base}/api/public/watch/{made['token']}/poster.jpg"
|
||||
)
|
||||
assert _og(page, "og:image:type") == "image/jpeg"
|
||||
|
||||
|
||||
def test_a_gif_too_big_to_move_previews_as_its_still(pages, shared, client, db):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
asset = _asset(db, made)
|
||||
asset.poster_path = "wave.jpg"
|
||||
asset.size_bytes = 15 * 1024 * 1024 + 1
|
||||
db.commit()
|
||||
page = client.get(f"/file/{made['token']}", headers={"User-Agent": _TEAMS}).text
|
||||
assert _og(page, "og:image").endswith("/poster.jpg")
|
||||
|
||||
|
||||
def test_a_password_gif_link_shows_nothing_of_it(pages, shared, client):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE, password="pw")
|
||||
page = client.get(f"/file/{made['token']}", headers={"User-Agent": _TEAMS}).text
|
||||
assert _og(page, "og:image") is None
|
||||
|
||||
|
||||
def test_the_card_builder_takes_no_agent_as_a_browser(shared, db):
|
||||
made = shared("wave.gif", _GIF, _GIF_PROBE)
|
||||
assert 'content="image/gif"' in og._watch_tags(made["token"], db)
|
||||
@@ -5,11 +5,11 @@ POST /session: after the last playback, an abandoned session (its ffmpeg still w
|
||||
film, its directory a whole segmented copy of it) lived until somebody pressed play again.
|
||||
|
||||
And it must not take a PAUSED viewer's session: a remux finishes long before the film does, and a
|
||||
paused player with a full buffer fetches nothing, so "finished and unfetched for 30 s" does not mean
|
||||
paused player with a full buffer fetches nothing, so "finished and unfetched for a while" does not mean
|
||||
"abandoned". An earlier fix bought that safety by keeping EVERY finished session for the full
|
||||
ten-minute idle window — one segmented copy of a film per item, on the volume the download center
|
||||
shares. Its successor replaced the blanket retention with the signal that already existed: a player
|
||||
parked mid-item checkpoints its position every 10 s, paused or not, and that route says "still
|
||||
parked mid-item keeps checkpointing its position, paused or not, and that route says "still
|
||||
here" (it stops once the item is finished, where reaping is correct).
|
||||
|
||||
Both call sites are pinned here, because the bug each time was that only one of them moved: the
|
||||
@@ -81,6 +81,9 @@ class _Proc:
|
||||
return 0
|
||||
|
||||
|
||||
_PAST_THE_RULE = stream._FINISHED_IDLE_S + 5
|
||||
|
||||
|
||||
class _Session:
|
||||
def __init__(self, tmp_path, name: str, finished: bool, idle_s: float):
|
||||
self.key = name
|
||||
@@ -98,11 +101,29 @@ def sessions(monkeypatch):
|
||||
|
||||
|
||||
def test_the_periodic_pass_reaps_a_finished_session_nobody_has_claimed(tmp_path, sessions):
|
||||
sessions["12345"] = _Session(tmp_path, "done", finished=True, idle_s=45)
|
||||
sessions["12345"] = _Session(tmp_path, "done", finished=True, idle_s=_PAST_THE_RULE)
|
||||
assert stream.reap_idle() == 1
|
||||
assert sessions == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("idle_s", [65, 125])
|
||||
def test_a_heartbeat_throttled_to_once_a_minute_keeps_a_finished_session(
|
||||
tmp_path, sessions, idle_s
|
||||
):
|
||||
"""A hidden page that makes no sound gets its timers cut to one a minute (measured on a paused and
|
||||
on a muted player, about a minute after hiding), so that is the viewer's heartbeat there: the
|
||||
reaper can see a beat over a minute old, and two if one is lost on the way."""
|
||||
s = _Session(tmp_path, "hidden", finished=True, idle_s=idle_s)
|
||||
sessions[(1, "12345")] = s
|
||||
assert stream.reap_idle() == 0
|
||||
assert (1, "12345") in sessions and s.dir.exists() and not s.proc.killed
|
||||
|
||||
|
||||
def test_an_abandoned_finished_session_is_still_gone_within_minutes(tmp_path, sessions):
|
||||
sessions[(1, "12345")] = _Session(tmp_path, "gone", finished=True, idle_s=185)
|
||||
assert stream.reap_idle() == 1
|
||||
|
||||
|
||||
def test_the_periodic_pass_still_reaps_what_nobody_has_touched_for_the_idle_limit(
|
||||
tmp_path, sessions
|
||||
):
|
||||
@@ -116,8 +137,32 @@ def test_the_periodic_pass_still_reaps_what_nobody_has_touched_for_the_idle_limi
|
||||
assert sessions == {}
|
||||
|
||||
|
||||
def test_every_reap_says_which_rule_took_the_session(tmp_path, sessions, caplog):
|
||||
"""A session that vanishes leaves the viewer a 410 and nothing else: with no line here, a live
|
||||
stall could only ever be blamed on the reaper by inference."""
|
||||
sessions["1"] = _Session(
|
||||
tmp_path, "gone_done", finished=True, idle_s=stream._SESSION_IDLE_S - 5
|
||||
)
|
||||
sessions["2"] = _Session(
|
||||
tmp_path, "gone_live", finished=False, idle_s=stream._SESSION_IDLE_S + 5
|
||||
)
|
||||
sessions["3"] = _Session(tmp_path, "kept", finished=False, idle_s=5)
|
||||
with caplog.at_level("INFO"):
|
||||
assert stream.reap_idle() == 2
|
||||
reaped = {
|
||||
r.attrs["key"]: r.attrs
|
||||
for r in caplog.records
|
||||
if getattr(r, "event", "") == "plex.stream.session_reaped"
|
||||
}
|
||||
assert set(reaped) == {"gone_done", "gone_live"}
|
||||
assert reaped["gone_done"]["reason"] == "finished_idle"
|
||||
assert reaped["gone_live"]["reason"] == "idle"
|
||||
assert reaped["gone_done"]["idle_s"] == pytest.approx(stream._SESSION_IDLE_S - 5, abs=2)
|
||||
assert reaped["gone_live"]["idle_s"] == pytest.approx(stream._SESSION_IDLE_S + 5, abs=2)
|
||||
|
||||
|
||||
def test_touching_a_session_keeps_it_through_the_short_rule(tmp_path, sessions):
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
|
||||
sessions[(1, "12345")] = s
|
||||
stream.touch_session(1, "12345")
|
||||
assert stream.reap_idle() == 0
|
||||
@@ -160,7 +205,7 @@ def test_a_paused_viewers_checkpoint_keeps_their_session_alive(
|
||||
user = User(email="paused@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
|
||||
sessions[(user.id, "paused1")] = s
|
||||
resp = _client(db, user, override_dep).post(
|
||||
f"/api/plex/item/{it.rating_key}/progress",
|
||||
@@ -175,7 +220,7 @@ def test_starting_a_session_leaves_another_viewers_paused_one_alone(
|
||||
db, monkeypatch, tmp_path, sessions, override_dep
|
||||
):
|
||||
"""The opportunistic sweep this route used to run was not scoped to the key being started: it
|
||||
walked the WHOLE registry with the 30 s finished rule, so pressing play on any item killed a
|
||||
walked the WHOLE registry with the finished rule, so pressing play on any item killed a
|
||||
paused viewer's session — the very case the retention fix set out to protect."""
|
||||
monkeypatch.setattr(state, "_configured_cache", True)
|
||||
monkeypatch.setattr(plex_paths, "local_media_path", lambda db_, fp: "/local/a.mkv")
|
||||
@@ -183,7 +228,7 @@ def test_starting_a_session_leaves_another_viewers_paused_one_alone(
|
||||
user = User(email="starter@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
paused = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
paused = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
|
||||
sessions[(999, "someone-else")] = paused
|
||||
|
||||
def fake_start(db_, item, user_id, start, audio=None, aoff=0.0, multi=False, transcode=False):
|
||||
|
||||
@@ -1,53 +1,88 @@
|
||||
"""No refusal answers in a hand-written English sentence.
|
||||
"""No refusal answers in a way the client cannot say.
|
||||
|
||||
The UI is bilingual and translates a structured `detail` (`{code, reason, …}`), but a string detail it
|
||||
can only print as it came — so ~150 of them reached the Hungarian UI verbatim, in the blocking error
|
||||
dialog and on every inline surface that shows `HttpError.detail`. `app/refusal.py` is the family they
|
||||
became; this test is what keeps a new one from being written the old way. Pure: it reads the sources.
|
||||
became; this test is what keeps a new one from being written the old way. It reads the same slots the
|
||||
refusal scan does (`refusal_sites.detail_slots`), so the two cannot disagree on where a detail is sent.
|
||||
Pure: it reads the sources.
|
||||
"""
|
||||
|
||||
import ast
|
||||
import pathlib
|
||||
import textwrap
|
||||
|
||||
APP = pathlib.Path(__file__).resolve().parents[1] / "app"
|
||||
from tests.refusal_sites import Scan, app_sources
|
||||
|
||||
# Answers that are no refusal of the app's, by the function they are sent from: the client shows a
|
||||
# detail only when it is an object (`lib/api/core.ts`), so these read as its generic line, and no screen
|
||||
# needs more from them.
|
||||
NO_REFUSAL = {
|
||||
("main.py", "setup_gate"): "the 503 before setup: the setup screen is what the user sees",
|
||||
("main.py", "spa_fallback"): "a bare 404 for an api path no route matches: no screen calls one",
|
||||
("main.py", "_validation_error"): "FastAPI's own 422 list: a body the client itself got wrong",
|
||||
("routes/diagnostics.py", "_parsed_events"): "the client-events beacon's 422: nothing shows it",
|
||||
}
|
||||
|
||||
|
||||
def _is_text(node: ast.AST) -> bool:
|
||||
"""A detail that is a sentence: a literal, an f-string, `str(exc)`, or `"…" + x`."""
|
||||
if isinstance(node, ast.Constant) and isinstance(node.value, str):
|
||||
return True
|
||||
if isinstance(node, ast.JoinedStr):
|
||||
return True
|
||||
if isinstance(node, ast.Call) and getattr(node.func, "id", "") == "str":
|
||||
return True
|
||||
if isinstance(node, ast.BoolOp):
|
||||
return any(_is_text(v) for v in node.values)
|
||||
if isinstance(node, ast.BinOp):
|
||||
return _is_text(node.left) or _is_text(node.right)
|
||||
return False
|
||||
def _offenders(sources: dict[str, str]) -> list[str]:
|
||||
return [
|
||||
f"{path}:{line}"
|
||||
for path, line, fn in Scan(sources).other_details()
|
||||
if (path, fn) not in NO_REFUSAL
|
||||
]
|
||||
|
||||
|
||||
def test_no_http_exception_carries_a_sentence():
|
||||
offenders = []
|
||||
for path in sorted(APP.rglob("*.py")):
|
||||
for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))):
|
||||
if not (
|
||||
isinstance(node, ast.Call)
|
||||
and getattr(node.func, "id", getattr(node.func, "attr", "")) == "HTTPException"
|
||||
):
|
||||
continue
|
||||
kw = {k.arg: k.value for k in node.keywords}
|
||||
detail = kw.get("detail", node.args[1] if len(node.args) > 1 else None)
|
||||
if detail is not None and _is_text(detail):
|
||||
offenders.append(f"{path.relative_to(APP).as_posix()}:{node.lineno}")
|
||||
def test_every_detail_is_one_the_client_can_say():
|
||||
offenders = _offenders(app_sources())
|
||||
assert offenders == [], (
|
||||
"a refusal with an English sentence for a detail — raise `refused(status, reason)` "
|
||||
f"(app/refusal.py) and name the reason in both locales: {offenders}"
|
||||
"a refusal the client can only answer with its generic line — send `{code, reason}` "
|
||||
f"(`refused(status, reason)`), or list it in NO_REFUSAL with why: {offenders}"
|
||||
)
|
||||
|
||||
|
||||
def test_the_scan_sees_a_sentence_when_there_is_one():
|
||||
def test_the_detail_scan_sees_each_shape_the_client_cannot_say():
|
||||
"""A guard that matches nothing is green over nothing."""
|
||||
sample = ast.parse('HTTPException(status_code=400, detail=f"bad {x}")').body[0].value
|
||||
assert _is_text(sample.keywords[1].value)
|
||||
assert not _is_text(ast.parse('{"code": "refused", "reason": "x"}').body[0].value)
|
||||
src = textwrap.dedent(
|
||||
"""
|
||||
def f(x):
|
||||
raise HTTPException(409, detail={"reason": "view_name_taken"})
|
||||
def g(x):
|
||||
raise HTTPException(409, dict(reason="taken"))
|
||||
def h(x):
|
||||
raise HTTPException(status_code=404)
|
||||
def i(x):
|
||||
return JSONResponse({"detail": "This instance isn't set up yet."}, status_code=503)
|
||||
def j(x):
|
||||
return JSONResponse(content={"detail": {"reason": "x"}}, status_code=409)
|
||||
def k(x):
|
||||
raise HTTPException(400, detail=f"bad {x}")
|
||||
def l(x):
|
||||
raise HTTPException(404, None)
|
||||
def n(x, msg):
|
||||
raise HTTPException(400, detail=msg)
|
||||
def o(x):
|
||||
raise HTTPException(400, detail=["a sentence"])
|
||||
def p(x):
|
||||
raise HTTPException(400, detail=describe(x))
|
||||
def q(x):
|
||||
raise HTTPException(400, detail={"code": "a"} if x else "text")
|
||||
def r(x):
|
||||
raise HTTPException(400, detail=nothing(x))
|
||||
def nothing(x):
|
||||
pass
|
||||
def ok(x):
|
||||
raise HTTPException(409, {"code": "refused", "reason": "x"})
|
||||
raise HTTPException(409, detail=quota(x))
|
||||
raise HTTPException(409, detail={"code": "a"} if x else {"code": "b"})
|
||||
return JSONResponse({"detail": {"code": "too_large"}}, status_code=413)
|
||||
return JSONResponse({"ok": True})
|
||||
def quota(x):
|
||||
return {"code": "quota", "reason": "files"}
|
||||
def setup_gate(x):
|
||||
return JSONResponse({"detail": "listed"}, status_code=503)
|
||||
"""
|
||||
)
|
||||
lines = (3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25)
|
||||
assert _offenders({"main.py": src}) == [f"main.py:{n}" for n in lines]
|
||||
# A file whose code dicts are data still answers its refusals in a way the client can say.
|
||||
data = 'def f():\n raise HTTPException(400, "text")\n'
|
||||
assert _offenders({"plex/tmdb.py": data}) == ["plex/tmdb.py:2"]
|
||||
|
||||
@@ -45,6 +45,9 @@ def test_an_interval_override_moves_into_sysconfig_and_back(db):
|
||||
}
|
||||
assert conn.execute(text("SELECT to_regclass('scheduler_settings')")).scalar() is None
|
||||
assert scheduler.load_intervals(db)["rss_poll"] == 25
|
||||
# The way down rebuilds an index CONCURRENTLY (0117), which waits for every open transaction —
|
||||
# the read above left this session's open.
|
||||
db.rollback()
|
||||
|
||||
command.downgrade(cfg, "0109_job_runs")
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Migration 0119's data step, DB-lane: the stored `transcode_hw` facet follows the rule that it says
|
||||
whether the GPU does the encode, and a 10-bit decodable item is probed again so its HDR flag decides.
|
||||
|
||||
The schema half is checked by test_migrations.py; a facet value left stale is invisible there.
|
||||
"""
|
||||
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db import engine
|
||||
|
||||
|
||||
def test_the_stored_facet_is_rewritten_by_the_gpu_encode_rule(db):
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0118_videos_shuffle_key")
|
||||
try:
|
||||
with engine.begin() as conn:
|
||||
lib = conn.execute(
|
||||
text(
|
||||
"INSERT INTO plex_libraries (plex_key, title, kind, enabled)"
|
||||
" VALUES ('m1', 'Movies', 'movie', true) RETURNING id"
|
||||
)
|
||||
).scalar_one()
|
||||
|
||||
def item(key, playable, codec, hw, depth=None, probed=False):
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO plex_items (rating_key, library_id, kind, title, playable,"
|
||||
" codec_video, transcode_hw, video_bit_depth, media_probed_at)"
|
||||
" VALUES (:k, :l, 'movie', :k, :p, :c, :hw, :d,"
|
||||
" CASE WHEN :pr THEN now() END)"
|
||||
),
|
||||
{
|
||||
"k": key,
|
||||
"l": lib,
|
||||
"p": playable,
|
||||
"c": codec,
|
||||
"hw": hw,
|
||||
"d": depth,
|
||||
"pr": probed,
|
||||
},
|
||||
)
|
||||
|
||||
item("vc1", "transcode", "VC1", True, 8, probed=True) # the old list said decodable
|
||||
item("xvid", "transcode", "mpeg4", False)
|
||||
item("divx", "transcode", "msmpeg4v3", False)
|
||||
item("av1", "transcode", "av1", False)
|
||||
item("h8", "transcode", "hevc", True, 8, probed=True)
|
||||
item("h10", "transcode", "hevc", False, 10, probed=True) # SDR or HDR: unknown here
|
||||
item("remux", "remux", "h264", None)
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
with engine.connect() as conn:
|
||||
rows = {
|
||||
k: (hw, probed)
|
||||
for k, hw, probed in conn.execute(
|
||||
text("SELECT rating_key, transcode_hw, media_probed_at IS NOT NULL FROM plex_items")
|
||||
).tuples()
|
||||
}
|
||||
assert rows == {
|
||||
"vc1": (True, True),
|
||||
"xvid": (True, False),
|
||||
"divx": (True, False),
|
||||
"av1": (False, False),
|
||||
"h8": (True, True),
|
||||
"h10": (False, False), # re-queued: the probe reads its HDR flag again
|
||||
"remux": (None, False),
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
"""A rename onto a name already taken says WHICH name is taken: the one asked for, not the row's
|
||||
old name (the rollback that follows the conflict puts the old name back on the row). DB lane.
|
||||
"""
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import app.state as state
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.models import User
|
||||
|
||||
|
||||
def _client(db, monkeypatch, override_dep):
|
||||
u = User(email="taken@example.com")
|
||||
db.add(u)
|
||||
db.commit()
|
||||
monkeypatch.setattr(
|
||||
state, "_configured_cache", True
|
||||
) # the request gate 503s an unconfigured app
|
||||
override_dep(get_db, db.info["request"])
|
||||
override_dep(current_user, lambda: u)
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_renaming_a_view_onto_a_taken_name_names_the_taken_one(db, monkeypatch, override_dep):
|
||||
c = _client(db, monkeypatch, override_dep)
|
||||
gaming = c.post("/api/saved-views", json={"name": "gaming", "filters": {}, "source": "feed"})
|
||||
assert gaming.status_code == 200
|
||||
hh = c.post("/api/saved-views", json={"name": "hh", "filters": {}, "source": "feed"}).json()
|
||||
|
||||
r = c.patch(f"/api/saved-views/{hh['id']}", json={"name": "gaming"})
|
||||
|
||||
assert r.status_code == 409
|
||||
assert r.json()["detail"] == {"code": "refused", "reason": "view_name_taken", "name": "gaming"}
|
||||
|
||||
|
||||
def test_renaming_a_tag_onto_a_taken_name_names_the_taken_one(db, monkeypatch, override_dep):
|
||||
c = _client(db, monkeypatch, override_dep)
|
||||
assert c.post("/api/tags", json={"name": "music"}).status_code == 200
|
||||
news = c.post("/api/tags", json={"name": "news"}).json()
|
||||
|
||||
r = c.patch(f"/api/tags/{news['id']}", json={"name": "music"})
|
||||
|
||||
assert r.status_code == 409
|
||||
assert r.json()["detail"] == {"code": "refused", "reason": "tag_name_taken", "name": "music"}
|
||||
@@ -181,7 +181,7 @@ def test_clearing_a_video_leaves_another_users_state(db, make_user, as_user):
|
||||
_videos(db, "v1")
|
||||
db.add(VideoState(user_id=other.id, video_id="v1", status="hidden", position_seconds=90))
|
||||
db.commit()
|
||||
assert as_user(me).delete("/api/videos/v1/state").status_code == 200
|
||||
assert as_user(me).delete("/api/videos/v1/progress").status_code == 200
|
||||
assert _video_state(db, other) == ("hidden", 90)
|
||||
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Adding a video AT a position: the list page's "Undo" of a removal.
|
||||
"""Putting a video back AT a position: the list page's "Undo" of a removal.
|
||||
|
||||
The row has to come back where it was in ONE request. The two-call shape (append, then send the
|
||||
whole order) leaves it at the end of the list whenever the second call fails, and the reorder would
|
||||
@@ -38,14 +38,17 @@ def _ids(db, pl_id):
|
||||
]
|
||||
|
||||
|
||||
def _add(client, pl_id, body):
|
||||
return client.post(f"/api/playlists/{pl_id}/items", json=body)
|
||||
def _restore(client, pl_id, video_id, position):
|
||||
return client.post(
|
||||
f"/api/playlists/{pl_id}/items/restore",
|
||||
json={"video_ids": [video_id], "positions": [position]},
|
||||
)
|
||||
|
||||
|
||||
def test_a_position_puts_the_video_back_in_its_place(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b", "c"])
|
||||
r = _add(as_user(user), pl.id, {"video_id": "vX", "position": 1})
|
||||
r = _restore(as_user(user), pl.id, "vX", 1)
|
||||
assert r.status_code == 200
|
||||
assert _ids(db, pl.id) == ["a", "vX", "b", "c"]
|
||||
|
||||
@@ -54,36 +57,37 @@ def test_the_position_counts_rows_not_stored_numbers(db, make_user, as_user):
|
||||
# A removal leaves a gap (0, 5, 9): index 2 is before the THIRD row, not "position 2".
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b", "c"], positions=[0, 5, 9])
|
||||
_add(as_user(user), pl.id, {"video_id": "vX", "position": 2})
|
||||
_restore(as_user(user), pl.id, "vX", 2)
|
||||
assert _ids(db, pl.id) == ["a", "b", "vX", "c"]
|
||||
|
||||
|
||||
def test_a_position_past_the_end_appends(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b"])
|
||||
_add(as_user(user), pl.id, {"video_id": "vX", "position": 40})
|
||||
_restore(as_user(user), pl.id, "vX", 40)
|
||||
assert _ids(db, pl.id) == ["a", "b", "vX"]
|
||||
|
||||
|
||||
def test_no_position_still_appends(db, make_user, as_user):
|
||||
def test_an_add_appends(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b"])
|
||||
_add(as_user(user), pl.id, {"video_id": "vX"})
|
||||
r = as_user(user).post(f"/api/playlists/{pl.id}/items", json={"video_id": "vX"})
|
||||
assert r.status_code == 200
|
||||
assert _ids(db, pl.id) == ["a", "b", "vX"]
|
||||
|
||||
|
||||
def test_a_video_already_in_the_list_does_not_move(db, make_user, as_user):
|
||||
# The add is idempotent; a position must not turn it into a reorder.
|
||||
# A restore is idempotent; a position must not turn it into a reorder.
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b", "c"])
|
||||
_add(as_user(user), pl.id, {"video_id": "c", "position": 0})
|
||||
_restore(as_user(user), pl.id, "c", 0)
|
||||
assert _ids(db, pl.id) == ["a", "b", "c"]
|
||||
|
||||
|
||||
def test_a_negative_position_is_refused(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a"])
|
||||
r = _add(as_user(user), pl.id, {"video_id": "vX", "position": -1})
|
||||
r = _restore(as_user(user), pl.id, "vX", -1)
|
||||
assert r.status_code == 422
|
||||
assert _ids(db, pl.id) == ["a"]
|
||||
|
||||
@@ -101,6 +105,6 @@ def test_putting_a_removed_video_back_leaves_a_linked_list_clean(db, make_user,
|
||||
assert client.delete(f"/api/playlists/{pl.id}/items/b").status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).dirty is True
|
||||
_add(client, pl.id, {"video_id": "b", "position": 1})
|
||||
_restore(client, pl.id, "b", 1)
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).dirty is False
|
||||
|
||||
@@ -0,0 +1,640 @@
|
||||
"""Many videos at once: remove, move, and the Undo of each. Every test reads both lists after the call,
|
||||
as `test_playlist_move` does: a half-applied move is the failure these routes exist to rule out."""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.catalog.models import Channel, Video
|
||||
from app.reader import queries, service
|
||||
from app.reader.models import Playlist, PlaylistItem
|
||||
from app.sync.playlists import current_fingerprint
|
||||
|
||||
|
||||
def _seed_videos(db, ids):
|
||||
db.add(Channel(id="UCtest", title="Test"))
|
||||
now = datetime.now(timezone.utc)
|
||||
for v in ids:
|
||||
db.add(Video(id=v, channel_id="UCtest", published_at=now))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _playlist(db, user, name, video_ids, **kw):
|
||||
pl = Playlist(user_id=user.id, name=name, **kw)
|
||||
db.add(pl)
|
||||
db.flush()
|
||||
for i, vid in enumerate(video_ids):
|
||||
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=i))
|
||||
db.commit()
|
||||
return pl
|
||||
|
||||
|
||||
def _ids(db, pl_id):
|
||||
db.expire_all()
|
||||
return list(
|
||||
db.execute(
|
||||
select(PlaylistItem.video_id)
|
||||
.where(PlaylistItem.playlist_id == pl_id)
|
||||
.order_by(PlaylistItem.position)
|
||||
).scalars()
|
||||
)
|
||||
|
||||
|
||||
def _positions(db, pl_id):
|
||||
db.expire_all()
|
||||
return list(
|
||||
db.execute(
|
||||
select(PlaylistItem.position)
|
||||
.where(PlaylistItem.playlist_id == pl_id)
|
||||
.order_by(PlaylistItem.position)
|
||||
).scalars()
|
||||
)
|
||||
|
||||
|
||||
def test_remove_bulk_takes_out_the_named_videos_and_counts_what_was_there(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "d"])
|
||||
pl = _playlist(db, user, "L", ["a", "b", "c", "d"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["b", "d", "gone"]}
|
||||
)
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {"ok": True, "playlist_id": pl.id, "removed": ["b", "d"]}
|
||||
assert _ids(db, pl.id) == ["a", "c"]
|
||||
|
||||
|
||||
def test_remove_bulk_of_nothing_leaves_the_list_untouched(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
pl = _playlist(db, user, "L", ["a"])
|
||||
stamp = datetime.now(timezone.utc) - timedelta(days=3)
|
||||
pl.updated_at = stamp
|
||||
db.commit()
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["gone"]}
|
||||
)
|
||||
|
||||
assert r.json()["removed"] == []
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).updated_at == stamp
|
||||
|
||||
|
||||
def test_restore_puts_the_videos_back_at_their_places(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "d"])
|
||||
pl = _playlist(db, user, "L", ["b", "d"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/restore",
|
||||
json={"video_ids": ["c", "a"], "positions": [2, 0]},
|
||||
)
|
||||
|
||||
assert r.json() == {"ok": True, "playlist_id": pl.id, "added": 2}
|
||||
assert _ids(db, pl.id) == ["a", "b", "c", "d"]
|
||||
assert _positions(db, pl.id) == [0, 1, 2, 3]
|
||||
|
||||
|
||||
def test_restore_leaves_a_video_already_back_where_it_is_and_skips_one_the_catalog_lost(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
pl = _playlist(db, user, "L", ["b", "a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/restore",
|
||||
json={"video_ids": ["a", "lost"], "positions": [0, 1]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 0
|
||||
assert _ids(db, pl.id) == ["b", "a"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"body",
|
||||
[
|
||||
{"video_ids": ["a"], "positions": []},
|
||||
{"video_ids": ["a"], "positions": [0, 1]},
|
||||
{"video_ids": ["a"], "positions": [-1]},
|
||||
],
|
||||
)
|
||||
def test_restore_refuses_positions_that_do_not_pair(db, make_user, as_user, body):
|
||||
user = make_user()
|
||||
pl = _playlist(db, user, "L", [])
|
||||
|
||||
assert as_user(user).post(f"/api/playlists/{pl.id}/items/restore", json=body).status_code == 422
|
||||
|
||||
|
||||
def test_move_bulk_lands_in_the_sources_order_whatever_order_they_were_picked_in(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "x"])
|
||||
src = _playlist(db, user, "Source", ["a", "b", "c"])
|
||||
dst = _playlist(db, user, "Target", ["x"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["c", "a"], "target_playlist_id": dst.id},
|
||||
)
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {
|
||||
"ok": True,
|
||||
"from_playlist_id": src.id,
|
||||
"to_playlist_id": dst.id,
|
||||
"moved": ["a", "c"],
|
||||
"already_there": [],
|
||||
}
|
||||
assert _ids(db, src.id) == ["b"]
|
||||
assert _ids(db, dst.id) == ["x", "a", "c"]
|
||||
|
||||
|
||||
def test_move_bulk_says_which_the_target_already_held_and_skips_one_already_gone(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c"])
|
||||
src = _playlist(db, user, "Source", ["a", "b"])
|
||||
dst = _playlist(db, user, "Target", ["b"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a", "b", "c"], "target_playlist_id": dst.id},
|
||||
)
|
||||
|
||||
assert r.json()["moved"] == ["a"]
|
||||
assert r.json()["already_there"] == ["b"]
|
||||
assert _ids(db, src.id) == []
|
||||
assert _ids(db, dst.id) == ["b", "a"]
|
||||
|
||||
|
||||
def test_move_bulk_keeps_when_each_video_was_saved(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"])
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
saved = datetime(2020, 1, 2, tzinfo=timezone.utc)
|
||||
row = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == src.id))
|
||||
row.added_at = saved
|
||||
db.commit()
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": dst.id},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
db.expire_all()
|
||||
moved = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == dst.id))
|
||||
assert moved.added_at == saved
|
||||
|
||||
|
||||
def test_a_target_that_gained_nothing_does_not_rise_in_the_recently_changed_sort(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
stamp = datetime.now(timezone.utc) - timedelta(days=3)
|
||||
dst.updated_at = stamp
|
||||
db.commit()
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": dst.id},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, dst.id).updated_at == stamp
|
||||
|
||||
|
||||
def test_move_bulk_into_the_same_list_is_refused(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
pl = _playlist(db, user, "L", ["a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": pl.id},
|
||||
)
|
||||
|
||||
assert r.status_code == 400
|
||||
assert r.json()["detail"] == {"code": "playlist_move", "reason": "same_playlist"}
|
||||
assert _ids(db, pl.id) == ["a"]
|
||||
|
||||
|
||||
def test_move_bulk_marks_a_linked_list_on_both_sides_as_unsynced(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"], yt_playlist_id="PLs")
|
||||
dst = _playlist(db, user, "Target", [], yt_playlist_id="PLd")
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": dst.id},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, src.id).dirty is True
|
||||
assert db.get(Playlist, dst.id).dirty is True
|
||||
|
||||
|
||||
def test_undo_move_restores_both_lists_as_they_stood(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "d"])
|
||||
src = _playlist(db, user, "Source", ["a", "b", "c", "d"])
|
||||
dst = _playlist(db, user, "Target", ["c"])
|
||||
client = as_user(user)
|
||||
|
||||
moved = client.post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a", "c"], "target_playlist_id": dst.id},
|
||||
).json()
|
||||
assert _ids(db, src.id) == ["b", "d"]
|
||||
assert _ids(db, dst.id) == ["c", "a"]
|
||||
|
||||
r = client.post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={
|
||||
"from_playlist_id": dst.id,
|
||||
"moved": moved["moved"],
|
||||
"already_there": moved["already_there"],
|
||||
"positions": [0, 2],
|
||||
},
|
||||
)
|
||||
|
||||
assert r.json() == {"ok": True, "playlist_id": src.id, "added": 2}
|
||||
assert _ids(db, src.id) == ["a", "b", "c", "d"]
|
||||
# The one the target held before the move stays there.
|
||||
assert _ids(db, dst.id) == ["c"]
|
||||
|
||||
|
||||
def test_undo_move_keeps_when_each_video_was_saved(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
saved = datetime(2020, 1, 2, tzinfo=timezone.utc)
|
||||
row = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == dst.id))
|
||||
row.added_at = saved
|
||||
db.commit()
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
db.expire_all()
|
||||
back = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == src.id))
|
||||
assert back.added_at == saved
|
||||
|
||||
|
||||
def test_undo_move_leaves_a_video_that_moved_on_where_it_is(db, make_user, as_user):
|
||||
"""A to B, then B to C, then the first Undo: the video is not in the list the Undo takes it from,
|
||||
and coming back would leave it in two lists the reader never put it in."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", ["b"])
|
||||
third = _playlist(db, user, "Third", ["a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a", "b"], "positions": [0, 1]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
assert _ids(db, src.id) == ["b"]
|
||||
assert _ids(db, dst.id) == []
|
||||
assert _ids(db, third.id) == ["a"]
|
||||
|
||||
|
||||
def test_an_undo_closes_up_the_places_of_a_video_that_stayed_out(db, make_user, as_user):
|
||||
"""Each index counts the rows put back before it, so one that stays out moves the rest up."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c", "d"])
|
||||
src = _playlist(db, user, "Source", ["c", "d"])
|
||||
dst = _playlist(db, user, "Target", ["b"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a", "b"], "positions": [0, 1]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
assert _ids(db, src.id) == ["b", "c", "d"]
|
||||
|
||||
|
||||
def test_a_restore_closes_up_the_places_of_a_video_the_catalog_lost(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["b", "c", "d"])
|
||||
pl = _playlist(db, user, "L", ["c", "d"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/restore",
|
||||
json={"video_ids": ["lost", "b"], "positions": [0, 1]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
assert _ids(db, pl.id) == ["b", "c", "d"]
|
||||
|
||||
|
||||
def test_undo_move_takes_a_video_out_of_the_target_when_the_source_has_it_again(
|
||||
db, make_user, as_user
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 0
|
||||
assert _ids(db, src.id) == ["a"]
|
||||
assert _ids(db, dst.id) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "body", "gone", "reason"),
|
||||
[
|
||||
("move-bulk", {"video_ids": ["a"], "target_playlist_id": "dst"}, "dst", "move_target"),
|
||||
("move-bulk", {"video_ids": ["a"], "target_playlist_id": "dst"}, "src", "playlist"),
|
||||
(
|
||||
"undo-move",
|
||||
{"from_playlist_id": "dst", "moved": ["a"], "positions": [0]},
|
||||
"src",
|
||||
"playlist",
|
||||
),
|
||||
(
|
||||
"undo-move",
|
||||
{"from_playlist_id": "dst", "moved": ["a"], "positions": [0]},
|
||||
"dst",
|
||||
"move_target",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_a_list_deleted_before_the_lock_is_not_found(
|
||||
db, make_user, as_user, monkeypatch, path, body, gone, reason
|
||||
):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", ["a"])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
ids = {"src": src.id, "dst": dst.id}
|
||||
body = {k: (dst.id if v == "dst" else v) for k, v in body.items()}
|
||||
# Another tab's delete committed between the route's read and this lock.
|
||||
monkeypatch.setattr(service, "lock_playlists", lambda _db, *got: set(got) - {ids[gone]})
|
||||
|
||||
r = as_user(user).post(f"/api/playlists/{src.id}/items/{path}", json=body)
|
||||
|
||||
assert r.status_code == 404
|
||||
assert reason in str(r.json()["detail"])
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path", "body"),
|
||||
[
|
||||
("post", "items", {"video_id": "a"}),
|
||||
("delete", "items/a", None),
|
||||
("post", "items/remove-bulk", {"video_ids": ["a"]}),
|
||||
("post", "items/restore", {"video_ids": ["a"], "positions": [0]}),
|
||||
],
|
||||
)
|
||||
def test_a_one_list_write_finds_its_list_under_the_lock(
|
||||
db, make_user, as_user, monkeypatch, method, path, body
|
||||
):
|
||||
"""Read with its lock, so a delete that won the row answers 404 rather than a foreign key's 500."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
pl = _playlist(db, user, "L", [])
|
||||
real = queries.own_playlist
|
||||
|
||||
def locked_read_finds_nothing(db_, user_, playlist_id, *, lock=False):
|
||||
return None if lock else real(db_, user_, playlist_id)
|
||||
|
||||
monkeypatch.setattr(queries, "own_playlist", locked_read_finds_nothing)
|
||||
kw = {"json": body} if body is not None else {}
|
||||
|
||||
r = getattr(as_user(user), method)(f"/api/playlists/{pl.id}/{path}", **kw)
|
||||
|
||||
assert r.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "body"),
|
||||
[
|
||||
("items/remove-bulk", {"ids": ["a"]}),
|
||||
("items/move-bulk", {"ids": ["a"], "target_playlist_id": 1}),
|
||||
("items/undo-move", {"ids": ["a"], "from_playlist_id": 1}),
|
||||
],
|
||||
)
|
||||
def test_a_body_without_its_videos_is_refused_rather_than_acting_on_none(
|
||||
db, make_user, as_user, path, body
|
||||
):
|
||||
user = make_user()
|
||||
pl = _playlist(db, user, "L", [])
|
||||
|
||||
assert as_user(user).post(f"/api/playlists/{pl.id}/{path}", json=body).status_code == 422
|
||||
|
||||
|
||||
def test_an_undo_of_only_videos_the_target_held_names_no_moved_ones(db, make_user, as_user):
|
||||
"""An empty `moved` is a real body: every video of the move was already in the target."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": [], "already_there": ["a"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.json()["added"] == 1
|
||||
assert _ids(db, src.id) == ["a"]
|
||||
assert _ids(db, dst.id) == ["a"]
|
||||
|
||||
|
||||
def test_an_undo_within_one_list_is_refused_and_keeps_its_videos(db, make_user, as_user):
|
||||
"""A move refuses the same list twice, so no Undo of one exists; a row must not go for asking."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
pl = _playlist(db, user, "L", ["a", "b"])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pl.id}/items/undo-move",
|
||||
json={"from_playlist_id": pl.id, "moved": ["a"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.status_code == 400
|
||||
assert r.json()["detail"] == {"code": "playlist_move", "reason": "same_playlist"}
|
||||
assert _ids(db, pl.id) == ["a", "b"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "key", "extra"),
|
||||
[
|
||||
("items/restore", "video_ids", {}),
|
||||
("items/undo-move", "moved", {"from_playlist_id": None}),
|
||||
],
|
||||
)
|
||||
def test_two_videos_cannot_ask_for_one_place(db, make_user, as_user, path, key, extra):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b", "c"])
|
||||
pl = _playlist(db, user, "L", ["c"])
|
||||
other = _playlist(db, user, "Other", ["a", "b"])
|
||||
body = {key: ["a", "b"], "positions": [0, 0]}
|
||||
body.update({k: other.id for k in extra})
|
||||
|
||||
r = as_user(user).post(f"/api/playlists/{pl.id}/{path}", json=body)
|
||||
|
||||
assert r.status_code == 422
|
||||
assert _ids(db, pl.id) == ["c"]
|
||||
|
||||
|
||||
def test_a_reorder_without_its_videos_is_refused(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _playlist(db, user, "L", [])
|
||||
|
||||
assert as_user(user).put(f"/api/playlists/{pl.id}/order", json={"ids": []}).status_code == 422
|
||||
|
||||
|
||||
def _synced(db, pl):
|
||||
pl.synced_fingerprint = current_fingerprint(db, pl)
|
||||
pl.dirty = False
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_a_bulk_remove_and_its_undo_leave_a_linked_list_as_it_was(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
pl = _playlist(db, user, "L", ["a", "b"], yt_playlist_id="PLx")
|
||||
_synced(db, pl)
|
||||
client = as_user(user)
|
||||
|
||||
r = client.post(f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["a"]})
|
||||
assert r.status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).dirty is True
|
||||
|
||||
r = client.post(
|
||||
f"/api/playlists/{pl.id}/items/restore", json={"video_ids": ["a"], "positions": [0]}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, pl.id).dirty is False
|
||||
|
||||
|
||||
def test_a_move_and_its_undo_leave_both_linked_lists_as_they_were(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a", "b"])
|
||||
src = _playlist(db, user, "Source", ["a", "b"], yt_playlist_id="PLs")
|
||||
dst = _playlist(db, user, "Target", ["b"], yt_playlist_id="PLd")
|
||||
_synced(db, src)
|
||||
_synced(db, dst)
|
||||
client = as_user(user)
|
||||
|
||||
moved = client.post(
|
||||
f"/api/playlists/{src.id}/items/move-bulk",
|
||||
json={"video_ids": ["a"], "target_playlist_id": dst.id},
|
||||
).json()
|
||||
assert moved["moved"] == ["a"]
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, src.id).dirty is True
|
||||
assert db.get(Playlist, dst.id).dirty is True
|
||||
|
||||
r = client.post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, src.id).dirty is False
|
||||
assert db.get(Playlist, dst.id).dirty is False
|
||||
|
||||
|
||||
def test_undo_move_is_one_transaction(db, make_user, as_user, monkeypatch):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", ["a"])
|
||||
|
||||
def boom(*a, **kw):
|
||||
raise RuntimeError("second step failed")
|
||||
|
||||
monkeypatch.setattr(service, "_insert_at", boom)
|
||||
client = as_user(user)
|
||||
body = {"from_playlist_id": dst.id, "moved": ["a"], "already_there": [], "positions": [0]}
|
||||
with pytest.raises(RuntimeError):
|
||||
_ = client.post(f"/api/playlists/{src.id}/items/undo-move", json=body)
|
||||
|
||||
db.rollback()
|
||||
assert _ids(db, dst.id) == ["a"]
|
||||
assert _ids(db, src.id) == []
|
||||
|
||||
|
||||
def test_undo_move_refuses_positions_that_do_not_pair(db, make_user, as_user):
|
||||
user = make_user()
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{src.id}/items/undo-move",
|
||||
json={"from_playlist_id": dst.id, "moved": ["a"], "already_there": ["b"], "positions": [0]},
|
||||
)
|
||||
|
||||
assert r.status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "body"),
|
||||
[
|
||||
("remove-bulk", {"video_ids": ["a"]}),
|
||||
("restore", {"video_ids": ["a"], "positions": [0]}),
|
||||
("move-bulk", {"video_ids": ["a"], "target_playlist_id": 0}),
|
||||
("undo-move", {"from_playlist_id": 0, "moved": ["a"], "positions": [0]}),
|
||||
],
|
||||
)
|
||||
def test_another_accounts_list_is_not_found(db, make_user, as_user, path, body):
|
||||
owner, other = make_user(), make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
theirs = _playlist(db, owner, "Theirs", ["a"])
|
||||
mine = _playlist(db, other, "Mine", [])
|
||||
body = {
|
||||
k: (mine.id if k in ("target_playlist_id", "from_playlist_id") else v)
|
||||
for k, v in body.items()
|
||||
}
|
||||
|
||||
r = as_user(other).post(f"/api/playlists/{theirs.id}/items/{path}", json=body)
|
||||
|
||||
assert r.status_code == 404
|
||||
assert _ids(db, theirs.id) == ["a"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "key"), [("move-bulk", "target_playlist_id"), ("undo-move", "from_playlist_id")]
|
||||
)
|
||||
def test_the_other_list_of_a_move_must_be_the_callers_too(db, make_user, as_user, path, key):
|
||||
owner, other = make_user(), make_user()
|
||||
_seed_videos(db, ["a"])
|
||||
theirs = _playlist(db, owner, "Theirs", ["a"])
|
||||
mine = _playlist(db, other, "Mine", ["a"])
|
||||
body = {"video_ids": ["a"], "moved": ["a"], "positions": [0], key: theirs.id}
|
||||
|
||||
r = as_user(other).post(f"/api/playlists/{mine.id}/items/{path}", json=body)
|
||||
|
||||
assert r.status_code == 404
|
||||
assert "move_target" in str(r.json()["detail"])
|
||||
assert _ids(db, theirs.id) == ["a"]
|
||||
assert _ids(db, mine.id) == ["a"]
|
||||
@@ -51,7 +51,8 @@ def _ids(db, pl_id):
|
||||
|
||||
def _move(client, src, video_id, dst):
|
||||
return client.post(
|
||||
f"/api/playlists/{src}/items/{video_id}/move", json={"target_playlist_id": dst}
|
||||
f"/api/playlists/{src}/items/move-bulk",
|
||||
json={"video_ids": [video_id], "target_playlist_id": dst},
|
||||
)
|
||||
|
||||
|
||||
@@ -67,7 +68,8 @@ def test_move_leaves_the_video_in_exactly_one_list(db, make_user, as_user):
|
||||
"ok": True,
|
||||
"from_playlist_id": src.id,
|
||||
"to_playlist_id": dst.id,
|
||||
"video_id": "v1",
|
||||
"moved": ["v1"],
|
||||
"already_there": [],
|
||||
}
|
||||
db.expire_all()
|
||||
assert _ids(db, src.id) == ["v2"]
|
||||
@@ -116,8 +118,9 @@ def test_move_of_a_video_that_left_the_source_reports_it(db, make_user, as_user)
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
|
||||
r = _move(as_user(user), src.id, "v1", dst.id)
|
||||
assert r.status_code == 404
|
||||
assert r.json()["detail"] == {"code": "playlist_move", "reason": "not_in_playlist"}
|
||||
assert r.status_code == 200
|
||||
assert (r.json()["moved"], r.json()["already_there"]) == ([], [])
|
||||
assert _ids(db, dst.id) == []
|
||||
|
||||
|
||||
def test_move_into_someone_elses_list_404s_and_changes_nothing(db, make_user, as_user):
|
||||
|
||||
@@ -102,16 +102,24 @@ PLEX = PlexPlaylist.__table__.name
|
||||
"call",
|
||||
[
|
||||
lambda c, pid, other: c.post(f"/api/playlists/{pid}/items", json={"video_id": "v2"}),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items", json={"video_id": "v2", "position": 0}
|
||||
),
|
||||
lambda c, pid, other: c.post("/api/playlists/watch-later", json={"video_id": "v2"}),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
|
||||
f"/api/playlists/{pid}/items/move-bulk",
|
||||
json={"video_ids": ["v1"], "target_playlist_id": other},
|
||||
),
|
||||
lambda c, pid, other: c.put(f"/api/playlists/{pid}/order", json={"video_ids": ["v1"]}),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/remove-bulk", json={"video_ids": ["v1"]}
|
||||
),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/restore", json={"video_ids": ["v2"], "positions": [0]}
|
||||
),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/undo-move",
|
||||
json={"from_playlist_id": other, "moved": ["v2"], "positions": [0]},
|
||||
),
|
||||
],
|
||||
ids=["add", "add-at", "watch-later", "move", "reorder"],
|
||||
ids=["add", "watch-later", "move", "reorder", "remove-bulk", "restore", "undo-move"],
|
||||
)
|
||||
def test_a_youtube_playlist_write_locks_the_playlist(db, make_user, as_user, locks, call):
|
||||
user = make_user()
|
||||
@@ -168,18 +176,35 @@ PLEX_ITEMS = PlexPlaylistItem.__table__.name
|
||||
"call",
|
||||
[
|
||||
lambda c, pid, other: c.post(f"/api/playlists/{pid}/items", json={"video_id": "v1"}),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items", json={"video_id": "v2", "position": 0}
|
||||
),
|
||||
lambda c, pid, other: c.delete(f"/api/playlists/{pid}/items/v1"),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
|
||||
f"/api/playlists/{pid}/items/move-bulk",
|
||||
json={"video_ids": ["v1"], "target_playlist_id": other},
|
||||
),
|
||||
lambda c, pid, other: c.put(f"/api/playlists/{pid}/order", json={"video_ids": ["v1"]}),
|
||||
# the ORM cascade deletes the items before the list row
|
||||
lambda c, pid, other: c.delete(f"/api/playlists/{pid}"),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/remove-bulk", json={"video_ids": ["v1"]}
|
||||
),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/restore", json={"video_ids": ["v2"], "positions": [0]}
|
||||
),
|
||||
lambda c, pid, other: c.post(
|
||||
f"/api/playlists/{pid}/items/undo-move",
|
||||
json={"from_playlist_id": other, "moved": ["v2"], "positions": [0]},
|
||||
),
|
||||
],
|
||||
ids=[
|
||||
"add-present",
|
||||
"remove",
|
||||
"move",
|
||||
"reorder",
|
||||
"delete",
|
||||
"remove-bulk",
|
||||
"restore",
|
||||
"undo-move",
|
||||
],
|
||||
ids=["add-present", "add-at", "remove", "move", "reorder", "delete"],
|
||||
)
|
||||
def test_a_youtube_item_write_locks_the_playlist_before_any_item(
|
||||
db, make_user, as_user, statements, call
|
||||
@@ -193,14 +218,23 @@ def test_a_youtube_item_write_locks_the_playlist_before_any_item(
|
||||
assert _locked_before_items(statements, YT, YT_ITEMS), statements
|
||||
|
||||
|
||||
def test_a_move_locks_both_lists_in_one_ordered_statement(db, make_user, as_user, statements):
|
||||
"""Two moves A→B and B→A each locking "its" list first would lock crosswise."""
|
||||
@pytest.mark.parametrize(
|
||||
("path", "body"),
|
||||
[
|
||||
("move-bulk", lambda other: {"video_ids": ["v1"], "target_playlist_id": other}),
|
||||
("undo-move", lambda other: {"from_playlist_id": other, "moved": ["v2"], "positions": [0]}),
|
||||
],
|
||||
ids=["move", "undo"],
|
||||
)
|
||||
def test_a_move_locks_both_lists_in_one_ordered_statement(
|
||||
db, make_user, as_user, statements, path, body
|
||||
):
|
||||
"""Two moves A→B and B→A each locking "its" list first would lock crosswise, and an Undo runs the
|
||||
other way round from a move by its nature."""
|
||||
user = make_user()
|
||||
pid, other = _yt(db, user)
|
||||
statements.clear()
|
||||
r = as_user(user).post(
|
||||
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
|
||||
)
|
||||
r = as_user(user).post(f"/api/playlists/{pid}/items/{path}", json=body(other))
|
||||
assert r.status_code == 200, r.text
|
||||
lock = _locked_before_items(statements, YT, YT_ITEMS)
|
||||
assert lock and "ORDER BY" in lock, lock
|
||||
|
||||
@@ -219,6 +219,7 @@ def test_item_detail_movie_serves_from_row_despite_live_outage(db, monkeypatch,
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="The Godfather",
|
||||
size_bytes=15_000_000_000,
|
||||
genres=["Crime", "Drama"],
|
||||
directors=["Francis Ford Coppola"],
|
||||
studio="Paramount",
|
||||
@@ -238,6 +239,7 @@ def test_item_detail_movie_serves_from_row_despite_live_outage(db, monkeypatch,
|
||||
resp = _get(db, user, monkeypatch, override_dep, "/api/plex/item/m1")
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["size_bytes"] == 15_000_000_000
|
||||
assert body["imdb_rating"] == 8.7
|
||||
assert body["imdb_id"] == "tt0068646"
|
||||
assert body["imdb_url"] == "https://www.imdb.com/title/tt0068646/"
|
||||
|
||||
@@ -5,7 +5,7 @@ seek-restart — or a second viewer of the same item — replaces the registered
|
||||
viewer's next request for `seg_12.ts`, resolved by rating_key, was answered 200 with the NEW
|
||||
session's twelfth segment: bytes produced from a different `-ss` offset under `-copyts`, i.e. a
|
||||
timeline the client is not on. Wrong content with a success status is the one failure a player cannot
|
||||
notice, and the client's reclaim-on-404 could never fire on it.
|
||||
notice, and the client's reclaim could never fire on it.
|
||||
|
||||
No `db` fixture: the route reads nothing from the database (its own `Depends(get_db)` went with the
|
||||
removed auto-start, and `current_user` is overridden wholesale), so these must not sit in the DB
|
||||
@@ -88,10 +88,10 @@ def test_another_viewers_session_of_the_same_item_is_not_this_viewers(client, tm
|
||||
(d / "index.m3u8").write_bytes(b"#EXTM3U")
|
||||
sessions[(7, "m1")] = _Session("sid-a", d)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/index.m3u8")
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.status_code == 410, r.text
|
||||
|
||||
|
||||
def test_a_segment_of_a_replaced_session_is_a_404_not_the_new_sessions_bytes(
|
||||
def test_a_segment_of_a_replaced_session_is_gone_not_the_new_sessions_bytes(
|
||||
client, tmp_path, sessions
|
||||
):
|
||||
"""Viewer A is at 0:00; viewer B seeks to 25:00 and their session takes the registry slot. A's
|
||||
@@ -102,15 +102,15 @@ def test_a_segment_of_a_replaced_session_is_a_404_not_the_new_sessions_bytes(
|
||||
(theirs / "seg_12.ts").write_bytes(b"theirs")
|
||||
sessions[(VIEWER, "m1")] = _Session("sid-b", theirs)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.status_code == 410, r.text
|
||||
assert b"theirs" not in r.content
|
||||
|
||||
|
||||
def test_a_playlist_with_no_live_session_is_a_404_not_a_silent_restart_at_zero(
|
||||
def test_a_playlist_with_no_live_session_is_gone_not_a_silent_restart_at_zero(
|
||||
client, monkeypatch, sessions
|
||||
):
|
||||
"""The entry playlist used to auto-start a session when none was registered — always at 0.0,
|
||||
silently throwing away the viewer's position. A 404 sends the player through its own reclaim,
|
||||
silently throwing away the viewer's position. A 410 sends the player through its own reclaim,
|
||||
which restarts at the playhead."""
|
||||
started: list[float] = []
|
||||
monkeypatch.setattr(
|
||||
@@ -119,7 +119,7 @@ def test_a_playlist_with_no_live_session_is_a_404_not_a_silent_restart_at_zero(
|
||||
lambda *a, **k: started.append(a[3] if len(a) > 3 else 0.0),
|
||||
)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-gone/index.m3u8")
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.status_code == 410, r.text
|
||||
assert started == []
|
||||
|
||||
|
||||
@@ -235,12 +235,12 @@ def test_an_absurd_segment_index_is_a_404_not_a_500(client, tmp_path, sessions,
|
||||
assert r.status_code == 404, r.text
|
||||
|
||||
|
||||
def test_a_session_torn_down_during_the_read_is_a_404_not_a_500(
|
||||
def test_a_session_torn_down_during_the_read_is_gone_not_a_500(
|
||||
client, monkeypatch, tmp_path, sessions
|
||||
):
|
||||
"""Teardown moves the whole directory aside and nothing holds `_lock` across the wait and the read,
|
||||
so the file can vanish in between — on every seek-restart, since `loadSession` awaits the new
|
||||
session before destroying the old hls instance. The player's recovery is keyed on the 404
|
||||
session before destroying the old hls instance. The player's recovery is keyed on the
|
||||
STATUS, so a 500 here skipped the reclaim and ended playback with "couldn't play"."""
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
@@ -249,7 +249,7 @@ def test_a_session_torn_down_during_the_read_is_a_404_not_a_500(
|
||||
# The file existed when the wait checked, and is gone by the time the route reads it.
|
||||
monkeypatch.setattr(plex_stream, "wait_for", lambda *a, **k: True)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.status_code == 410, r.text
|
||||
|
||||
|
||||
def test_a_dead_producer_is_not_waited_out(client, tmp_path, sessions):
|
||||
|
||||
@@ -32,15 +32,25 @@ class TestTranscodeHwCapable:
|
||||
assert sync.transcode_hw_capable("transcode", "hevc") is True
|
||||
assert sync.transcode_hw_capable("transcode", "vp9", bit_depth=8) is True
|
||||
|
||||
def test_non_decodable_codec_is_sw(self):
|
||||
# AV1 / MPEG-4 ASP (XviD) can't be HW-decoded on Gen 9.5 → software regardless of bit depth.
|
||||
def test_a_codec_neither_path_knows_is_sw(self):
|
||||
assert sync.transcode_hw_capable("transcode", "av1") is False
|
||||
assert sync.transcode_hw_capable("transcode", "mpeg4", bit_depth=8) is False
|
||||
assert sync.transcode_hw_capable("transcode", "av1", bit_depth=8) is False
|
||||
|
||||
def test_10bit_or_hdr_hevc_is_sw(self):
|
||||
# A HW-decodable codec that is 10-bit or HDR still breaks the VAAPI nv12 pipeline → software.
|
||||
def test_a_cpu_decoded_8bit_codec_is_encoded_on_the_gpu(self):
|
||||
# VC-1 (the iGPU has no VC-1 Advanced decode), XviD, DivX: the CPU decodes, the GPU encodes.
|
||||
for codec in ("vc1", "mpeg4", "msmpeg4v3"):
|
||||
assert sync.transcode_hw_capable("transcode", codec) is True
|
||||
assert sync.transcode_hw_capable("transcode", "mpeg4", bit_depth=10) is False
|
||||
|
||||
def test_10bit_sdr_hevc_is_sw(self):
|
||||
# 10-bit SDR: an exact downconvert on the CPU (libx264), never the HDR tone curve.
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=10) is False
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=8, hdr=True) is False
|
||||
|
||||
def test_hdr_hevc_is_encoded_on_the_gpu(self):
|
||||
# HDR goes through the OpenCL tonemap between VAAPI decode and encode (the prod image ships the
|
||||
# runtime); the facet said software for it while every such session ran on the GPU.
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=10, hdr=True) is True
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=8, hdr=True) is True
|
||||
|
||||
|
||||
class TestVideoStreamProps:
|
||||
@@ -87,6 +97,9 @@ class TestSyncDerivation:
|
||||
row2 = PlexItem(rating_key="y", kind="movie")
|
||||
sync._apply_item(row2, lib_id=1, kind="movie", meta=self._meta("av1"))
|
||||
assert row2.transcode_hw is False # av1 not HW-decodable
|
||||
row3 = PlexItem(rating_key="z", kind="movie")
|
||||
sync._apply_item(row3, lib_id=1, kind="movie", meta=self._meta("vc1"))
|
||||
assert row3.transcode_hw is True # CPU decode, GPU encode
|
||||
|
||||
def test_probed_value_survives_resync_when_codec_unchanged(self):
|
||||
row = PlexItem(rating_key="x", kind="movie")
|
||||
@@ -124,21 +137,24 @@ class TestSyncDerivation:
|
||||
assert row.video_bit_depth is None and row.media_probed_at is None # re-probe scheduled
|
||||
|
||||
|
||||
def test_migration_codec_set_matches_hw_decodable_codecs():
|
||||
# Migration 0077 hardcodes the HW-decodable codec set in raw SQL and its comment mandates it stay
|
||||
# identical to plex.sync.HW_DECODABLE_CODECS. A comment isn't enforcement (house rule) — pin the
|
||||
# equality so adding/removing a codec from the frozenset without editing the migration fails here
|
||||
# (else already-migrated prod rows keep a wrong transcode_hw the runtime derivation wouldn't catch).
|
||||
def test_migration_codec_sets_match_the_sync_sets():
|
||||
# The newest migration that rewrites `transcode_hw` hardcodes both codec sets in raw SQL. Pin them
|
||||
# to plex.sync, so a codec added to or removed from a frozenset without a migration that rewrites
|
||||
# the stored rows fails here (else prod rows keep a facet the runtime choice disagrees with).
|
||||
mig = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "alembic"
|
||||
/ "versions"
|
||||
/ "0077_plex_playback_class.py"
|
||||
/ "0119_plex_transcode_gpu.py"
|
||||
)
|
||||
m = re.search(r"lower\(codec_video\)\s+IN\s+\(([^)]*)\)", mig.read_text())
|
||||
assert m, "could not find the codec IN(...) list in migration 0077"
|
||||
codecs = {c.strip().strip("'\"").lower() for c in m.group(1).split(",")}
|
||||
assert codecs == {c.lower() for c in sync.HW_DECODABLE_CODECS}
|
||||
src = mig.read_text()
|
||||
sets = {}
|
||||
for name in ("HW_DECODABLE_CODECS", "CPU_DECODE_CODECS"):
|
||||
m = re.search(name + r"\s*=\s*\(([^)]*)\)", src)
|
||||
assert m, f"could not find {name} in migration 0119"
|
||||
sets[name] = {c.strip().strip("'\"").lower() for c in m.group(1).split(",") if c.strip()}
|
||||
assert sets["HW_DECODABLE_CODECS"] == {c.lower() for c in sync.HW_DECODABLE_CODECS}
|
||||
assert sets["CPU_DECODE_CODECS"] == {c.lower() for c in sync.CPU_DECODE_CODECS}
|
||||
|
||||
|
||||
class _FakePlex:
|
||||
@@ -182,22 +198,42 @@ def test_probe_media_batch_refines_hw_to_sw_for_10bit(db):
|
||||
codec_video="mpeg4",
|
||||
transcode_hw=False,
|
||||
)
|
||||
db.add_all([hevc8, hevc10, xvid])
|
||||
hdr10 = PlexItem(
|
||||
rating_key="hdr",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="hdr",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
)
|
||||
db.add_all([hevc8, hevc10, xvid, hdr10])
|
||||
db.commit()
|
||||
plex = _FakePlex(
|
||||
{
|
||||
"h8": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]},
|
||||
"h10": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 10}]}]}]},
|
||||
"hdr": {
|
||||
"Media": [
|
||||
{
|
||||
"Part": [
|
||||
{"Stream": [{"streamType": 1, "bitDepth": 10, "colorTrc": "smpte2084"}]}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
)
|
||||
probed = sync._probe_media_batch(db, plex)
|
||||
db.refresh(hevc8)
|
||||
db.refresh(hevc10)
|
||||
db.refresh(xvid)
|
||||
# Only the two HW-decodable HEVC items are probed; the XviD is SW-by-codec and never fetched.
|
||||
assert probed == 2
|
||||
db.refresh(hdr10)
|
||||
# Only the HW-decodable HEVC items are probed; the XviD's class follows from its codec alone.
|
||||
assert probed == 3
|
||||
assert hevc8.video_bit_depth == 8 and hevc8.transcode_hw is True and hevc8.media_probed_at
|
||||
assert hevc10.video_bit_depth == 10 and hevc10.transcode_hw is False # downgraded to software
|
||||
assert hevc10.video_bit_depth == 10 and hevc10.transcode_hw is False # 10-bit SDR: software
|
||||
assert hdr10.video_bit_depth == 10 and hdr10.transcode_hw is True # HDR: the GPU tonemap
|
||||
assert xvid.media_probed_at is None # not in the work-list
|
||||
# A second run has nothing left to do (watermarks set).
|
||||
assert sync._probe_media_batch(db, plex) == 0
|
||||
|
||||
@@ -6,6 +6,8 @@ rows, since positions have gaps after a removal. One request: an append plus a s
|
||||
leaves the rows at the end whenever that call fails, and would overwrite another tab's changes.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from app.plex.models import PlexItem, PlexLibrary, PlexPlaylist, PlexPlaylistItem
|
||||
|
||||
|
||||
@@ -101,6 +103,22 @@ def test_positions_must_pair_with_the_keys(db, make_user, as_user):
|
||||
assert _order(db, pl.id)[0] == ["a"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path"),
|
||||
[("post", "items/bulk"), ("post", "items/remove-bulk"), ("put", "order")],
|
||||
)
|
||||
def test_a_body_without_its_keys_is_refused_rather_than_acting_on_none(
|
||||
db, make_user, as_user, method, path
|
||||
):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a"], extra=())
|
||||
|
||||
r = getattr(as_user(user), method)(f"/api/plex/playlists/{pl.id}/{path}", json={"keys": ["a"]})
|
||||
|
||||
assert r.status_code == 422
|
||||
assert _order(db, pl.id)[0] == ["a"]
|
||||
|
||||
|
||||
def test_remove_then_put_back_is_no_change(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user, ["a", "b", "c", "d"], extra=())
|
||||
|
||||
@@ -284,6 +284,24 @@ class TestTranscodeCommand:
|
||||
)
|
||||
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # not a software scale
|
||||
|
||||
def test_cpu_decode_uploads_the_frames_to_the_gpu_encoder(self):
|
||||
# A source the GPU cannot decode (VC-1 Advanced on Gen 9.5): the CPU decodes, the frames go to
|
||||
# nv12 and up to the GPU, and h264_vaapi encodes. No -hwaccel: asking the GPU to decode is what
|
||||
# failed.
|
||||
cmd = stream._ffmpeg_cmd(
|
||||
self.SRC, 0.0, self.OUT, None, transcode=True, hw=True, cpu_decode=True
|
||||
)
|
||||
assert "-hwaccel" not in cmd
|
||||
assert cmd[cmd.index("-init_hw_device") + 1] == f"vaapi=va:{stream._VAAPI_DEVICE}"
|
||||
assert cmd[cmd.index("-filter_hw_device") + 1] == "va"
|
||||
assert cmd.index("-init_hw_device") < cmd.index("-i")
|
||||
assert cmd[cmd.index("-vf") + 1] == "format=nv12,hwupload"
|
||||
assert _venc(cmd) == "h264_vaapi"
|
||||
assert "libx264" not in cmd
|
||||
assert (
|
||||
cmd[cmd.index("-force_key_frames") + 1] == f"expr:gte(t,n_forced*{stream._SEG_SECONDS})"
|
||||
)
|
||||
|
||||
def test_hdr_tonemap_uses_the_opencl_graph(self):
|
||||
# The HDR/10-bit HW path: VAAPI decode → OpenCL tonemap → VAAPI encode. Gen 9.5 can't tonemap in
|
||||
# the VAAPI VPP, so this is the only HW route for a 10-bit/HDR source.
|
||||
@@ -365,19 +383,48 @@ class TestWantHw:
|
||||
is True
|
||||
)
|
||||
|
||||
def test_auto_falls_back_to_software_for_a_non_decodable_source(self, monkeypatch):
|
||||
# AV1 / MPEG-4 ASP (XviD) aren't decodable on Gen 9.5 → the GPU can't feed its own encoder.
|
||||
def test_auto_falls_back_to_software_for_a_codec_neither_path_knows(self, monkeypatch):
|
||||
# AV1 is neither GPU-decodable on Gen 9.5 nor on the CPU-decode list (it can be 10-bit HDR).
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._hwsafe(monkeypatch, True) # decodability gate must fire BEFORE the probe
|
||||
self._hwsafe(monkeypatch, True) # the codec gate must fire BEFORE the probe
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="mpeg4"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("codec", ["vc1", "mpeg4", "msmpeg4v3"])
|
||||
def test_auto_cpu_decodes_what_the_gpu_cannot_and_still_encodes_on_it(self, monkeypatch, codec):
|
||||
# Measured on prod (UHD 630, iHD 25.2.3): "No support for codec vc1 profile 3", so every VC-1
|
||||
# HW session died and fell back to libx264 (~19 s to start). Decoding on the CPU and encoding on
|
||||
# the GPU produced three segments in ~5 s (VC-1 1080p) and ~1 s (XviD, DivX SD).
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._kind(monkeypatch, stream._VK_HWSAFE)
|
||||
self._opencl(monkeypatch, True)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video=codec), transcode=True, src=self.SRC
|
||||
) == (True, False, True)
|
||||
|
||||
def test_auto_a_cpu_decoded_codec_that_is_not_8bit_sdr_stays_software(self, monkeypatch):
|
||||
# The upload path converts to nv12 for the encoder; only an 8-bit SDR picture is known to come
|
||||
# through it unchanged.
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._kind(monkeypatch, stream._VK_SDR10)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="mpeg4"), transcode=True, src=self.SRC
|
||||
) == (False, False, False)
|
||||
|
||||
def test_on_cpu_decodes_a_codec_the_gpu_cannot(self, monkeypatch):
|
||||
# "on" trusts the GPU, but asking it to DECODE a codec it is known not to decode only buys the
|
||||
# fallback's wasted second; the upload path is the GPU route for such a source.
|
||||
self._policy(monkeypatch, "on")
|
||||
self._opencl(monkeypatch, False)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="vc1"), transcode=True, src=self.SRC
|
||||
) == (True, False, True)
|
||||
|
||||
def test_auto_hdr_source_falls_back_to_software_without_opencl(self, monkeypatch):
|
||||
# A decodable HEVC that is true-HDR (PQ/HLG) can't use scale_vaapi=format=nv12 (Gen 9.5), and with
|
||||
@@ -388,7 +435,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (False, False)
|
||||
) == (False, False, False)
|
||||
|
||||
def test_auto_hdr_source_uses_hw_tonemap_with_opencl(self, monkeypatch):
|
||||
# With the OpenCL runtime present, a true-HDR HEVC tone-maps on the GPU (VAAPI decode → OpenCL
|
||||
@@ -399,7 +446,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, True)
|
||||
) == (True, True, False)
|
||||
|
||||
def test_auto_10bit_SDR_source_uses_software_not_tonemap(self, monkeypatch):
|
||||
# F1 regression: a 10-bit *SDR* (bt709) source is NOT HDR, so it must NOT get the hable HDR tone
|
||||
@@ -411,7 +458,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (False, False)
|
||||
) == (False, False, False)
|
||||
|
||||
def test_auto_8bit_source_uses_plain_hw_not_tonemap(self, monkeypatch):
|
||||
# An 8-bit SDR source takes the plain full-GPU scale path — never the tonemap graph — even where
|
||||
@@ -422,7 +469,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
) == (True, False, False)
|
||||
|
||||
def test_off_forces_software_even_for_a_decodable_source(self, monkeypatch):
|
||||
self._policy(monkeypatch, "off")
|
||||
@@ -460,7 +507,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, True)
|
||||
) == (True, True, False)
|
||||
|
||||
def test_on_10bit_SDR_uses_plain_hw_not_tonemap(self, monkeypatch):
|
||||
# F1 regression under "on": a 10-bit SDR source is NOT HDR, so even with OpenCL present "on" must
|
||||
@@ -471,7 +518,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
) == (True, False, False)
|
||||
|
||||
def test_on_without_opencl_is_plain_hw(self, monkeypatch):
|
||||
# "on" with no OpenCL runtime → plain HW, tonemap=False, regardless of source class (the runtime
|
||||
@@ -483,7 +530,7 @@ class TestWantHw:
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
) == (True, False, False)
|
||||
|
||||
|
||||
class TestColdStartWarmup:
|
||||
@@ -620,6 +667,30 @@ class TestTranscodeStartSession:
|
||||
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # no SW cap
|
||||
assert kwargs.get("preexec_fn") is None # GPU encode is cheap — never niced
|
||||
|
||||
def test_a_cpu_decode_session_encodes_on_the_gpu_and_falls_back_to_software(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (True, False, True)
|
||||
)
|
||||
seen = {"n": 0}
|
||||
|
||||
def flaky_wait(path, timeout=20.0, proc=None):
|
||||
seen["n"] += 1
|
||||
return seen["n"] > 1 # the upload session yields nothing; the software respawn does
|
||||
|
||||
monkeypatch.setattr(stream, "wait_for", flaky_wait)
|
||||
s = stream.start_session(
|
||||
db=None, user_id=7, item=_FakeItem(codec_video="vc1"), start_s=0.0, transcode=True
|
||||
)
|
||||
(up_cmd, *_), up_kwargs = _CapturingProc.calls[0]
|
||||
assert "format=nv12,hwupload" in up_cmd and _venc(up_cmd) == "h264_vaapi"
|
||||
if os.name == "posix": # the decode is CPU work: niced like a software transcode
|
||||
assert up_kwargs.get("preexec_fn") is stream._sw_child_preexec
|
||||
(sw_cmd, *_), _ = _CapturingProc.calls[1]
|
||||
assert _venc(sw_cmd) == "libx264" and "format=nv12,hwupload" not in sw_cmd
|
||||
assert s is not None and s.hw is False and s.cpu_decode is False
|
||||
|
||||
def test_hdr_tonemap_transcode_threads_the_opencl_graph_and_flags_the_session(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
|
||||
@@ -14,7 +14,7 @@ from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from app.discover import pins
|
||||
from app.catalog.models import Channel, Video
|
||||
from app.catalog.models import Channel, Subscription, Video
|
||||
from app.reader import queries, service
|
||||
from app.reader.models import ChannelTag, ComfortPin, Playlist, PlaylistItem, Tag, VideoState
|
||||
|
||||
@@ -60,12 +60,44 @@ def test_a_system_tag_is_nobodys_to_edit(db, make_user):
|
||||
service.create_tag(db, other, "Theirs", "other", None) # another user's tag stays theirs
|
||||
db.commit()
|
||||
assert queries.own_tag(db, user, system.id) is None
|
||||
assert [t.id for t, _ in queries.tags_with_counts(db, user)] == [system.id]
|
||||
assert [t.id for t, *_ in queries.tags_with_counts(db, user)] == [system.id]
|
||||
|
||||
|
||||
# --- tags ------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _music_on_four(db, make_user):
|
||||
"""A tag on four channels: two followed, one followed but hidden, one no longer followed."""
|
||||
user, other = make_user(), make_user()
|
||||
tag = service.create_tag(db, user, "Music", "other", None)
|
||||
for cid in ("UCa", "UCb", "UChid", "UCgone"):
|
||||
db.add(Channel(id=cid, title=cid))
|
||||
db.flush()
|
||||
for cid in ("UCa", "UCb"):
|
||||
db.add(Subscription(user_id=user.id, channel_id=cid))
|
||||
db.add(Subscription(user_id=user.id, channel_id="UChid", hidden=True))
|
||||
db.add(Subscription(user_id=other.id, channel_id="UCgone"))
|
||||
for cid in ("UCa", "UCb", "UChid", "UCgone"):
|
||||
db.add(ChannelTag(channel_id=cid, tag_id=tag.id, user_id=user.id, source="user"))
|
||||
db.commit()
|
||||
return user, tag
|
||||
|
||||
|
||||
def test_a_tag_counts_apart_the_links_on_channels_the_user_still_follows(db, make_user):
|
||||
"""An unsubscribe and a hide keep the channel's tag links; the Channels page's filter lists only
|
||||
the followed channels not hidden, so its count must not hold the others."""
|
||||
user, tag = _music_on_four(db, make_user)
|
||||
[(_, count, subscribed)] = [r for r in queries.tags_with_counts(db, user) if r[0].id == tag.id]
|
||||
assert (count, subscribed) == (4, 2)
|
||||
|
||||
|
||||
def test_the_tag_list_sends_each_count_under_its_own_name(db, make_user, as_user):
|
||||
"""Both counts are ints, so a swap would type-check: the answer itself pins which is which."""
|
||||
user, tag = _music_on_four(db, make_user)
|
||||
[row] = [t for t in as_user(user).get("/api/tags").json() if t["id"] == tag.id]
|
||||
assert (row["channel_count"], row["subscribed_count"]) == (4, 2)
|
||||
|
||||
|
||||
def test_a_tag_is_made_once_per_name_and_an_unknown_category_is_other(db, make_user):
|
||||
user = make_user()
|
||||
tag = service.create_tag(db, user, "Later", "nonsense", "#fff")
|
||||
@@ -190,9 +222,6 @@ def test_a_cleared_resume_point_drops_a_bare_row_and_zeroes_one_with_a_status(db
|
||||
db.commit()
|
||||
assert _state(db, user, "v1") is None
|
||||
assert (_state(db, user, "v2").status, _state(db, user, "v2").position_seconds) == ("hidden", 0)
|
||||
service.clear_state(db, user, "v2")
|
||||
db.commit()
|
||||
assert _state(db, user, "v2") is None
|
||||
|
||||
|
||||
# --- playlists -------------------------------------------------------------------------------
|
||||
@@ -213,9 +242,8 @@ def test_an_add_is_idempotent_and_lands_where_asked(db, make_user):
|
||||
pl = service.create_playlist(db, user, "L")
|
||||
assert service.add_item(db, pl, "a", mark_dirty=False) is True
|
||||
assert service.add_item(db, pl, "a", mark_dirty=False) is False
|
||||
service.add_to_playlist(db, pl, "b", None)
|
||||
service.add_to_playlist(db, pl, "c", 0)
|
||||
service.add_to_playlist(db, pl, "a", 2) # already there: stays put
|
||||
assert service.add_item(db, pl, "b", mark_dirty=True) is True
|
||||
assert service.restore_items(db, pl, ["c", "a"], [0, 2]) == 1 # "a" is there: it stays put
|
||||
assert _uncommitted(db)
|
||||
db.commit()
|
||||
assert _ids(db, pl) == ["c", "a", "b"]
|
||||
@@ -224,7 +252,7 @@ def test_an_add_is_idempotent_and_lands_where_asked(db, make_user):
|
||||
db.commit()
|
||||
assert _ids(db, pl) == ["c", "b"]
|
||||
assert queries.item_counts(db, [pl.id]) == {pl.id: 2}
|
||||
assert queries.durations(db, [pl.id]) == {pl.id: 120}
|
||||
assert queries.durations(db, [pl.id]) == {pl.id: (120, 0)}
|
||||
assert queries.containing(db, [pl.id], "b") == {pl.id}
|
||||
|
||||
|
||||
@@ -238,14 +266,13 @@ def test_a_move_never_leaves_a_video_in_two_lists(db, make_user):
|
||||
service.add_item(db, dst, "b", mark_dirty=False)
|
||||
db.commit()
|
||||
with pytest.raises(service.MoveRefused, match="same_playlist"):
|
||||
service.move_item(db, src, src, "a")
|
||||
service.move_item(db, src, dst, "a")
|
||||
service.move_item(db, src, dst, "b") # already in the target: dropped from the source
|
||||
service.move_items(db, src, src, ["a"])
|
||||
assert service.move_items(db, src, dst, ["a"]) == (["a"], [])
|
||||
# already in the target: dropped from the source
|
||||
assert service.move_items(db, src, dst, ["b"]) == ([], ["b"])
|
||||
db.commit()
|
||||
assert (_ids(db, src), _ids(db, dst)) == ([], ["b", "a"])
|
||||
with pytest.raises(service.MoveRefused, match="not_in_playlist"):
|
||||
service.move_item(db, src, dst, "a")
|
||||
db.rollback()
|
||||
assert service.move_items(db, src, dst, ["a"]) == ([], [])
|
||||
|
||||
|
||||
def test_a_reorder_keeps_what_it_left_out_after_the_listed(db, make_user):
|
||||
@@ -267,7 +294,7 @@ def test_a_rename_and_a_delete(db, make_user):
|
||||
service.add_item(db, pl, "a", mark_dirty=False)
|
||||
service.rename_playlist(db, pl, "New")
|
||||
db.commit()
|
||||
assert queries.playlist_totals(db, pl.id) == (1, 60, None)
|
||||
assert queries.playlist_totals(db, pl.id) == (1, 60, 0, None)
|
||||
assert pl.name == "New"
|
||||
service.delete_playlist(db, pl)
|
||||
db.commit()
|
||||
|
||||
@@ -0,0 +1,770 @@
|
||||
"""The refusal scan (`refusal_sites.py`) on small sources: each shape on its closed list is followed to its
|
||||
values, and each exit from that list is reported, unread or as an unread code. A scan that silently drops a
|
||||
shape is a guard green over nothing, so every exit is pinned here, not only the paths the app takes. Pure:
|
||||
no app import, no database."""
|
||||
|
||||
import textwrap
|
||||
|
||||
from tests.refusal_sites import scan, scan_app
|
||||
|
||||
|
||||
def _scan(**files: str) -> dict:
|
||||
return scan(
|
||||
{f"{name.replace('__', '/')}.py": textwrap.dedent(src) for name, src in files.items()}
|
||||
)
|
||||
|
||||
|
||||
def _code(result: dict, code: str) -> dict:
|
||||
return result["codes"][code]
|
||||
|
||||
|
||||
def _unread(result: dict, code: str) -> list[str]:
|
||||
return [u.split(": ", 1)[1] for u in _code(result, code)["unread"]]
|
||||
|
||||
|
||||
REFUSAL = """
|
||||
from fastapi import HTTPException
|
||||
|
||||
def refused(status: int, reason: str, **params: object) -> HTTPException:
|
||||
return HTTPException(status_code=status, detail={**params, "code": "refused", "reason": reason})
|
||||
"""
|
||||
|
||||
|
||||
def test_a_literal_wherever_it_sits_in_the_dict():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(x):
|
||||
raise HTTPException(409, {"code": "clip", "limit_s": 5, "reason": "too_long"})
|
||||
def g(x):
|
||||
raise HTTPException(409, {"reason": "before", "code": "clip"})
|
||||
def h(x):
|
||||
raise HTTPException(409, {
|
||||
"code": "clip", # a comment between
|
||||
"reason": "after_comment",
|
||||
})
|
||||
def i(x):
|
||||
raise HTTPException(409, detail=dict(code="clip", reason="by_call"))
|
||||
"""
|
||||
)
|
||||
assert _code(r, "clip")["reasons"] == ["after_comment", "before", "by_call", "too_long"]
|
||||
assert _code(r, "clip")["unread"] == []
|
||||
|
||||
|
||||
def test_both_arms_of_a_conditional_and_a_none_arm_makes_the_site_reasonless():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(x):
|
||||
raise HTTPException(409, {"code": "edit", "reason": "a" if x else "b"})
|
||||
def g(x):
|
||||
raise HTTPException(409, {"code": "playlist", "reason": "lit" if x else None})
|
||||
def h(x):
|
||||
raise HTTPException(409, {"code": "busy", "reason": None})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "edit")["reasons"] == ["a", "b"] and _code(r, "edit")["reasonless"] == []
|
||||
assert _code(r, "playlist")["reasons"] == ["lit"]
|
||||
assert _code(r, "playlist")["reasonless"] == ["a.py:5"]
|
||||
assert _code(r, "busy")["reasonless"] == ["a.py:7"]
|
||||
|
||||
|
||||
def test_what_the_list_does_not_hold_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(x, svc):
|
||||
raise HTTPException(409, {"code": "edit", "reason": x.why or "c"})
|
||||
def g(svc):
|
||||
raise HTTPException(409, {"code": "edit", "reason": svc.reason()})
|
||||
def h():
|
||||
raise HTTPException(409, {"code": "edit", "reason": 5})
|
||||
def i(e):
|
||||
raise HTTPException(409, {"code": "edit", "reason": f"x{e}"})
|
||||
"""
|
||||
)
|
||||
assert _unread(r, "edit") == ["x.why or 'c'", "svc.reason()", "5", "f'x{e}'"]
|
||||
assert _code(r, "edit")["reasons"] == []
|
||||
|
||||
|
||||
def test_a_helper_is_followed_to_every_call_however_it_is_written():
|
||||
r = _scan(
|
||||
refusal=REFUSAL,
|
||||
routes__a="""
|
||||
from http import HTTPStatus
|
||||
from app import refusal
|
||||
from app.refusal import refused
|
||||
|
||||
def f(status):
|
||||
raise refused(status, "by_name_status")
|
||||
def g():
|
||||
raise refused(HTTPStatus.CONFLICT, reason="by_keyword")
|
||||
def h():
|
||||
raise refusal.refused(409, "qualified")
|
||||
def i(x):
|
||||
raise refused(409, "a" if x else "second_arm")
|
||||
""",
|
||||
routes__b="""
|
||||
from other.lib import refused
|
||||
|
||||
def f():
|
||||
refused("an outside library's, not the app's")
|
||||
""",
|
||||
)
|
||||
assert _code(r, "refused")["reasons"] == [
|
||||
"a",
|
||||
"by_keyword",
|
||||
"by_name_status",
|
||||
"qualified",
|
||||
"second_arm",
|
||||
]
|
||||
assert _code(r, "refused")["unread"] == []
|
||||
|
||||
|
||||
def test_a_call_the_helper_cannot_be_followed_through_is_unread():
|
||||
r = _scan(
|
||||
refusal=REFUSAL,
|
||||
routes__a="""
|
||||
from app.refusal import refused
|
||||
|
||||
def f(args, kw):
|
||||
pick = refused
|
||||
raise refused(*args)
|
||||
def g(kw):
|
||||
raise refused(409, **kw)
|
||||
def h():
|
||||
raise refused(409)
|
||||
""",
|
||||
routes__b="""
|
||||
import app.refusal
|
||||
|
||||
def f():
|
||||
raise app.refusal.refused(409, "dotted")
|
||||
""",
|
||||
routes__c="""
|
||||
from ..refusal import refused
|
||||
|
||||
def f():
|
||||
raise refused(409, "relative")
|
||||
""",
|
||||
)
|
||||
unread = _unread(r, "refused")
|
||||
assert len(unread) == 6
|
||||
assert sum("the helper used as a value" in u for u in unread) == 1
|
||||
assert sum("a reference the scan cannot resolve" in u for u in unread) == 2
|
||||
assert sum("does not pass reason" in u for u in unread) == 2 # `**kw`, and nothing at all
|
||||
assert any(u == "refused(*args)" for u in unread)
|
||||
assert _code(r, "refused")["reasons"] == []
|
||||
|
||||
|
||||
def test_a_default_is_not_followed_a_call_must_pass_the_reason():
|
||||
r = _scan(
|
||||
routes___common="""
|
||||
def not_found(what: str = "generic"):
|
||||
return HTTPException(404, {"code": "not_found", "reason": what})
|
||||
|
||||
def get_or_404(db, id_, *, what: str = "generic"):
|
||||
raise not_found(what)
|
||||
""",
|
||||
routes__a="""
|
||||
from app.routes._common import get_or_404, not_found
|
||||
|
||||
def f(db):
|
||||
get_or_404(db, 1, what="user")
|
||||
get_or_404(db, 2)
|
||||
raise not_found("debug_session")
|
||||
def g():
|
||||
raise not_found()
|
||||
""",
|
||||
)
|
||||
assert _code(r, "not_found")["reasons"] == ["debug_session", "user"]
|
||||
assert [u.endswith("(does not pass what)") for u in _unread(r, "not_found")] == [True, True]
|
||||
|
||||
|
||||
def test_a_positional_only_reason_passed_by_name_is_not_bound_to_it():
|
||||
r = _scan(
|
||||
a="""
|
||||
def deny(reason: str, /, **params):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
|
||||
def f():
|
||||
raise deny("by_position")
|
||||
def g():
|
||||
raise deny(reason="lands_in_params")
|
||||
"""
|
||||
)
|
||||
assert _code(r, "playlist")["reasons"] == ["by_position"]
|
||||
assert _unread(r, "playlist") == ["deny(reason='lands_in_params') (does not pass reason)"]
|
||||
|
||||
|
||||
def test_a_helper_reaching_two_sites_gives_both_its_reasons_and_one_reached_again_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
def deny(reason: str, admin):
|
||||
if admin:
|
||||
return HTTPException(409, {"code": "activate", "reason": reason})
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
|
||||
def again(reason: str, n):
|
||||
if n:
|
||||
return again(reason, n - 1)
|
||||
return HTTPException(409, {"code": "edit", "reason": reason})
|
||||
|
||||
def f():
|
||||
raise deny("foo", True)
|
||||
def g():
|
||||
raise again("bar", 3)
|
||||
"""
|
||||
)
|
||||
assert _code(r, "activate")["reasons"] == ["foo"]
|
||||
assert _code(r, "playlist")["reasons"] == ["foo"]
|
||||
assert _code(r, "edit")["reasons"] == ["bar"]
|
||||
assert any("reached again while it is followed" in u for u in _unread(r, "edit"))
|
||||
|
||||
|
||||
def test_a_parameter_its_body_binds_again_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
from typing import Literal
|
||||
|
||||
def refused(status, reason: str = ""):
|
||||
reason = reason or "fallback"
|
||||
return HTTPException(status, {"code": "refused", "reason": reason})
|
||||
|
||||
def typed(reason: Literal["a"]):
|
||||
if not reason:
|
||||
reason = "b"
|
||||
return HTTPException(409, {"code": "edit", "reason": reason})
|
||||
|
||||
def f():
|
||||
raise refused(409, "x")
|
||||
"""
|
||||
)
|
||||
assert _code(r, "refused")["reasons"] == []
|
||||
assert _unread(r, "refused") == ["reason (a parameter its body binds again)"]
|
||||
assert _unread(r, "edit") == ["reason (a parameter its body binds again)"]
|
||||
|
||||
|
||||
def test_a_local_is_followed_only_through_plain_assignments():
|
||||
r = _scan(
|
||||
a="""
|
||||
from typing import Literal
|
||||
|
||||
def f(x):
|
||||
why = "first"
|
||||
if x:
|
||||
why = "second" if x > 1 else "third"
|
||||
raise HTTPException(409, {"code": "edit", "reason": why})
|
||||
def g(svc):
|
||||
hit, why = svc.fetch()
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
def h(items):
|
||||
for why in items:
|
||||
pass
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
def i():
|
||||
why: Literal["typed"] = pick()
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
def j(items):
|
||||
LABELS[why] = "NotAReason"
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
def k(*why):
|
||||
raise HTTPException(409, {"code": "playlist", "reason": why})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "edit")["reasons"] == ["first", "second", "third"]
|
||||
assert _code(r, "playlist")["reasons"] == []
|
||||
assert _unread(r, "playlist") == [
|
||||
"why",
|
||||
"why",
|
||||
"why",
|
||||
"why",
|
||||
"why",
|
||||
] # unpacked, a loop, annotated, unbound, `*args`
|
||||
|
||||
|
||||
def test_a_call_answers_with_its_literal_return_type():
|
||||
r = _scan(
|
||||
a="""
|
||||
from typing import Literal
|
||||
|
||||
def why(x) -> Literal["quota", "gone"]:
|
||||
return "quota"
|
||||
def how(x) -> str:
|
||||
return "x"
|
||||
|
||||
def f(x):
|
||||
raise HTTPException(409, {"code": "youtube", "reason": why(x)})
|
||||
def g(x):
|
||||
raise HTTPException(409, {"code": "youtube", "reason": how(x)})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "youtube")["reasons"] == ["gone", "quota"]
|
||||
assert _unread(r, "youtube") == ["how(x)"]
|
||||
|
||||
|
||||
def test_a_module_constant_is_read_only_as_mod_name_and_only_when_bound_once():
|
||||
r = _scan(
|
||||
urlguard="""
|
||||
REASON_MALFORMED = "malformed"
|
||||
TWICE = "a"
|
||||
TWICE = "b"
|
||||
if FLAG:
|
||||
LATER = "c"
|
||||
""",
|
||||
routes__a="""
|
||||
from app import urlguard
|
||||
|
||||
BARE = "bare"
|
||||
|
||||
def f():
|
||||
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.REASON_MALFORMED})
|
||||
def g():
|
||||
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.TWICE})
|
||||
def h():
|
||||
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.LATER})
|
||||
def i():
|
||||
raise HTTPException(409, {"code": "url_blocked", "reason": BARE})
|
||||
""",
|
||||
)
|
||||
assert _code(r, "url_blocked")["reasons"] == ["malformed"]
|
||||
assert _unread(r, "url_blocked") == ["urlguard.TWICE", "urlguard.LATER", "BARE"]
|
||||
|
||||
|
||||
def test_a_name_imported_from_two_places_or_under_an_if_is_not_looked_up():
|
||||
r = _scan(
|
||||
m1='R = Literal["one"]',
|
||||
m2='R = Literal["two"]',
|
||||
a="""
|
||||
def f():
|
||||
from app.m1 import R
|
||||
def g():
|
||||
from app.m2 import R
|
||||
def h(reason: R):
|
||||
return HTTPException(409, {"code": "edit", "reason": reason})
|
||||
""",
|
||||
b="""
|
||||
if FLAG:
|
||||
from app.m1 import R
|
||||
def h(reason: R):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
""",
|
||||
c="""
|
||||
from app.m1 import R
|
||||
def h(reason: R):
|
||||
return HTTPException(409, {"code": "clip", "reason": reason})
|
||||
""",
|
||||
)
|
||||
assert _code(r, "edit")["reasons"] == [] and _unread(r, "edit") == ["reason"]
|
||||
assert _code(r, "playlist")["reasons"] == [] and _unread(r, "playlist") == ["reason"]
|
||||
assert _code(r, "clip")["reasons"] == ["one"]
|
||||
|
||||
|
||||
def test_a_caught_exception_answers_with_the_literal_its_class_declares_on_self():
|
||||
r = _scan(
|
||||
errors="""
|
||||
from typing import Literal
|
||||
ClipReason = Literal["bad_range", "too_short"]
|
||||
EditReason = Literal["not_media"]
|
||||
|
||||
class ClipError(Exception):
|
||||
def __init__(self, reason: ClipReason, **params: float):
|
||||
self.reason: ClipReason = reason
|
||||
self.params = params
|
||||
|
||||
class EditError(Exception):
|
||||
reason: EditReason
|
||||
|
||||
class Untyped(Exception):
|
||||
def __init__(self, reason: str):
|
||||
self.reason = reason
|
||||
|
||||
class Narrower(ClipError):
|
||||
pass
|
||||
""",
|
||||
routes__a="""
|
||||
from app import errors
|
||||
|
||||
def f():
|
||||
try:
|
||||
pass
|
||||
except errors.ClipError as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except (errors.ClipError, errors.EditError) as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except errors.EditError as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except errors.Untyped as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except errors.Narrower as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e.reason})
|
||||
except errors.ClipError as e:
|
||||
raise HTTPException(422, {"code": "clip", "reason": e})
|
||||
""",
|
||||
)
|
||||
assert _code(r, "clip")["reasons"] == ["bad_range", "too_short"]
|
||||
assert _unread(r, "clip") == [
|
||||
"e.reason", # a tuple of classes
|
||||
"e.reason (EditError.reason has no Literal type)", # declared on the class body only
|
||||
"e.reason (Untyped.reason has no Literal type)",
|
||||
"e.reason (Narrower.reason has no Literal type)", # a subclass: not looked up the bases
|
||||
"e", # the exception itself
|
||||
]
|
||||
|
||||
|
||||
def test_a_parameter_answers_with_its_literal_or_its_class_and_any_other_type_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
from typing import Literal
|
||||
Why = Literal["one", "two"]
|
||||
|
||||
class Err(Exception):
|
||||
def __init__(self, reason: Literal["three"]):
|
||||
self.reason: Literal["three"] = reason
|
||||
|
||||
def typed(reason: Why):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
def inline(reason: Literal["four"]):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
def by_class(exc: Err):
|
||||
return HTTPException(409, {"code": "playlist", "reason": exc.reason})
|
||||
def rebound(exc: Err):
|
||||
exc = other()
|
||||
return HTTPException(409, {"code": "playlist", "reason": exc.reason})
|
||||
def optional(reason: Literal["five"] | None = None):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
def quoted(reason: "Why"):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
def number(reason: int):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
|
||||
class C:
|
||||
def method(self, reason: str):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "playlist")["reasons"] == ["four", "one", "three", "two"]
|
||||
unread = _unread(r, "playlist")
|
||||
# a parameter bound again, `| None`, a quoted annotation, `int`
|
||||
assert unread[:4] == ["exc.reason", "reason", "reason", "reason"]
|
||||
assert "method(reason) is not a module-level helper" in unread[4]
|
||||
assert len(unread) == 5
|
||||
|
||||
|
||||
def test_a_str_helper_nobody_calls_sends_nothing():
|
||||
r = _scan(
|
||||
a="""
|
||||
def helper(reason: str):
|
||||
return HTTPException(409, {"code": "playlist", "reason": reason})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "playlist") == {"reasons": [], "unread": [], "sites": 1, "reasonless": []}
|
||||
|
||||
|
||||
def test_a_map_lookup_is_its_values_and_its_keys_must_be_exactly_what_the_index_can_be():
|
||||
src = """
|
||||
from typing import Literal
|
||||
SettingReason = Literal["too_low", "too_high"]
|
||||
|
||||
class InvalidSetting(ValueError):
|
||||
def __init__(self, key: str, reason: SettingReason, **params: object):
|
||||
self.reason: SettingReason = reason
|
||||
self.params = params
|
||||
|
||||
_MAP: dict[SettingReason, str] = {{{keys}}}
|
||||
|
||||
def refused(status: int, reason: str, **params: object):
|
||||
return HTTPException(status, {{**params, "code": "refused", "reason": reason}})
|
||||
|
||||
def setting_refused(exc: InvalidSetting):
|
||||
return refused(400, _MAP[{index}], **exc.params)
|
||||
"""
|
||||
both = '"too_low": "setting_too_low", "too_high": "setting_too_high"'
|
||||
whole = _scan(a=src.format(keys=both, index="exc.reason"))
|
||||
assert _code(whole, "refused")["reasons"] == ["setting_too_high", "setting_too_low"]
|
||||
assert whole["errors"] == [] and whole["unreadCodes"] == []
|
||||
for keys, index in [
|
||||
('"too_low": "setting_too_low"', "exc.reason"), # a key short
|
||||
(both + ', "gone": "setting_gone"', "exc.reason"), # a key no index can be
|
||||
(both, "exc.key"), # an index the scan cannot read
|
||||
]:
|
||||
r = _scan(a=src.format(keys=keys, index=index))
|
||||
assert len(r["errors"]) == 1 and "not what its index can be" in r["errors"][0]
|
||||
|
||||
|
||||
def test_a_lookup_in_anything_but_a_module_dict_of_literal_keys_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
BUILT = make()
|
||||
NUMBERED = {1: "one"}
|
||||
|
||||
def f(table, k):
|
||||
raise HTTPException(409, {"code": "edit", "reason": table[k]})
|
||||
def g(k):
|
||||
raise HTTPException(409, {"code": "edit", "reason": BUILT[k]})
|
||||
def h(k):
|
||||
raise HTTPException(409, {"code": "edit", "reason": NUMBERED[k]})
|
||||
"""
|
||||
)
|
||||
assert _unread(r, "edit") == ["table[k]", "BUILT[k]", "NUMBERED[k]"]
|
||||
assert r["errors"] == []
|
||||
|
||||
|
||||
def test_a_code_must_be_one_literal():
|
||||
r = _scan(
|
||||
a="""
|
||||
BUSY = "busy"
|
||||
def f(code):
|
||||
raise HTTPException(409, {"code": "busy"})
|
||||
def g(code):
|
||||
raise HTTPException(409, {"code": code})
|
||||
def h():
|
||||
raise HTTPException(409, {"code": BUSY})
|
||||
def i(x):
|
||||
raise HTTPException(409, {"code": "a" if x else "b"})
|
||||
def j(x):
|
||||
raise HTTPException(409, {"code": "a" if x else None})
|
||||
def k(key):
|
||||
raise HTTPException(409, {"code": "edit", key: "late"})
|
||||
"""
|
||||
)
|
||||
assert r["codes"]["busy"]["reasonless"] == ["a.py:4"]
|
||||
assert set(r["codes"]) == {"busy"}
|
||||
assert len(r["unreadCodes"]) == 5
|
||||
assert r["unreadCodes"][-1].endswith(
|
||||
"(a spread or a computed key the code and the reason do not follow)"
|
||||
)
|
||||
|
||||
|
||||
def test_a_spread_or_a_computed_key_stands_before_both_the_code_and_the_reason():
|
||||
def unread_codes(detail: str) -> list[str]:
|
||||
return _scan(a=f"def f(x, more, key):\n raise HTTPException(409, {detail})\n")[
|
||||
"unreadCodes"
|
||||
]
|
||||
|
||||
assert unread_codes('{**more, "code": "refused", "reason": "a"}') == []
|
||||
assert unread_codes('{key: 1, **more, "reason": "a", "code": "refused"}') == []
|
||||
for late in [
|
||||
'{"code": "refused", "reason": "a", **more}', # the spread may replace both
|
||||
'{"code": "refused", **more, "reason": "a"}', # … the code
|
||||
'{"reason": "a", **more, "code": "refused"}', # … the reason
|
||||
'{**more, "code": "refused"}', # no reason after it: the spread may carry one
|
||||
'{"code": "refused", "reason": "a", key: "x"}', # a key that may be either
|
||||
'dict(**more, code="refused", reason="a")', # dict() raises on a repeated key
|
||||
'dict(more, code="refused", reason="a")',
|
||||
]:
|
||||
assert unread_codes(late), late
|
||||
|
||||
|
||||
def test_a_detail_stands_only_where_it_is_sent():
|
||||
def result(body: str, extra: str = "") -> dict:
|
||||
return _scan(a=f"def f(x, cached, fix):\n{textwrap.indent(body, ' ')}\n{extra}")
|
||||
|
||||
for body in [
|
||||
'detail = {"code": "login", "reason": "a"}\nraise HTTPException(401, detail)',
|
||||
'if (detail := {"code": "login", "reason": "a"}):\n raise HTTPException(401, detail)',
|
||||
'raise HTTPException(401, {"code": "login", "reason": "a"}.copy())',
|
||||
'raise HTTPException(401, fix({"code": "login", "reason": "a"}))',
|
||||
'raise HTTPException(401, cached or {"code": "login", "reason": "a"})',
|
||||
'return JSONResponse({"code": "login", "reason": "a"}, status_code=401)', # the whole body
|
||||
'raise HTTPException(401, None, {"code": "login", "reason": "a"})', # the headers
|
||||
'raise HTTPException(401, headers={"code": "login", "reason": "a"})',
|
||||
'raise HTTPException(401, "a" if {"code": "login", "reason": "a"} else "b")',
|
||||
]:
|
||||
r = result(body)
|
||||
assert r["unreadCodes"] and "login" not in r["codes"], body
|
||||
assert "(a detail put where the scan cannot follow it)" in r["unreadCodes"][0], body
|
||||
held = _scan(a='class C:\n detail = {"code": "login"}\n')
|
||||
assert "(a detail put where the scan cannot follow it)" in held["unreadCodes"][0]
|
||||
|
||||
sent = result(
|
||||
'raise HTTPException(401, {"code": "login", "reason": "a"} if x else {"code": "login"})\n'
|
||||
'raise HTTPException(401, detail=dict(code="login", reason="b"))\n'
|
||||
'return JSONResponse(content={"detail": {"code": "login", "reason": "c"}})'
|
||||
)
|
||||
assert sent["unreadCodes"] == [] and _code(sent, "login")["reasons"] == ["a", "b", "c"]
|
||||
|
||||
|
||||
def test_a_factory_is_a_detail_only_when_each_return_is_one_and_each_call_is_sent():
|
||||
factory = """
|
||||
def quota(e) -> dict:
|
||||
return {"code": "quota", "reason": "files"}
|
||||
"""
|
||||
sent = "def f(e):\n raise HTTPException(422, detail=quota(e))\n"
|
||||
assert _scan(a=factory + sent)["codes"]["quota"]["reasons"] == ["files"]
|
||||
for change in [
|
||||
"def g(e):\n d = quota(e)\n d['code'] = 'other'\n raise HTTPException(422, d)\n",
|
||||
"def g(e):\n raise HTTPException(422, detail=quota(e).copy())\n",
|
||||
"pick = quota\n",
|
||||
]:
|
||||
r = _scan(a=factory + sent + change)
|
||||
assert r["unreadCodes"] and "quota" not in r["codes"], change
|
||||
twice = _scan(a="quota = None\n" + factory + sent) # the def is the last of two bindings
|
||||
assert twice["unreadCodes"] and "quota" not in twice["codes"]
|
||||
for other in [
|
||||
" if e:\n return None\n",
|
||||
" if e:\n return cached\n",
|
||||
" if e:\n return {'code': 'quota', 'reason': 'x'} if e > 1 else None\n",
|
||||
" log({'code': 'quota', 'reason': 'logged'})\n", # a detail it does not return
|
||||
]:
|
||||
bent = factory.replace("-> dict:\n", "-> dict:\n" + other)
|
||||
assert _scan(a=bent + sent)["unreadCodes"], other
|
||||
method = "class C:\n def quota(self):\n return {'code': 'quota', 'reason': 'm'}\n"
|
||||
assert _scan(a=method)["unreadCodes"]
|
||||
dotted = _scan(
|
||||
a=factory,
|
||||
routes__b="import app.a\n\ndef g(e):\n raise HTTPException(422, detail=app.a.quota(e))\n",
|
||||
)
|
||||
assert dotted["unreadCodes"] and "quota" not in dotted["codes"]
|
||||
|
||||
|
||||
def test_a_name_a_lambda_a_comprehension_a_nonlocal_or_a_global_binds_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(svc):
|
||||
why = "invalid"
|
||||
make = lambda why: HTTPException(401, {"code": "login", "reason": why})
|
||||
raise make(svc.state)
|
||||
def g(svc):
|
||||
why = "invalid"
|
||||
raise [HTTPException(401, {"code": "login", "reason": why}) for why in svc.all][0]
|
||||
def h(svc):
|
||||
why = "invalid"
|
||||
def inner():
|
||||
nonlocal why
|
||||
why = svc.reason
|
||||
inner()
|
||||
raise HTTPException(401, {"code": "login", "reason": why})
|
||||
def i(svc):
|
||||
global why
|
||||
why = "invalid"
|
||||
raise HTTPException(401, {"code": "login", "reason": why})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "login")["reasons"] == []
|
||||
assert sorted(_unread(r, "login")) == [
|
||||
"why (bound by a lambda or a comprehension)",
|
||||
"why (bound by a lambda or a comprehension)",
|
||||
"why (named by a nonlocal or a global)",
|
||||
"why (named by a nonlocal or a global)",
|
||||
]
|
||||
|
||||
|
||||
def test_a_local_that_reaches_itself_is_unread():
|
||||
r = _scan(
|
||||
a="""
|
||||
def f(x):
|
||||
why = "a"
|
||||
if x:
|
||||
why = why if x > 1 else "b"
|
||||
raise HTTPException(409, {"code": "edit", "reason": why})
|
||||
def g(x):
|
||||
a = "z"
|
||||
b = a
|
||||
a = b
|
||||
raise HTTPException(409, {"code": "playlist", "reason": a})
|
||||
def h(x):
|
||||
why = "c"
|
||||
if x:
|
||||
raise HTTPException(409, {"code": "busy", "reason": why})
|
||||
raise HTTPException(409, {"code": "busy", "reason": why})
|
||||
"""
|
||||
)
|
||||
assert _code(r, "busy")["reasons"] == ["c"] and _code(r, "busy")["unread"] == []
|
||||
assert _code(r, "edit")["reasons"] == ["a", "b"]
|
||||
assert _unread(r, "edit") == ["why (reached again while it is resolved)"]
|
||||
assert _code(r, "playlist")["reasons"] == ["z"]
|
||||
assert any("reached again while it is resolved" in u for u in _unread(r, "playlist"))
|
||||
|
||||
|
||||
def test_a_module_constant_or_a_map_changed_elsewhere_is_unread():
|
||||
def reasons(urlguard: str, route: str = "", **more: str) -> tuple[list[str], list[str]]:
|
||||
r = _scan(
|
||||
**more,
|
||||
urlguard="REASON = 'malformed'\n_MAP = {'a': 'map_a'}\n" + urlguard,
|
||||
routes__a="from app import urlguard\n"
|
||||
+ route
|
||||
+ "\ndef f(k: Literal['a']):\n"
|
||||
+ " raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard.REASON})\n"
|
||||
+ " raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard._MAP[k]})\n",
|
||||
)
|
||||
return _code(r, "url_blocked")["reasons"], _unread(r, "url_blocked")
|
||||
|
||||
assert reasons("def get(k):\n return _MAP.get(k)\n") == (["malformed", "map_a"], [])
|
||||
other = "from app import other\ndef g():\n other.REASON = 'x'\n" # another module's REASON
|
||||
assert reasons("", other, other="REASON = 'y'\n") == (["malformed", "map_a"], [])
|
||||
shadow = (
|
||||
"def h(*urlguard):\n"
|
||||
" raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard.REASON})\n"
|
||||
)
|
||||
assert reasons("", shadow)[1] == ["urlguard.REASON"]
|
||||
for urlguard, route in [
|
||||
("def reset():\n global REASON\n REASON = 'x'\n", ""),
|
||||
("", "def g():\n urlguard.REASON = 'x'\n"),
|
||||
("", "def g():\n del urlguard.REASON\n"),
|
||||
]:
|
||||
assert reasons(urlguard, route)[0] == ["map_a"], (urlguard, route)
|
||||
for urlguard in [
|
||||
"_MAP['a'] = compute()\n",
|
||||
"def init(x):\n _MAP['a'] = x\n",
|
||||
"_MAP.update(a='other')\n",
|
||||
"def drop():\n del _MAP['a']\n",
|
||||
"register(_MAP)\n",
|
||||
]:
|
||||
got, unread = reasons(urlguard)
|
||||
assert got == ["malformed"] and "(the map is changed after it is built)" in unread[0]
|
||||
|
||||
|
||||
def test_a_helper_bound_twice_is_not_followed():
|
||||
r = _scan(
|
||||
refusal="refused = None\n" + textwrap.dedent(REFUSAL),
|
||||
routes__a="from app.refusal import refused\n\ndef f():\n raise refused(409, 'lost')\n",
|
||||
)
|
||||
assert _code(r, "refused")["reasons"] == []
|
||||
assert _unread(r, "refused")[0].endswith("refused(reason) is not a module-level helper")
|
||||
|
||||
|
||||
def test_a_dotted_path_through_an_app_module_that_goes_nowhere_is_unread():
|
||||
r = scan(
|
||||
{
|
||||
"__init__.py": "",
|
||||
"refusal.py": textwrap.dedent(REFUSAL),
|
||||
"routes/a.py": "import app as a\n\ndef f():\n raise a.refusal.refused(409, 'dotted')\n",
|
||||
}
|
||||
)
|
||||
assert _unread(r, "refused") == ["a.refusal.refused (a reference the scan cannot resolve)"]
|
||||
|
||||
|
||||
def test_a_helper_reached_through_an_app_module_that_does_not_bind_it_is_unread():
|
||||
for shim in [
|
||||
"from app.refusal import *\n",
|
||||
"try:\n from app.refusal import refused\nexcept ImportError:\n refused = None\n",
|
||||
]:
|
||||
r = _scan(
|
||||
refusal=REFUSAL,
|
||||
shim=shim,
|
||||
routes__a="from app.shim import refused\n\ndef f():\n raise refused(409, 'hidden')\n",
|
||||
)
|
||||
assert _unread(r, "refused") == ["refused (a reference the scan cannot resolve)"], shim
|
||||
package = scan(
|
||||
{
|
||||
"pkg/refusal.py": textwrap.dedent(REFUSAL),
|
||||
"pkg/__init__.py": "from .refusal import refused\n",
|
||||
"routes/a.py": "from app.pkg import refused\n\ndef f():\n raise refused(409, 'hidden')\n",
|
||||
}
|
||||
)
|
||||
assert _unread(package, "refused") == ["refused (a reference the scan cannot resolve)"]
|
||||
|
||||
|
||||
def test_the_data_files_are_excluded_by_name_only():
|
||||
r = _scan(
|
||||
plex__tmdb='def f():\n return {"code": "HU", "name": "Hungary"}',
|
||||
plex__other='def f():\n return {"code": "HU"}',
|
||||
)
|
||||
assert r["excluded"] == {"plex/tmdb.py": 1}
|
||||
assert "HU" in r["codes"]
|
||||
|
||||
|
||||
def test_the_app_scans():
|
||||
"""The real tree, on the interpreter the backend runs: every refusal it holds is read (the
|
||||
frontend's contract judges what is read)."""
|
||||
result = scan_app()
|
||||
assert {"refused", "not_found", "login", "clip"} <= set(result["codes"])
|
||||
assert result["errors"] == [] and result["unreadCodes"] == []
|
||||
@@ -101,8 +101,10 @@ def test_a_video_state_cleared_through_the_route_is_gone(db, make_user, as_user)
|
||||
_video(db)
|
||||
db.add(VideoState(user_id=user.id, video_id="v1", status="hidden", position_seconds=90))
|
||||
db.commit()
|
||||
assert as_user(user).delete("/api/videos/v1/state").status_code == 200
|
||||
assert _state(db, user) is None
|
||||
assert as_user(user).delete("/api/videos/v1/progress").status_code == 200
|
||||
db.expire_all()
|
||||
row = _state(db, user)
|
||||
assert (row.status, row.position_seconds) == ("hidden", 0)
|
||||
|
||||
|
||||
# --- saved views ------------------------------------------------------------------------------
|
||||
@@ -308,6 +310,14 @@ def test_clearing_leaves_a_video_the_caller_never_found(db, make_user, as_user):
|
||||
assert r.json() == {"ok": True, "removed": 0}
|
||||
|
||||
|
||||
def test_clearing_refuses_a_body_without_its_list(db, make_user, as_user):
|
||||
# A misnamed key read as an empty list: 200, and nothing cleared.
|
||||
user = make_user()
|
||||
r = as_user(user).post("/api/search/clear", json={"ids": ["s1"]})
|
||||
assert r.status_code == 422
|
||||
assert as_user(user).post("/api/search/clear", json={"video_ids": []}).status_code == 200
|
||||
|
||||
|
||||
# --- Plex show and season marks ---------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -22,6 +22,31 @@ from app.runstatus import RunStatus, RunTrigger
|
||||
_JOB = "unit_test_claim" # not a real job id — exercises the primitive without touching live jobs
|
||||
|
||||
|
||||
class _StrictlyLater(datetime):
|
||||
"""A clock that never repeats or goes back: each reading at least a microsecond after the last."""
|
||||
|
||||
last: datetime | None = None
|
||||
clock: type[datetime] = datetime
|
||||
|
||||
@classmethod
|
||||
def now(cls, tz=None):
|
||||
t = cls.clock.now(tz)
|
||||
if cls.last is not None and t <= cls.last:
|
||||
t = cls.last + timedelta(microseconds=1)
|
||||
cls.last = t
|
||||
return t
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _runs_start_in_order(monkeypatch):
|
||||
"""Runs started in a row start in that order. These tests open several runs within microseconds
|
||||
and read them back newest first, by start time; the wall clock of a container on a VM can step
|
||||
back between two readings (seen: the third of three runs sorted before the second)."""
|
||||
_StrictlyLater.last = None
|
||||
_StrictlyLater.clock = jobruns.datetime
|
||||
monkeypatch.setattr(jobruns, "datetime", _StrictlyLater)
|
||||
|
||||
|
||||
def _release():
|
||||
scheduler._record(_JOB, running=False)
|
||||
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
"""A share link's address says what it opens: /watch/ a video, /file/ anything else, /folder/ a folder.
|
||||
|
||||
Only a word for the reader: the token is the key, so every path opens every link, and the links
|
||||
already sent out on /watch/ keep working for good. DB-lane.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from cryptography.fernet import Fernet
|
||||
|
||||
from app import main, security
|
||||
from app.config import settings
|
||||
from app.downloads import og
|
||||
from app.downloads import probe as probemod
|
||||
|
||||
_PAGE = """<!doctype html><html><head>
|
||||
<!--OG:START--><meta property="og:title" content="Siftlode" /><!--OG:END-->
|
||||
</head><body><script src="/assets/{entry}.js"></script></body></html>"""
|
||||
|
||||
_VIDEO = {
|
||||
"streams": [
|
||||
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
|
||||
{"codec_type": "audio", "codec_name": "aac"},
|
||||
],
|
||||
"format": {"format_name": "mov,mp4", "duration": "5"},
|
||||
}
|
||||
_AUDIO = {
|
||||
"streams": [{"codec_type": "audio", "codec_name": "aac"}],
|
||||
"format": {"format_name": "mov,mp4", "duration": "5"},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pages(tmp_path, monkeypatch):
|
||||
index, watch = tmp_path / "index.html", tmp_path / "watch.html"
|
||||
index.write_text(_PAGE.format(entry="app"), encoding="utf-8")
|
||||
watch.write_text(_PAGE.format(entry="watch"), encoding="utf-8")
|
||||
monkeypatch.setattr(main, "INDEX_HTML", index)
|
||||
monkeypatch.setattr(main, "WATCH_HTML", watch)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def key(monkeypatch):
|
||||
"""A link's address is read back from its encrypted token."""
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def shared(as_user, make_user, tmp_path, monkeypatch, upload_headers):
|
||||
"""Uploads a file ffprobe reads as `data` (None: it cannot read it) and shares it."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path / "media"))
|
||||
owner = as_user(make_user())
|
||||
|
||||
def share(name: str, data: dict | None, **link: object) -> dict:
|
||||
monkeypatch.setattr(probemod, "probe_json", lambda _p: data)
|
||||
job = owner.post(
|
||||
"/api/files/upload",
|
||||
content=b"some bytes",
|
||||
headers=upload_headers(name, display_name=name),
|
||||
)
|
||||
assert job.status_code == 200, job.text
|
||||
made = owner.post(f"/api/downloads/{job.json()['id']}/links", json=link)
|
||||
assert made.status_code == 200, made.text
|
||||
return made.json()
|
||||
|
||||
share.owner = owner # type: ignore[attr-defined]
|
||||
return share
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "data", "prefix"),
|
||||
[
|
||||
("clip.mp4", _VIDEO, "/watch/"),
|
||||
("song.m4a", _AUDIO, "/file/"),
|
||||
("backup.tar.gz", None, "/file/"),
|
||||
],
|
||||
)
|
||||
def test_a_link_is_given_out_on_the_path_of_what_it_opens(shared, name, data, prefix):
|
||||
made = shared(name, data)
|
||||
assert made["url"] == f"{prefix}{made['token']}"
|
||||
|
||||
|
||||
def test_a_folder_link_is_given_out_on_folder(shared, key):
|
||||
owner = shared.owner
|
||||
folder = owner.post("/api/files/folders", json={"name": "Friends"}).json()
|
||||
made = owner.post(f"/api/downloads/folders/{folder['id']}/links", json={}).json()
|
||||
assert made["url"] == f"/folder/{made['token']}"
|
||||
listed = owner.get(f"/api/downloads/folders/{folder['id']}/links").json()
|
||||
assert [entry["url"] for entry in listed] == [made["url"]]
|
||||
|
||||
|
||||
def test_the_list_gives_the_same_address_as_the_making(shared, key):
|
||||
made = shared("backup.tar.gz", None)
|
||||
owner = shared.owner
|
||||
files = owner.get("/api/files").json()
|
||||
job_id = next(f["id"] for f in files if f["display_name"] == "backup.tar.gz")
|
||||
listed = owner.get(f"/api/downloads/{job_id}/links").json()
|
||||
assert [entry["url"] for entry in listed] == [made["url"]]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["watch", "file", "folder"])
|
||||
@pytest.mark.parametrize(
|
||||
("name", "data"), [("clip.mp4", _VIDEO), ("backup.tar.gz", None)], ids=["video", "file"]
|
||||
)
|
||||
def test_every_path_opens_every_link(pages, shared, client, path, name, data):
|
||||
"""The links already sent on /watch/ open whatever they open, for good."""
|
||||
token = shared(name, data)["token"]
|
||||
r = client.get(f"/{path}/{token}")
|
||||
assert r.status_code == 200
|
||||
assert "/assets/watch.js" in r.text and "/assets/app.js" not in r.text
|
||||
assert f'content="{name}"' in r.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["/file/no-such-token", "/folder/no-such-token", "/file/a/b"])
|
||||
def test_a_new_path_without_a_live_link_still_gets_the_player(pages, client, path):
|
||||
r = client.get(path)
|
||||
assert r.status_code == 200
|
||||
assert "/assets/watch.js" in r.text and "/assets/app.js" not in r.text
|
||||
|
||||
|
||||
def test_a_card_names_a_file_as_a_file(shared, db):
|
||||
made = shared("backup.tar.gz", None)
|
||||
tags = og._watch_tags(made["token"], db)
|
||||
assert f'{made["url"]}"' in tags
|
||||
assert "video.other" not in tags
|
||||
|
||||
locked = shared("notes.tar.gz", None, password="pw")
|
||||
hidden = og._watch_tags(locked["token"], db)
|
||||
assert "Password-protected file" in hidden and "notes" not in hidden
|
||||
assert "video.other" not in hidden
|
||||
|
||||
|
||||
def test_a_card_of_a_video_stays_a_video(shared, db):
|
||||
made = shared("clip.mp4", _VIDEO)
|
||||
tags = og._watch_tags(made["token"], db)
|
||||
assert f"/watch/{made['token']}" in tags and "video.other" in tags
|
||||
locked = shared("other.mp4", _VIDEO, password="pw")
|
||||
assert "Password-protected video" in og._watch_tags(locked["token"], db)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "data", "video"),
|
||||
[("clip.mp4", _VIDEO, True), ("song.m4a", _AUDIO, False), ("backup.tar.gz", None, False)],
|
||||
)
|
||||
def test_a_password_link_says_whether_it_opens_a_video(shared, client, name, data, video):
|
||||
"""The lock screen asks to watch a video and to open anything else, whichever path it is on."""
|
||||
token = shared(name, data, password="pw")["token"]
|
||||
meta = client.get(f"/api/public/watch/{token}").json()
|
||||
assert meta == {"needs_password": True, "kind": "file", "video": video}
|
||||
@@ -0,0 +1,223 @@
|
||||
"""A seeded shuffle walks a stored random key instead of sorting the whole scope. DB lane.
|
||||
|
||||
Every video carries a random number drawn once (`Video.shuffle_key`). A seed picks a point on that
|
||||
ring; the feed's Shuffle sort pages along the ring from there and wraps, and the Home surprise reads
|
||||
a window of it and shuffles that window by the seed (`discover.queries.yt_surprise`). Ordering the
|
||||
scope by md5(id || seed) cost ~350 ms per call on prod and grew with every Shuffle page.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import random
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import event, insert, text
|
||||
|
||||
from app import feedquery
|
||||
from app.catalog.models import Channel, Subscription, Video
|
||||
from app.discover import queries
|
||||
from app.discover.queries import _FEED_BASE
|
||||
|
||||
NOW = datetime.now(timezone.utc)
|
||||
KEYS = {"k05": 0.05, "k20": 0.20, "k40": 0.40, "k60": 0.60, "k80": 0.80, "k95": 0.95}
|
||||
|
||||
|
||||
def _seed(db, make_user, keys=KEYS):
|
||||
user = make_user()
|
||||
db.add(Channel(id="UCring", title="Ring"))
|
||||
db.flush()
|
||||
db.add(Subscription(user_id=user.id, channel_id="UCring"))
|
||||
db.add_all(
|
||||
[
|
||||
Video(
|
||||
id=vid,
|
||||
channel_id="UCring",
|
||||
title=vid,
|
||||
duration_seconds=20 * 60,
|
||||
published_at=NOW - timedelta(days=i + 1),
|
||||
shuffle_key=key,
|
||||
)
|
||||
for i, (vid, key) in enumerate(keys.items())
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
return user
|
||||
|
||||
|
||||
def _ring(seed, keys=KEYS):
|
||||
"""The ring order from the seed's point: the keys at or after it ascending, then the rest."""
|
||||
r = feedquery.shuffle_start(seed)
|
||||
return sorted(keys, key=lambda v: (keys[v] < r, keys[v]))
|
||||
|
||||
|
||||
def _seed_landing_between(lo, hi):
|
||||
return next(s for s in range(1000) if lo < feedquery.shuffle_start(s) < hi)
|
||||
|
||||
|
||||
def test_the_walk_starts_at_the_seeds_point_and_wraps(db, make_user):
|
||||
user = _seed(db, make_user)
|
||||
query, _ = feedquery.filtered_query(db, user, **_FEED_BASE)
|
||||
seed = _seed_landing_between(0.45, 0.55) # starts at k60, wraps to k05 after k95
|
||||
got = [r.id for r in feedquery.shuffle_walk(db, query, seed, 6)]
|
||||
assert got == ["k60", "k80", "k95", "k05", "k20", "k40"]
|
||||
assert [r.id for r in feedquery.shuffle_walk(db, query, seed, 4)] == got[:4]
|
||||
|
||||
|
||||
def test_the_feed_shuffle_pages_along_the_ring(db, make_user, as_user):
|
||||
user = _seed(db, make_user)
|
||||
client = as_user(user)
|
||||
for seed in (_seed_landing_between(0.1, 0.15), _seed_landing_between(0.85, 0.9)):
|
||||
walked, cursor = [], None
|
||||
for _ in range(5):
|
||||
url = f"/api/feed?sort=shuffle&seed={seed}&limit=2" + (
|
||||
f"&cursor={cursor}" if cursor else ""
|
||||
)
|
||||
body = client.get(url).json()
|
||||
walked += [it["id"] for it in body["items"]]
|
||||
cursor = body["next_cursor"]
|
||||
if not body["has_more"]:
|
||||
break
|
||||
assert walked == _ring(seed), seed
|
||||
|
||||
|
||||
def test_a_new_seed_opens_the_shuffle_elsewhere(db, make_user, as_user):
|
||||
user = _seed(db, make_user)
|
||||
client = as_user(user)
|
||||
first = {
|
||||
seed: client.get(f"/api/feed?sort=shuffle&seed={seed}&limit=2").json()["items"][0]["id"]
|
||||
for seed in (_seed_landing_between(0.1, 0.15), _seed_landing_between(0.65, 0.7))
|
||||
}
|
||||
assert sorted(first.values()) == ["k20", "k80"]
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _statements(db):
|
||||
seen: list[str] = []
|
||||
|
||||
def grab(conn, cursor, statement, *args):
|
||||
seen.append(statement)
|
||||
|
||||
engine = db.get_bind()
|
||||
event.listen(engine, "before_cursor_execute", grab)
|
||||
try:
|
||||
yield seen
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", grab)
|
||||
|
||||
|
||||
def _draws_of_videos(seen):
|
||||
return [s for s in seen if "FROM videos" in s and "LIMIT" in s]
|
||||
|
||||
|
||||
def test_neither_draw_orders_the_scope_by_a_hash(db, make_user):
|
||||
# Each draw reads the videos as an ascending range of the key (what ix_videos_shuffle_key serves),
|
||||
# never ordered by a hash or by the ring's CASE (which no index can serve).
|
||||
user = _seed(db, make_user)
|
||||
with _statements(db) as surprise:
|
||||
queries.yt_surprise(db, user, "any", 7, 3)
|
||||
with _statements(db) as page:
|
||||
feedquery.feed_page(
|
||||
db, user, dict(_FEED_BASE), sort="shuffle", seed=7, limit=2, cursor=None, lang="en"
|
||||
)
|
||||
for seen in (surprise, page):
|
||||
draws = _draws_of_videos(seen)
|
||||
assert draws
|
||||
for s in draws:
|
||||
assert "md5(" not in s.lower()
|
||||
assert "ORDER BY videos.shuffle_key, videos.id" in s.replace("\n", " "), s
|
||||
|
||||
|
||||
def test_the_walk_reads_the_ring_index_and_sorts_nothing(db, make_user):
|
||||
# The text of the query is not the plan: each arc must be served by ix_videos_shuffle_key in key
|
||||
# order (an index on another shape, or none, leaves a Sort of the whole scope). A scope of a few
|
||||
# thousand videos, analysed, so the planner weighs it as it does a real one.
|
||||
user = _seed(db, make_user)
|
||||
rng = random.Random(7)
|
||||
db.execute(
|
||||
insert(Video),
|
||||
[
|
||||
{
|
||||
"id": f"bulk{i:05}",
|
||||
"channel_id": "UCring",
|
||||
"title": "b",
|
||||
"duration_seconds": 600,
|
||||
"published_at": NOW - timedelta(minutes=i),
|
||||
"shuffle_key": rng.random(),
|
||||
}
|
||||
for i in range(4000)
|
||||
],
|
||||
)
|
||||
db.commit()
|
||||
db.execute(text("ANALYZE videos"))
|
||||
query, _ = feedquery.filtered_query(db, user, **_FEED_BASE)
|
||||
captured: list[tuple[str, object]] = []
|
||||
|
||||
def grab(conn, cursor, statement, params, *args):
|
||||
if "FROM videos" in statement and "LIMIT" in statement:
|
||||
captured.append((statement, params))
|
||||
|
||||
engine = db.get_bind()
|
||||
event.listen(engine, "before_cursor_execute", grab)
|
||||
try:
|
||||
seed = _seed_landing_between(0.45, 0.55)
|
||||
feedquery.shuffle_walk(db, query, seed, 6) # the first arc, from the seed's point
|
||||
feedquery.shuffle_walk(db, query, seed, 6, (1, 0.3, "bulk00001")) # the wrap, resumed
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", grab)
|
||||
assert len(captured) == 2
|
||||
conn = db.connection()
|
||||
for statement, params in captured:
|
||||
plan = "\n".join(r[0] for r in conn.exec_driver_sql("EXPLAIN " + statement, params))
|
||||
assert "ix_videos_shuffle_key" in plan, plan
|
||||
assert "Sort Key: videos.shuffle_key" not in plan, plan
|
||||
|
||||
|
||||
def test_the_surprise_reads_a_window_of_the_ring(db, make_user, monkeypatch):
|
||||
user = _seed(db, make_user)
|
||||
monkeypatch.setattr(queries, "SURPRISE_WINDOW", 3)
|
||||
for seed in range(20):
|
||||
drawn = {r["id"] for r in queries.yt_surprise(db, user, "any", seed, 3)}
|
||||
assert drawn == set(_ring(seed)[:3]), seed
|
||||
|
||||
|
||||
def test_a_wide_gap_on_the_ring_does_not_make_a_favourite(db, make_user):
|
||||
# One video after an empty half of the ring, twenty packed into the other half. Taking the first
|
||||
# video after the seed's point would deal the lonely one about half the time; the window is
|
||||
# shuffled by the seed, so it leads about as often as any other.
|
||||
keys = {"lonely": 0.5} | {f"p{i:02}": 0.51 + i * 0.02 for i in range(20)}
|
||||
user = _seed(db, make_user, keys)
|
||||
leads = [queries.yt_surprise(db, user, "any", s, 1)[0]["id"] for s in range(90)]
|
||||
assert leads.count("lonely") < 15, leads.count("lonely")
|
||||
|
||||
|
||||
def test_any_seed_lands_on_the_ring():
|
||||
# An int too large for a float overflowed the multiplication: a 500 for a hand-typed seed.
|
||||
for seed in (10**400, -(10**400), 2**53 + 1, -7):
|
||||
assert 0 <= feedquery.shuffle_start(seed) < 1, seed
|
||||
|
||||
|
||||
def test_a_seed_out_of_range_is_refused_not_a_500(db, make_user, as_user):
|
||||
client = as_user(_seed(db, make_user))
|
||||
huge = "9" * 400
|
||||
for url in (
|
||||
f"/api/feed?sort=shuffle&seed={huge}",
|
||||
f"/api/discover?seed={huge}",
|
||||
f"/api/discover/pick?seed={huge}",
|
||||
):
|
||||
assert client.get(url).status_code == 422, url
|
||||
|
||||
|
||||
def test_a_shuffle_cursor_past_a_float_is_a_bad_cursor(db, make_user, as_user):
|
||||
client = as_user(_seed(db, make_user))
|
||||
for k in ("[Infinity, 0.5]", f"[0, {'9' * 400}]"):
|
||||
raw = '{"k": %s, "id": "k20"}' % k
|
||||
cursor = base64.urlsafe_b64encode(raw.encode()).decode()
|
||||
assert client.get(f"/api/feed?sort=shuffle&seed=1&cursor={cursor}").status_code == 400, k
|
||||
|
||||
|
||||
def test_a_new_video_gets_its_own_key(db, make_user):
|
||||
_seed(db, make_user, {})
|
||||
db.add_all([Video(id=f"n{i}", channel_id="UCring", title="n") for i in range(3)])
|
||||
db.commit()
|
||||
keys = [db.get(Video, f"n{i}").shuffle_key for i in range(3)]
|
||||
assert len(set(keys)) == 3 and all(0 <= k < 1 for k in keys)
|
||||
@@ -0,0 +1,349 @@
|
||||
"""Surprise me leans on what the user watches, and stays a surprise.
|
||||
|
||||
The Home hero's surprise tier and the Plex Discover rail were uniform draws over the unseen. They are
|
||||
now weighted draws (`app.draw`): a title's weight is 1 + its affinity — for Plex the genres of the
|
||||
films and shows the user watched (recent ones weigh more) and the TMDB recommendations stored on them;
|
||||
for YouTube the channels the user watches. One slot of three stays a blind draw, and with no watch
|
||||
history the order is exactly the old one. DB lane.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app import draw
|
||||
from app.catalog.models import Channel, Subscription, Video
|
||||
from app.discover import queries
|
||||
from app.plex import affinity, discover as plex_discover
|
||||
from app.plex.models import PlexItem, PlexLibrary, PlexShow, PlexState
|
||||
from app.plex.query import LibraryFilters
|
||||
from app.reader.models import VideoState
|
||||
|
||||
NOW = datetime.now(timezone.utc)
|
||||
SEEDS = range(90)
|
||||
|
||||
|
||||
# ── the draw itself ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_every_third_slot_is_blind_and_nothing_repeats():
|
||||
weighted = ["a", "b", "c", "d", "e", "f"]
|
||||
blind = ["f", "e", "d", "c", "b", "a"]
|
||||
# offset 0: slots 0 and 3 are blind
|
||||
assert draw.blend(weighted, blind, 0, 6) == ["f", "a", "b", "e", "c", "d"]
|
||||
# offset 1: slot 2 and 5
|
||||
assert draw.blend(weighted, blind, 1, 6) == ["a", "b", "f", "c", "d", "e"]
|
||||
assert draw.blend(weighted, blind, 2, 3) == ["a", "f", "b"]
|
||||
|
||||
|
||||
def test_a_short_side_is_topped_up_from_the_other():
|
||||
assert draw.blend(["a"], ["a", "b", "c"], 0, 3) == ["a", "b", "c"]
|
||||
assert draw.blend(["a", "b", "c"], [], 0, 3) == ["a", "b", "c"]
|
||||
|
||||
|
||||
def test_a_watch_weighs_by_its_age_an_undated_one_two_half_lives_and_a_future_one_now():
|
||||
# Both affinities weigh every watch through this one rule; an old import has no date, and a
|
||||
# clock ahead of the server's must not weigh more than a watch made now.
|
||||
assert draw.recency(NOW, NOW, 30) == 1.0
|
||||
assert draw.recency(NOW - timedelta(days=30), NOW, 30) == 0.5
|
||||
assert draw.recency(None, NOW, 30) == 0.25
|
||||
assert draw.recency(NOW + timedelta(days=5), NOW, 30) == 1.0
|
||||
|
||||
|
||||
def test_scores_are_scaled_to_the_strongest_and_nothing_scored_is_empty():
|
||||
assert draw.scaled({"a": 4.0, "b": 1.0}) == {"a": 1.0, "b": 0.25}
|
||||
assert draw.scaled({}) == {}
|
||||
|
||||
|
||||
# ── YouTube: the channels the user watches ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _md5_order(ids, seed):
|
||||
return sorted(ids, key=lambda i: hashlib.md5(f"{i}{seed}".encode()).hexdigest())
|
||||
|
||||
|
||||
def _seed_yt(db, make_user, *, watched_in_fav: int):
|
||||
user = make_user()
|
||||
fav, other = Channel(id="UCfav", title="Fav"), Channel(id="UCother", title="Other")
|
||||
db.add_all([fav, other])
|
||||
db.flush()
|
||||
db.add_all(
|
||||
[
|
||||
Subscription(user_id=user.id, channel_id="UCfav"),
|
||||
Subscription(user_id=user.id, channel_id="UCother"),
|
||||
]
|
||||
)
|
||||
|
||||
def video(vid, ch, days):
|
||||
return Video(
|
||||
id=vid,
|
||||
channel_id=ch,
|
||||
title=vid,
|
||||
duration_seconds=20 * 60,
|
||||
published_at=NOW - timedelta(days=days),
|
||||
)
|
||||
|
||||
# One unseen video per channel, plus the user's watched history on the favourite.
|
||||
db.add_all([video("fav_new", "UCfav", 1), video("other_new", "UCother", 1)])
|
||||
for i in range(watched_in_fav):
|
||||
db.add(video(f"fav_seen{i}", "UCfav", 10 + i))
|
||||
db.flush()
|
||||
for i in range(watched_in_fav):
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id=f"fav_seen{i}",
|
||||
status="watched",
|
||||
watched_at=NOW - timedelta(days=i),
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
return user
|
||||
|
||||
|
||||
def _first(rows):
|
||||
return rows[0]["id"] if rows else None
|
||||
|
||||
|
||||
def test_a_watched_channel_wins_most_draws_but_not_all(db, make_user):
|
||||
user = _seed_yt(db, make_user, watched_in_fav=12)
|
||||
leads = [_first(queries.yt_surprise(db, user, "any", s, 2)) for s in SEEDS]
|
||||
fav = leads.count("fav_new") / len(leads)
|
||||
# weight 1 + 3 against 1: 0.8 in a weighted slot, 0.5 in the blind one (a third of the seeds)
|
||||
assert 0.6 < fav < 0.9, fav
|
||||
|
||||
|
||||
def test_the_blind_slot_deals_the_plain_shuffle_however_strong_the_lean(db, make_user):
|
||||
user = _seed_yt(db, make_user, watched_in_fav=12)
|
||||
for s in range(0, 30, 3): # (0 + seed) % 3 == 0: the first slot is the blind one
|
||||
got = _first(queries.yt_surprise(db, user, "any", s, 2))
|
||||
assert got == _md5_order(["fav_new", "other_new"], s)[0]
|
||||
|
||||
|
||||
def test_with_no_history_the_youtube_order_is_the_old_shuffle(db, make_user):
|
||||
user = _seed_yt(db, make_user, watched_in_fav=0)
|
||||
for s in range(12):
|
||||
got = [r["id"] for r in queries.yt_surprise(db, user, "any", s, 2)]
|
||||
assert got == _md5_order(["fav_new", "other_new"], s)
|
||||
|
||||
|
||||
def test_a_watched_video_is_never_drawn(db, make_user):
|
||||
user = _seed_yt(db, make_user, watched_in_fav=12)
|
||||
drawn = {r["id"] for s in SEEDS for r in queries.yt_surprise(db, user, "any", s, 5)}
|
||||
assert drawn == {"fav_new", "other_new"}
|
||||
|
||||
|
||||
# ── Plex: genres and recommendations of what the user watched ────────────────────────────────────
|
||||
|
||||
|
||||
def _seed_plex(db, make_user, *, history: bool):
|
||||
user = make_user()
|
||||
lib = PlexLibrary(plex_key="77", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
tmdb_ids = iter(range(5000, 6000))
|
||||
|
||||
def movie(key, genres, recs=None):
|
||||
m = PlexItem(
|
||||
rating_key=key,
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title=key,
|
||||
duration_s=100 * 60,
|
||||
genres=genres,
|
||||
guid=f"tmdb:{next(tmdb_ids)}",
|
||||
recommendations=recs,
|
||||
)
|
||||
db.add(m)
|
||||
return m
|
||||
|
||||
horror = movie("horror_new", ["Horror"])
|
||||
comedy = movie("comedy_new", ["Comedy"])
|
||||
tipped = movie("tipped_new", ["Western"])
|
||||
db.flush()
|
||||
tip = [{"tmdb_id": int(tipped.guid.split(":")[1]), "media_type": "movie", "title": "t"}]
|
||||
seen = [movie(f"horror_seen{i}", ["Horror"], tip if i == 0 else None) for i in range(3)]
|
||||
db.flush()
|
||||
if history:
|
||||
for i, m in enumerate(seen):
|
||||
db.add(
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=m.id,
|
||||
status="watched",
|
||||
watched_at=NOW - timedelta(days=i),
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
from app import sysconfig
|
||||
|
||||
sysconfig.set_value(db, "plex_enabled", True)
|
||||
db.commit()
|
||||
return user, horror, comedy, tipped
|
||||
|
||||
|
||||
def test_the_genre_and_the_recommendation_of_what_was_watched_win_most_draws(db, make_user):
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
leads = [_first(queries.plex_surprise(db, user.id, "any", s, 3, lang="en")) for s in SEEDS]
|
||||
# Weights: Horror 1 + 2 (its genre), the recommended Western 1 + 3, the Comedy nobody chose 1. In
|
||||
# a weighted slot the Comedy leads 1 time in 8, in the blind third 1 in 3: about 0.19 against
|
||||
# 0.81 for the other two together (a uniform draw: 1 in 3 each). It still leads now and then —
|
||||
# a surprise, not a ranking.
|
||||
comedy = leads.count("comedy_new")
|
||||
assert comedy > 0
|
||||
assert leads.count("horror_new") > 1.5 * comedy and leads.count("tipped_new") > 1.5 * comedy
|
||||
assert leads.count("horror_new") + leads.count("tipped_new") > 3 * comedy
|
||||
|
||||
|
||||
def test_with_no_history_the_plex_order_is_the_old_shuffle(db, make_user):
|
||||
user, *_ = _seed_plex(db, make_user, history=False)
|
||||
keys = ["horror_new", "comedy_new", "tipped_new"] + [f"horror_seen{i}" for i in range(3)]
|
||||
for s in range(12):
|
||||
got = [c["id"] for c in queries.plex_surprise(db, user.id, "any", s, 6, lang="en")]
|
||||
assert got == _md5_order(keys, s)
|
||||
|
||||
|
||||
def test_a_watched_film_is_never_drawn(db, make_user):
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
drawn = {
|
||||
c["id"] for s in SEEDS for c in queries.plex_surprise(db, user.id, "any", s, 6, lang="en")
|
||||
}
|
||||
assert drawn == {"horror_new", "comedy_new", "tipped_new"}
|
||||
|
||||
|
||||
def test_the_plex_blind_slot_deals_the_plain_shuffle(db, make_user):
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
unseen = ["horror_new", "comedy_new", "tipped_new"]
|
||||
for s in range(0, 30, 3):
|
||||
got = _first(queries.plex_surprise(db, user.id, "any", s, 3, lang="en"))
|
||||
assert got == _md5_order(unseen, s)[0]
|
||||
|
||||
|
||||
def test_a_binged_show_counts_once_and_recent_watches_weigh_more(db, make_user):
|
||||
user = make_user()
|
||||
lib = PlexLibrary(plex_key="78", title="Shows", kind="show", enabled=True)
|
||||
mlib = PlexLibrary(plex_key="79", title="Movies", kind="movie", enabled=True)
|
||||
db.add_all([lib, mlib])
|
||||
db.flush()
|
||||
show = PlexShow(rating_key="sitcom", library_id=lib.id, title="Sitcom", genres=["Comedy"])
|
||||
db.add(show)
|
||||
db.flush()
|
||||
eps = [
|
||||
PlexItem(
|
||||
rating_key=f"ep{i}",
|
||||
library_id=lib.id,
|
||||
kind="episode",
|
||||
title=f"e{i}",
|
||||
show_id=show.id,
|
||||
season_number=1,
|
||||
episode_number=i,
|
||||
)
|
||||
for i in range(10)
|
||||
]
|
||||
films = [
|
||||
PlexItem(
|
||||
rating_key=f"h{i}", library_id=mlib.id, kind="movie", title=f"h{i}", genres=["Horror"]
|
||||
)
|
||||
for i in range(2)
|
||||
]
|
||||
old = PlexItem(
|
||||
rating_key="w0", library_id=mlib.id, kind="movie", title="w0", genres=["Western"]
|
||||
)
|
||||
db.add_all([*eps, *films, old])
|
||||
db.flush()
|
||||
for it in [*eps, *films]:
|
||||
db.add(PlexState(user_id=user.id, item_id=it.id, status="watched", watched_at=NOW))
|
||||
db.add(
|
||||
PlexState(
|
||||
user_id=user.id, item_id=old.id, status="watched", watched_at=NOW - timedelta(days=720)
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
genres = affinity.for_user(db, user.id).genres
|
||||
# Ten episodes of one show are one title: two horror films outweigh it.
|
||||
assert genres["Horror"] == 1.0
|
||||
assert genres["Comedy"] == 0.5
|
||||
# Four half-lives ago weighs a sixteenth of today.
|
||||
assert abs(genres["Western"] - 0.5 / 16) < 0.01
|
||||
|
||||
|
||||
def test_the_discover_rail_leans_the_same_way(db, make_user):
|
||||
# The rail draws with random(), not a seed; setseed makes this session's draws repeatable.
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
db.execute(text("SELECT setseed(0.42)"))
|
||||
leads = [
|
||||
plex_discover.surprise(db, user, "movie", "en", 3, [], [], LibraryFilters())["items"][0][
|
||||
"id"
|
||||
]
|
||||
for _ in SEEDS
|
||||
]
|
||||
assert leads.count("comedy_new") > 0
|
||||
assert leads.count("horror_new") + leads.count("tipped_new") > 3 * leads.count("comedy_new")
|
||||
|
||||
|
||||
# ── whose history: only the user's own ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_another_users_youtube_history_does_not_lean_my_draw(db, make_user):
|
||||
watcher = _seed_yt(db, make_user, watched_in_fav=12)
|
||||
me = make_user()
|
||||
db.add_all(
|
||||
[
|
||||
Subscription(user_id=me.id, channel_id="UCfav"),
|
||||
Subscription(user_id=me.id, channel_id="UCother"),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
assert queries._channel_affinity(db, watcher)
|
||||
assert queries._channel_affinity(db, me) == {}
|
||||
|
||||
|
||||
def test_another_users_plex_history_does_not_lean_my_draw(db, make_user):
|
||||
_seed_plex(db, make_user, history=True)
|
||||
me = make_user()
|
||||
assert not affinity.for_user(db, me.id)
|
||||
keys = ["horror_new", "comedy_new", "tipped_new"] + [f"horror_seen{i}" for i in range(3)]
|
||||
for s in range(12):
|
||||
got = [c["id"] for c in queries.plex_surprise(db, me.id, "any", s, 6, lang="en")]
|
||||
assert got == _md5_order(keys, s)
|
||||
|
||||
|
||||
# ── the rail: its blind slot, and a show's own recommendations ───────────────────────────────────
|
||||
|
||||
|
||||
def test_the_rails_third_slot_is_the_blind_draw(db, make_user, monkeypatch):
|
||||
# The rail draws with random(), so the blend is watched rather than its odds: every leaning draw
|
||||
# goes through it, with the blind order beside the weighted one and the third slot blind.
|
||||
user, *_ = _seed_plex(db, make_user, history=True)
|
||||
calls = []
|
||||
real = draw.blend
|
||||
|
||||
def spy(weighted, blind, offset, n, key):
|
||||
calls.append((offset, n, sorted(map(key, weighted)), sorted(map(key, blind))))
|
||||
return real(weighted, blind, offset, n, key=key)
|
||||
|
||||
monkeypatch.setattr(draw, "blend", spy)
|
||||
got = plex_discover.surprise(db, user, "movie", "en", 3, [], [], LibraryFilters())["items"]
|
||||
assert calls and all(offset == 1 and n == 3 for offset, n, *_ in calls)
|
||||
pool = sorted(["horror_new", "comedy_new", "tipped_new"])
|
||||
assert calls[0][2] == pool and calls[0][3] == pool
|
||||
assert len(got) == 3
|
||||
|
||||
|
||||
def test_a_show_leans_on_the_shows_recommended_not_the_films(db, make_user):
|
||||
lib = PlexLibrary(plex_key="80", title="Shows", kind="show", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
tipped = PlexShow(rating_key="tv1", library_id=lib.id, title="t", guid="tmdb:1", genres=[])
|
||||
db.add(tipped)
|
||||
db.commit()
|
||||
shows = affinity.Affinity(show_recs={"tmdb:1": 1.0})
|
||||
films = affinity.Affinity(movie_recs={"tmdb:1": 1.0})
|
||||
|
||||
def weight(aff):
|
||||
return db.execute(select(aff.weight(PlexShow)).where(PlexShow.id == tipped.id)).scalar_one()
|
||||
|
||||
assert weight(shows) == 1 + affinity.REC_WEIGHT
|
||||
assert weight(films) == 1
|
||||
@@ -292,17 +292,16 @@ def test_a_known_video_is_marked_watched_at_its_takeout_time_and_the_rest_wait_o
|
||||
assert (states["older000000"].status, states["older000000"].position_seconds) == ("watched", 40)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("undo", ["unmark", "clear_state"])
|
||||
def test_a_video_the_user_un_marked_here_is_not_marked_again_by_an_older_view(db, make_user, undo):
|
||||
"""Un-marking (or resetting) a video leaves no state row, and the import inserted a watched one
|
||||
over the user's own choice. A view on YouTube after the un-mark is a new view, and counts."""
|
||||
user = make_user(f"{undo}@example.com")
|
||||
def test_a_video_the_user_un_marked_here_is_not_marked_again_by_an_older_view(db, make_user):
|
||||
"""Un-marking a video leaves no state row, and the import inserted a watched one over the user's
|
||||
own choice. A view on YouTube after the un-mark is a new view, and counts."""
|
||||
user = make_user("unmark@example.com")
|
||||
for vid in ("undone00000", "again000000"):
|
||||
_video(db, vid)
|
||||
db.commit()
|
||||
for vid in ("undone00000", "again000000"):
|
||||
service.mark(db, user, vid, "watched")
|
||||
getattr(service, undo)(db, user, vid)
|
||||
service.unmark(db, user, vid)
|
||||
db.commit()
|
||||
assert _states(db, user) == {}
|
||||
later = datetime.now(timezone.utc) + timedelta(minutes=5)
|
||||
@@ -603,7 +602,6 @@ def test_staging_and_settling_take_the_users_one_lock(db, make_user):
|
||||
),
|
||||
# Between the import's read of the un-marks and its marking, one would be overwritten.
|
||||
"unmark": lambda s, u: service.unmark(s, u, "aaaaaaaaaaa"),
|
||||
"reset": lambda s, u: service.clear_state(s, u, "aaaaaaaaaaa"),
|
||||
}
|
||||
with SessionLocal() as holder:
|
||||
service.lock_takeout(holder, holder.get(User, user.id))
|
||||
|
||||
@@ -579,7 +579,7 @@ def _display_write_sites() -> dict[tuple[str, str], set[str]]:
|
||||
# enqueue_clip slices the source's title to the same width.
|
||||
# files.ingest_spooled passes through `spool_upload`'s value, bounded there.
|
||||
# routes enqueue_clip the `display_name=` keyword passes on the name bound above it.
|
||||
# pipeline._download / _process_clip / worker._finish_from_cache
|
||||
# pipeline._download / _process_clip / worker._finish_from_cache / backfill._retitle
|
||||
# slice a fetched or stored title to the column's width.
|
||||
#
|
||||
# Held as EXACT EQUALITY rather than as a list of offenders. The offender form could only fail
|
||||
@@ -596,6 +596,7 @@ DISPLAY_WRITE_SITES = {
|
||||
("downloads/service.py", "enqueue_clip"): {"display_name or asset.title[:DISPLAY_TEXT_MAX]"},
|
||||
("downloads/pipeline.py", "_download"): {"meta.title[:DISPLAY_TEXT_MAX]"},
|
||||
("downloads/pipeline.py", "_process_clip"): {"named[:DISPLAY_TEXT_MAX]"},
|
||||
("downloads/backfill.py", "_retitle"): {"named[:DISPLAY_TEXT_MAX]"},
|
||||
("worker.py", "_finish_from_cache"): {"asset.title[:DISPLAY_TEXT_MAX]"},
|
||||
("downloads/upload.py", "ingest_upload"): {"display_text(display_name)"},
|
||||
("routes/auth.py", "callback"): {"display_text(userinfo.get('name'))"},
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
"""A YouTube length past a day is unknown, everywhere a length is used (`catalog.models.LENGTH_MAX_S`).
|
||||
|
||||
Measured on prod (2026-11-06): two 24/7 streams' archives report 122,962,914 s and 34,762,459 s, and the
|
||||
owner's Watch later total read "44,073 hr 55 min". Such a video now has no length on any API answer, the
|
||||
totals leave it out (a playlist's counts it as `without_length`), and the duration filter and sort pass it over
|
||||
like a video whose length is not fetched yet. An AST check holds who may still read the raw column.
|
||||
DB lane (the AST check needs none).
|
||||
"""
|
||||
|
||||
import ast
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from app.catalog.models import LENGTH_MAX_S, Channel, Subscription, Video, known_length
|
||||
from app.reader.models import Playlist, PlaylistItem
|
||||
|
||||
YEARS = 122_962_914 # the measured archive
|
||||
HOUR = 3600
|
||||
|
||||
|
||||
def _seed(db, user):
|
||||
db.add(Channel(id="UClen", title="Len"))
|
||||
db.flush()
|
||||
db.add(Subscription(user_id=user.id, channel_id="UClen", priority=0, hidden=False))
|
||||
rows = [
|
||||
("short", 600, "none"),
|
||||
("long", 2 * HOUR, "none"),
|
||||
("day", LENGTH_MAX_S, "was_live"), # exactly a day still is a length
|
||||
("archive", YEARS, "was_live"),
|
||||
("onair", None, "live"),
|
||||
("unfetched", None, "none"), # not live, but left out of the sum: counted all the same
|
||||
]
|
||||
for i, (vid, seconds, live) in enumerate(rows):
|
||||
db.add(
|
||||
Video(
|
||||
id=vid,
|
||||
channel_id="UClen",
|
||||
published_at=datetime(2024, 1, i + 1, tzinfo=timezone.utc),
|
||||
title=vid,
|
||||
duration_seconds=seconds,
|
||||
live_status=live,
|
||||
is_short=False,
|
||||
via_search=False,
|
||||
via_explore=False,
|
||||
)
|
||||
)
|
||||
pl = Playlist(user_id=user.id, name="Later")
|
||||
db.add(pl)
|
||||
db.flush()
|
||||
for i, (vid, _s, _l) in enumerate(rows):
|
||||
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=i))
|
||||
db.commit()
|
||||
return pl
|
||||
|
||||
|
||||
KNOWN_SUM = 600 + 2 * HOUR + LENGTH_MAX_S
|
||||
|
||||
|
||||
def test_the_rule():
|
||||
assert known_length(None) is None
|
||||
assert known_length(LENGTH_MAX_S) == LENGTH_MAX_S
|
||||
assert known_length(LENGTH_MAX_S + 1) is None
|
||||
assert known_length(YEARS) is None
|
||||
|
||||
|
||||
def test_the_sql_length_carries_its_limit_as_a_literal():
|
||||
# The duration sort indexes (migration 0117) are built on this expression with 86400 in it. A
|
||||
# bound parameter matches no index once the driver reuses a prepared, generic plan, and the
|
||||
# sort-index test cannot see that: it compiles with literal binds.
|
||||
compiled = Video.length_seconds.expression.compile(dialect=postgresql.dialect())
|
||||
assert compiled.params == {}
|
||||
assert f"> {LENGTH_MAX_S}" in str(compiled)
|
||||
|
||||
|
||||
def test_a_years_long_archive_has_no_length_in_the_feed_or_a_playlist(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user)
|
||||
client = as_user(user)
|
||||
|
||||
feed = client.get("/api/feed?show=all&include_live=true&limit=50")
|
||||
assert feed.status_code == 200
|
||||
by_id = {v["id"]: v["duration_seconds"] for v in feed.json()["items"]}
|
||||
assert by_id["archive"] is None
|
||||
assert by_id["day"] == LENGTH_MAX_S
|
||||
|
||||
detail = client.get(f"/api/playlists/{pl.id}")
|
||||
assert detail.status_code == 200
|
||||
items = {v["id"]: v["duration_seconds"] for v in detail.json()["items"]}
|
||||
assert items["archive"] is None
|
||||
|
||||
one = client.get("/api/videos/archive")
|
||||
assert one.status_code == 200
|
||||
assert one.json()["duration_seconds"] is None
|
||||
|
||||
|
||||
def test_an_off_catalog_archive_has_no_length_either(db, make_user, as_user, monkeypatch):
|
||||
# A video not in the catalog is looked up on YouTube (videos.list); its length is judged the same.
|
||||
from app.routes import feed as feed_routes
|
||||
|
||||
class Yt:
|
||||
def __init__(self, *_):
|
||||
pass
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
def get_videos(self, ids):
|
||||
hours = {"offyears": YEARS // HOUR, "offshort": 1}[ids[0]]
|
||||
return [
|
||||
{"snippet": {}, "statistics": {}, "contentDetails": {"duration": f"PT{hours}H"}}
|
||||
]
|
||||
|
||||
monkeypatch.setattr(feed_routes, "YouTubeClient", Yt)
|
||||
client = as_user(make_user())
|
||||
assert client.get("/api/videos/offyears").json()["duration_seconds"] is None
|
||||
assert client.get("/api/videos/offshort").json()["duration_seconds"] == HOUR
|
||||
|
||||
|
||||
def test_the_totals_leave_it_out_and_count_what_they_left_out(db, make_user, as_user):
|
||||
user = make_user()
|
||||
pl = _seed(db, user)
|
||||
client = as_user(user)
|
||||
|
||||
listed = client.get("/api/playlists")
|
||||
assert listed.status_code == 200
|
||||
(summary,) = [p for p in listed.json() if p["id"] == pl.id]
|
||||
assert summary["total_duration_seconds"] == KNOWN_SUM
|
||||
assert summary["without_length"] == 3 # the archive, the one on air, the unfetched one
|
||||
|
||||
renamed = client.patch(f"/api/playlists/{pl.id}", json={"name": "Later 2"})
|
||||
assert renamed.status_code == 200
|
||||
assert renamed.json()["total_duration_seconds"] == KNOWN_SUM
|
||||
assert renamed.json()["without_length"] == 3
|
||||
|
||||
channels = client.get("/api/channels")
|
||||
assert channels.status_code == 200
|
||||
(ch,) = channels.json()
|
||||
assert ch["total_duration_seconds"] == KNOWN_SUM
|
||||
# The channel table shows the sum alone (the count did not fit its column), so it is not sent.
|
||||
assert "without_length" not in ch and "live_without_length" not in ch
|
||||
|
||||
|
||||
def test_the_duration_filter_and_sort_pass_it_over(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed(db, user)
|
||||
client = as_user(user)
|
||||
base = "/api/feed?show=all&include_live=true&limit=50"
|
||||
|
||||
longer = client.get(f"{base}&min_duration={HOUR}")
|
||||
assert longer.status_code == 200
|
||||
assert {v["id"] for v in longer.json()["items"]} == {"long", "day"}
|
||||
|
||||
longest = client.get(f"{base}&sort=duration_desc")
|
||||
assert longest.status_code == 200
|
||||
order = [v["id"] for v in longest.json()["items"]]
|
||||
assert order[:3] == ["day", "long", "short"]
|
||||
assert set(order[3:]) == {"archive", "onair", "unfetched"}
|
||||
|
||||
|
||||
# Who may read `duration_seconds` itself, by (file, function): what each one asks is not "how long".
|
||||
RAW_READERS = {
|
||||
# The column is defined and turned into a length here.
|
||||
("catalog/models.py", "length_seconds"),
|
||||
("catalog/models.py", "_length_seconds_expression"),
|
||||
# Ingest writes it; the live refresh asks whether YouTube has answered for an archive yet; the
|
||||
# Shorts classification needs only "longer than a Short", which a years-long value proves.
|
||||
("sync/videos.py", "apply_video_details"),
|
||||
("sync/videos.py", "refresh_live"),
|
||||
("sync/videos.py", "run_shorts_classification"),
|
||||
("catalog/search.py", "classify_shorts"),
|
||||
("catalog/queries.py", "work_queue"),
|
||||
# Not a video's column: the player's own report, and the watch history's looked-up lengths, which
|
||||
# count a watched video as more than a glance (a watched archive is that too).
|
||||
("routes/_common.py", "kept_position"),
|
||||
("routes/plex/playback.py", "item_progress"),
|
||||
("reader/queries.py", "_counted_channels"),
|
||||
("reader/queries.py", "watched_channels"),
|
||||
("reader/queries.py", "watched_channel_videos"),
|
||||
}
|
||||
|
||||
|
||||
def _raw_reads() -> set[tuple[str, str]]:
|
||||
root = Path(__file__).resolve().parents[1] / "app"
|
||||
found: set[tuple[str, str]] = set()
|
||||
for path in root.rglob("*.py"):
|
||||
if "alembic" in path.parts:
|
||||
continue
|
||||
rel = path.relative_to(root).as_posix()
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
in_a_function: set[int] = set()
|
||||
for fn in ast.walk(tree):
|
||||
if not isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
continue
|
||||
for node in ast.walk(fn):
|
||||
if _reads_raw(node):
|
||||
found.add((rel, fn.name))
|
||||
in_a_function.add(id(node))
|
||||
# A read outside every def (module or class level, a lambda there) is named `<module>`.
|
||||
if any(_reads_raw(n) and id(n) not in in_a_function for n in ast.walk(tree)):
|
||||
found.add((rel, "<module>"))
|
||||
return found
|
||||
|
||||
|
||||
def _reads_raw(node: ast.AST) -> bool:
|
||||
return isinstance(node, ast.Attribute) and node.attr == "duration_seconds"
|
||||
|
||||
|
||||
def test_only_the_listed_readers_take_the_raw_column():
|
||||
found = _raw_reads()
|
||||
stray = sorted(found - RAW_READERS)
|
||||
assert not stray, f"read `length_seconds` (a length) instead of the raw column: {stray}"
|
||||
# A listed reader that stopped reading it leaves the list.
|
||||
assert not RAW_READERS - found, sorted(RAW_READERS - found)
|
||||
@@ -216,7 +216,7 @@ def test_a_line_with_no_url_is_left_alone(sink):
|
||||
|
||||
|
||||
def test_the_download_options_carry_the_logger():
|
||||
opts = formats.build_ydl_opts({}, "%(id)s.%(ext)s", lambda d: None)
|
||||
opts = formats.build_ydl_opts({}, "%(id)s.%(ext)s", lambda d: None, egress=False)
|
||||
|
||||
assert opts["logger"] is formats._YDL_LOG
|
||||
|
||||
@@ -227,7 +227,7 @@ def test_a_source_adapter_cannot_silence_it():
|
||||
adapter must not be able to take away the one account of what went wrong that a failed job
|
||||
leaves behind."""
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "%(id)s.%(ext)s", lambda d: None, extra={"logger": None, "quiet": False}
|
||||
{}, "%(id)s.%(ext)s", lambda d: None, extra={"logger": None, "quiet": False}, egress=False
|
||||
)
|
||||
|
||||
assert opts["logger"] is formats._YDL_LOG
|
||||
|
||||
+199
-134
@@ -256,7 +256,7 @@ Also in force:
|
||||
- **A glyph means one thing.** Play (▷) starts playback, or resumes a paused transfer; a retry is the
|
||||
refresh arrow (`RefreshCw`), and so is doing a download again; a reset or revert is the
|
||||
counter-clockwise arrow (`RotateCcw`), and so is playing from the beginning, on the card and in its
|
||||
menu alike; clearing a video's progress and status is the eraser (`Eraser`). A failed download's Retry
|
||||
menu alike; clearing a video's watch progress is the eraser (`Eraser`). A failed download's Retry
|
||||
had worn the Play glyph, which promised a video. The skip arrows (⏮ ⏭) are Previous /
|
||||
Next; the auto-advance mode beside them wears list glyphs (`ListEnd` next, `ListStart` previous,
|
||||
`Shuffle` random; off is `ListEnd`, not lit) — a mode in the skip arrows put two identical ⏭ on the
|
||||
@@ -301,7 +301,8 @@ Also in force:
|
||||
`fade-below` utility (`fade-below-panel` / `fade-above-panel` on a panel); the page band and a
|
||||
scrolling panel under its header fade by the scroller's own edge mask (`useScrollFade`), because what
|
||||
they sit on is the backdrop, not a colour. The chip rows fold their overflow into "+N" rather than
|
||||
scroll, so no row has clipped ends; the carousels fade theirs. The dialog edges come with C4's pinned
|
||||
scroll, so no row has clipped ends, and a filter row puts its picked chips first so the fold never
|
||||
holds the only sign that a filter is on; the carousels fade theirs. The dialog edges come with C4's pinned
|
||||
footer (S3f).
|
||||
- **Viewport height:** `dvh`, never `vh` / `h-screen`: `100vh` is the viewport with a mobile browser's
|
||||
toolbar retracted, so an `h-screen` shell runs its last rows under the toolbar (contract-tested).
|
||||
@@ -536,6 +537,26 @@ of a full-width panel and nothing on the right.
|
||||
shows them as four buttons with the item's place beside the selected view; a side panel in edit mode
|
||||
gives each group an Up / Down pair (`ui/OrderButtons`, `compact`). A move that would go nowhere is
|
||||
shown off, not hidden. A grip also picks up from the keyboard (space, the arrows, space).
|
||||
- **Selecting rows:** a list whose rows take bulk actions shows a box first in every row, always, and one
|
||||
over the column for the rows shown. Shift+click selects a range, `x` toggles the row under the keyboard,
|
||||
Esc clears. A selected row wears the accent tint as well as its tick. An action covers the selected rows
|
||||
shown; a row a filter hides stays selected and comes back ticked; another list starts over. The row's own
|
||||
menu still acts on that row alone. While anything is selected the toolbar's second row is the action band:
|
||||
the count (and how many a filter hides), the last move's target as one click, Move to, Remove, Clear
|
||||
selection. An action clears the selection, undoes by its toast and, when it fails, ticks its rows again.
|
||||
An Undo brings back what is still there to bring: a video the move put in the target and the target no
|
||||
longer holds stays where it went, and a warning says how many came back. One the target held already only
|
||||
left the source, so it returns there.
|
||||
Delete takes the selection when there is one, else the row under the keyboard, with no confirm. With every
|
||||
tick hidden by a filter it removes nothing and says so. An Esc during a drag puts the drag down, and only that.
|
||||
Below `xl` the band's buttons are glyphs named in their tooltips, and a filter's hidden count is "+N" with
|
||||
its sentence in the tooltip. The row the band replaces keeps its height, invisible, so a tick moves no row.
|
||||
On a phone the count is its number (the words read out, and in the tooltip) and the undo pair stays out.
|
||||
A row's grip also carries it onto a list in the rail, under a sort or a filter too (the grip then says
|
||||
so, and no key lifts it); a selected row carries every selected one, named on the drag image below the
|
||||
pointer. The target row lights up (`ring-accent`); the open list is none; an unpinned rail peeks by the
|
||||
drag's own hover and scrolls at its edges. Dropped inside the list, one row takes its new place and many
|
||||
stay. A key never leaves the list and moves its own row: the band and the row's menu are the other way.
|
||||
- **Motion:** 150–300ms micro-interactions, transform/opacity only; `prefers-reduced-motion` is respected.
|
||||
[WCAG 2.3.3]
|
||||
|
||||
@@ -547,6 +568,9 @@ of a full-width panel and nothing on the right.
|
||||
the first tap however long the video played).
|
||||
- **The bars stay up while the pointer rests on them** (F22, owner); leaving starts a fresh 3 s. YouTube and
|
||||
Plex Web do the same: before, the row faded out from under a mouse parked on Next.
|
||||
- **Text over the picture sits on a dark panel** (2026-11-07): loading, stalls, errors and the notices are
|
||||
readable on any frame. A stall that lasts 5 s on a link measured slower than the stream says "Slow
|
||||
connection" with both rates.
|
||||
- **Volume is the native `Slider`** (A6); the hand-drawn bar had no hover, focus, keyboard or screen-reader
|
||||
state. A pointer drag gives the focus back, so Space and the arrows keep driving the player.
|
||||
- **The seek bar:** the track stays thin inside a 2.75rem (44px) hit area with `touch-action: none`, so a finger
|
||||
@@ -622,6 +646,19 @@ of a full-width panel and nothing on the right.
|
||||
press outside the frame is still the video's. A cut is a toast with a Downloads link and the usual
|
||||
"Ready to watch" notification. Limits are the server's: 1 s to 10 min (`lib/clipRange`, held to
|
||||
`downloads/clip.py` by `clipLimits.contract.test.ts`; the "too long" message names the server's cap).
|
||||
**GIF (owner's picks D1 A, D2 A, D3):** a GIF is made only here and in the Downloads editor, never from the
|
||||
download dialog. The panel's Format · Video / GIF toggle sits above the name row; with GIF it shows the size
|
||||
(the long side: 320 / 480 / 640 px), the expected size ("about 4–10 MB"; one number when both ends read the
|
||||
same), a note over 15 MB (a chat's link preview will not move) or 30 MB (it may be refused), and over 30 s
|
||||
the reason, with Make GIF unavailable. A refused length is shown rounded up, never as one the cap allows. The
|
||||
cap is judged in whole milliseconds between the ends as sent, the server's own rule (the limits in `lib/gif`
|
||||
are held to `downloads/gif.py` by `gif.contract.test.ts`; the rule itself is written on each side and
|
||||
tested there against the same boundary cases). The editor has its own Format row (a video only): with GIF
|
||||
each kept segment is a GIF of its own, Join is unavailable with its reason in a tooltip, the precise / fast
|
||||
choice gives way to one line (a GIF is always re-encoded), and the notes name each segment over a limit,
|
||||
in one live region. A shared GIF's page shows the picture itself (no player), its name, kind and size, its
|
||||
Download, and a tip: the downloaded file moves in any chat. A picture that fails says so only once its
|
||||
retry failed too.
|
||||
|
||||
## Dialogs — _Decided_ (C4, built in S3f)
|
||||
|
||||
@@ -697,135 +734,163 @@ of a full-width panel and nothing on the right.
|
||||
|
||||
## Decision log
|
||||
|
||||
| Date | Decision | Chosen | Over | Why |
|
||||
| ---------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
|
||||
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
|
||||
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
|
||||
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
|
||||
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
|
||||
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
|
||||
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
|
||||
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
|
||||
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
|
||||
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
|
||||
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
|
||||
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
|
||||
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
|
||||
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
|
||||
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
|
||||
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
|
||||
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
|
||||
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
|
||||
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
|
||||
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
|
||||
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
|
||||
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
|
||||
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
|
||||
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
|
||||
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
|
||||
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
|
||||
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
|
||||
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
|
||||
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
|
||||
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
|
||||
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
|
||||
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
|
||||
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
|
||||
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
|
||||
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
|
||||
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
|
||||
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
|
||||
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
|
||||
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
|
||||
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
|
||||
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
|
||||
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
|
||||
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
|
||||
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
|
||||
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
|
||||
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
|
||||
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
|
||||
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
|
||||
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
|
||||
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
|
||||
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
|
||||
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
|
||||
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
|
||||
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
|
||||
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
|
||||
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
|
||||
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
|
||||
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
|
||||
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
|
||||
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
|
||||
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
|
||||
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
|
||||
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
|
||||
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
|
||||
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
|
||||
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
|
||||
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
|
||||
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
|
||||
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
|
||||
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
|
||||
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
|
||||
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
|
||||
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
|
||||
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
|
||||
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
|
||||
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
|
||||
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
|
||||
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
|
||||
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
|
||||
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
|
||||
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
|
||||
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
|
||||
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
|
||||
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
|
||||
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
|
||||
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
|
||||
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
|
||||
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
|
||||
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
|
||||
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
|
||||
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
|
||||
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
|
||||
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
|
||||
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
|
||||
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
|
||||
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
|
||||
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
|
||||
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
|
||||
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
|
||||
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
|
||||
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
|
||||
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
|
||||
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
|
||||
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
|
||||
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
|
||||
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
|
||||
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
|
||||
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
|
||||
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
|
||||
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
|
||||
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
|
||||
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
|
||||
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
|
||||
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
|
||||
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
|
||||
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
|
||||
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
|
||||
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
|
||||
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
|
||||
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
|
||||
| 2026-10-17 | Copy-only changes (owner ★) | Step 4 of the Conformance check alone (not 1–3 or 5): every state the changed text renders in, seen in both languages | The whole five-step check; no screenshot at all (DOM test + key parity) | Steps 1–3 and 5 judge controls, patterns and composition, which a text change does not touch; the one risk left is a longer Hungarian string wrapping or clipping, and only a screenshot shows that |
|
||||
| 2026-10-17 | Text on glass (owner ★) | Light `--muted` 15% toward `--fg` in every scheme (≥ 6:1 on the bare page); the item count, the badges and the current Files row on opaque tints mixed into the surface | Only the two failing components; the item count's number in `--fg` | Pixel-measured over every scheme × mode × backdrop, the count fell to 3.8:1 (dark too: thumbnails scroll under the toolbar's glass) and Channels' tabs to 4.1:1; an `--fg` count outshouts the filters beside it |
|
||||
| 2026-10-18 | Scheduler run history (owner D4, E1, E2 ★) | A row's last three runs as dots, newest first (interrupted: a hollow warn ring, set apart from skipped and failed), and a history button that opens the job's runs on the Activity page; a user's ad-hoc runs in their own block under the jobs, with no interval and no run button | An inline expandable history per row; a dark-grey interrupted dot; ad-hoc runs only on the Activity page | Three dots answer "did it run, did it work" at a glance; the whole history needs filters, which the Activity page has |
|
||||
| 2026-10-18 | Activity page (owner D1, D2, G1–G3 ★) | One page, "Napló" / "Activity": four tabs (Runs, Admin actions, Server log, Quota) under one filter bar (text, a period of 1 h · 24 h · 7 d · 30 d · 90 d defaulting to 7 days, an account), the view in the URL; runs and admin actions as dense rows that open in place as a log line does, filtered and paged on the server; a run's or a request's log lines one pivot away, as a history entry | One merged timeline; a page per source; a default period per tab; card rows; the audit log's client-side table | Each source keeps its own table and retention, read in one place; one bar keeps what you narrowed across tabs; about 1.8k runs a day need a log's density and a server-side filter |
|
||||
| 2026-10-24 | Related videos (owner U1–U3 ★, D1 A, D3) | On demand: a tab beside the description (wide), a drawer (stacked); the narrow thumbnailed row, its menu alone, a followed check; plays in place and the list stays the opened video's; an on-topic / Hungarian-trending switch kept on the account | Always shown; the end-of-video grid alone; a dialog; a big card or a text row; a list that follows the playing video | The modal is full on a small screen; a picture is what a choice among strangers rests on; a list that stays put is tried one by one, and spends no more of the day's cap |
|
||||
| 2026-10-25 | Related: one played; off (owner D1 a, D2 a) | Played from the list, it is the player's item: its title, controls and progress, marked watched at its end, the queue left where it was; a daily limit of 0 takes the tab and the button away | A linked video's in-place play (no controls, nothing kept); a tab that only shows a refusal | What is watched should count as watched, wherever it was started from; a control that can only be refused is not offered |
|
||||
| 2026-10-25 | Player panel glyphs (owner 1A, 2A, 3) | In the side panel the queue row is glyph-only on one line, and Mark watched is a glyph toggle among the actions (✓ / ✓✓); the docked bar keeps its labels; the related switch at the default size, "Trending" | Labelled buttons wrapping onto three lines; glyphs on the docked bar too; Mark watched on its own line | The panel is narrow and the list controls are the same few every time; each glyph keeps its full name and shortcut in its tooltip; the docked bar has the width for words |
|
||||
| 2026-10-25 | Auto-advance glyphs (owner E A, H a) | Auto-advance wears list glyphs (`ListEnd` next and off, `ListStart` previous, `Shuffle` random), in the panel and the docked bar; the auto-advance and loop cycles are `lit`, not `pressed` | The skip arrows for the mode; an X'd list for off; `aria-pressed` on a cycle | Beside Previous / Next the skip arrows read as a second Next; a cycle of three or four modes is no two-state switch, and its name already says which |
|
||||
| 2026-10-25 | Restart and Reset glyphs (owner L A) | A menu's Play from the beginning wears `RotateCcw`, as the card's own Restart does; Reset (clear progress and status) wears `Eraser`; Redownload wears `RefreshCw`, the retry arrow | The skip arrow ⏮ on the menu's Restart; `RotateCcw` on both Restart and Reset; `TimerReset` for Restart | ⏮ is Previous beside the player; one action wore two glyphs, and two actions in one menu would have worn one |
|
||||
| 2026-10-28 | Notification order (owner A-D1 a, A-D2 a, ★ a) | What asks for the reader first, from either source ("Needs you · n"), then one stream newest first; read rows older than a week fold under "Earlier (n)", closed, its label in the section-heading style with a chevron | One stream with no band; this browser's activity above the server's (the two fixed sections); a server-side retention that deletes old rows | An admin's access requests sat under up to a hundred server rows; week-old read notices crowded what is new; folding keeps the rows (nothing is deleted) and an unread row never folds, whatever its age |
|
||||
| Date | Decision | Chosen | Over | Why |
|
||||
| ---------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
|
||||
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
|
||||
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
|
||||
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
|
||||
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
|
||||
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
|
||||
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
|
||||
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
|
||||
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
|
||||
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
|
||||
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
|
||||
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
|
||||
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
|
||||
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
|
||||
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
|
||||
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
|
||||
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
|
||||
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
|
||||
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
|
||||
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
|
||||
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
|
||||
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
|
||||
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
|
||||
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
|
||||
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
|
||||
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
|
||||
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
|
||||
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
|
||||
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
|
||||
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
|
||||
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
|
||||
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
|
||||
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
|
||||
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
|
||||
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
|
||||
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
|
||||
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
|
||||
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
|
||||
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
|
||||
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
|
||||
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
|
||||
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
|
||||
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
|
||||
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
|
||||
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
|
||||
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
|
||||
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
|
||||
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
|
||||
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
|
||||
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
|
||||
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
|
||||
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
|
||||
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
|
||||
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
|
||||
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
|
||||
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
|
||||
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
|
||||
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
|
||||
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
|
||||
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
|
||||
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
|
||||
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
|
||||
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
|
||||
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
|
||||
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
|
||||
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
|
||||
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
|
||||
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
|
||||
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
|
||||
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
|
||||
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
|
||||
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
|
||||
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
|
||||
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
|
||||
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
|
||||
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
|
||||
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
|
||||
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
|
||||
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
|
||||
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
|
||||
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
|
||||
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
|
||||
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
|
||||
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
|
||||
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
|
||||
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
|
||||
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
|
||||
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
|
||||
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
|
||||
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
|
||||
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
|
||||
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
|
||||
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
|
||||
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
|
||||
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
|
||||
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
|
||||
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
|
||||
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
|
||||
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
|
||||
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
|
||||
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
|
||||
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
|
||||
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
|
||||
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
|
||||
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
|
||||
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
|
||||
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
|
||||
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
|
||||
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
|
||||
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
|
||||
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
|
||||
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
|
||||
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
|
||||
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
|
||||
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
|
||||
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
|
||||
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
|
||||
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
|
||||
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
|
||||
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
|
||||
| 2026-10-17 | Copy-only changes (owner ★) | Step 4 of the Conformance check alone (not 1–3 or 5): every state the changed text renders in, seen in both languages | The whole five-step check; no screenshot at all (DOM test + key parity) | Steps 1–3 and 5 judge controls, patterns and composition, which a text change does not touch; the one risk left is a longer Hungarian string wrapping or clipping, and only a screenshot shows that |
|
||||
| 2026-10-17 | Text on glass (owner ★) | Light `--muted` 15% toward `--fg` in every scheme (≥ 6:1 on the bare page); the item count, the badges and the current Files row on opaque tints mixed into the surface | Only the two failing components; the item count's number in `--fg` | Pixel-measured over every scheme × mode × backdrop, the count fell to 3.8:1 (dark too: thumbnails scroll under the toolbar's glass) and Channels' tabs to 4.1:1; an `--fg` count outshouts the filters beside it |
|
||||
| 2026-10-18 | Scheduler run history (owner D4, E1, E2 ★) | A row's last three runs as dots, newest first (interrupted: a hollow warn ring, set apart from skipped and failed), and a history button that opens the job's runs on the Activity page; a user's ad-hoc runs in their own block under the jobs, with no interval and no run button | An inline expandable history per row; a dark-grey interrupted dot; ad-hoc runs only on the Activity page | Three dots answer "did it run, did it work" at a glance; the whole history needs filters, which the Activity page has |
|
||||
| 2026-10-18 | Activity page (owner D1, D2, G1–G3 ★) | One page, "Napló" / "Activity": four tabs (Runs, Admin actions, Server log, Quota) under one filter bar (text, a period of 1 h · 24 h · 7 d · 30 d · 90 d defaulting to 7 days, an account), the view in the URL; runs and admin actions as dense rows that open in place as a log line does, filtered and paged on the server; a run's or a request's log lines one pivot away, as a history entry | One merged timeline; a page per source; a default period per tab; card rows; the audit log's client-side table | Each source keeps its own table and retention, read in one place; one bar keeps what you narrowed across tabs; about 1.8k runs a day need a log's density and a server-side filter |
|
||||
| 2026-10-24 | Related videos (owner U1–U3 ★, D1 A, D3) | On demand: a tab beside the description (wide), a drawer (stacked); the narrow thumbnailed row, its menu alone, a followed check; plays in place and the list stays the opened video's; an on-topic / Hungarian-trending switch kept on the account | Always shown; the end-of-video grid alone; a dialog; a big card or a text row; a list that follows the playing video | The modal is full on a small screen; a picture is what a choice among strangers rests on; a list that stays put is tried one by one, and spends no more of the day's cap |
|
||||
| 2026-10-25 | Related: one played; off (owner D1 a, D2 a) | Played from the list, it is the player's item: its title, controls and progress, marked watched at its end, the queue left where it was; a daily limit of 0 takes the tab and the button away | A linked video's in-place play (no controls, nothing kept); a tab that only shows a refusal | What is watched should count as watched, wherever it was started from; a control that can only be refused is not offered |
|
||||
| 2026-10-25 | Player panel glyphs (owner 1A, 2A, 3) | In the side panel the queue row is glyph-only on one line, and Mark watched is a glyph toggle among the actions (✓ / ✓✓); the docked bar keeps its labels; the related switch at the default size, "Trending" | Labelled buttons wrapping onto three lines; glyphs on the docked bar too; Mark watched on its own line | The panel is narrow and the list controls are the same few every time; each glyph keeps its full name and shortcut in its tooltip; the docked bar has the width for words |
|
||||
| 2026-10-25 | Auto-advance glyphs (owner E A, H a) | Auto-advance wears list glyphs (`ListEnd` next and off, `ListStart` previous, `Shuffle` random), in the panel and the docked bar; the auto-advance and loop cycles are `lit`, not `pressed` | The skip arrows for the mode; an X'd list for off; `aria-pressed` on a cycle | Beside Previous / Next the skip arrows read as a second Next; a cycle of three or four modes is no two-state switch, and its name already says which |
|
||||
| 2026-10-25 | Restart and Reset glyphs (owner L A) | A menu's Play from the beginning wears `RotateCcw`, as the card's own Restart does; Reset (clear progress and status) wears `Eraser`; Redownload wears `RefreshCw`, the retry arrow | The skip arrow ⏮ on the menu's Restart; `RotateCcw` on both Restart and Reset; `TimerReset` for Restart | ⏮ is Previous beside the player; one action wore two glyphs, and two actions in one menu would have worn one |
|
||||
| 2026-10-28 | Notification order (owner A-D1 a, A-D2 a, ★ a) | What asks for the reader first, from either source ("Needs you · n"), then one stream newest first; read rows older than a week fold under "Earlier (n)", closed, its label in the section-heading style with a chevron | One stream with no band; this browser's activity above the server's (the two fixed sections); a server-side retention that deletes old rows | An admin's access requests sat under up to a hundred server rows; week-old read notices crowded what is new; folding keeps the rows (nothing is deleted) and an unread row never folds, whatever its age |
|
||||
| 2026-11-01 | Docked bar glyphs (owner D8 a) | The maximised view's docked bar is glyph-only too (tooltips carry the names and keys): transport, auto-advance, loop, Mark watched (✓ / ✓✓); its controls scroll on a narrow window while Fullscreen and Exit stay pinned at its right end | Labels from `xl` up | On a tablet the labelled bar ran past the right edge and took Exit with it; the side panel had already lost its labels |
|
||||
| 2026-11-01 | Toolbar group glyphs (owner D9 a, D10 a, D11 a) | The feed's, channel page's and Plex Browse's filter pills name their group by a glyph (`Eye` watch state, `Shapes` content types, `ArrowDownUp` sort), the word in the tooltip and the accessible name; Sort + View end the toolbar's row, the chip row shows only with a chip; a channel page has no count pill | The value alone; Sort + View on a row of their own | "Watch state:" and "Sort:" cost a tablet row its room; the value says enough once the glyph names the group; the channel header already counts the videos |
|
||||
| 2026-11-01 | Channel header on a tablet (owner E2) | Below `xl` the action row drops its two words: "Prio" (the stepper's tooltip names it) and "Subscribed" (a ✓ glyph; its name in the aria-label, Unsubscribe in the tooltip); the tags keep the row's rest, folding into "+N" | The tag lane on a full line of its own below ~12rem; the tags under the channel name | On an iPad (820 px, 44 px targets) the lane got 29 px, a sliver; now 130 px, and every tag shows at 1180 |
|
||||
| 2026-11-01 | Clear watch progress (owner E a) | A video menu's Reset reads "Clear watch progress" in English (HU "Megtekintés visszaállítása" stays); "Reset view" is only a page's view reset (sort, filters, grouping) | "Reset view" for both; the playlist page's view reset renamed instead | On a playlist page the two stood one right-click apart under one name, and one of them erases the video's progress with no undo |
|
||||
| 2026-11-01 | Clear watch progress clears only that (owner A) | The row shows only on a video with a resume point and clears that alone: a hidden video stays hidden, a watched one watched. Un-hiding and un-watching have rows of their own | The old Reset, which also un-hid and un-watched | A hidden video stays hidden: what the row's name says is all it does |
|
||||
| 2026-11-01 | Selecting a list's rows (owner D2 a) | An always visible box first in the row and one over the column for the rows shown; Shift+click a range, `x` the row under the keyboard, Esc clears; the selected row wears `accent/12` with its tick. Rows a filter hides stay selected, an action covers those shown; Watch later offers no selecting. | A "Select" button that shows the boxes on demand | Found at once and works on touch, with no mode to leave; costs about 2 rem a row |
|
||||
| 2026-11-01 | Selection band (owner D1 a, D3 a, D4 a) | The toolbar's second row while rows are selected, no floating bar: move (the last target as one click) and remove, no copy; both undo back to the places, a video the target held before stays there; a single row's move undoes too. | A floating bar at the bottom; copying to a list too; no undo on a move | It covers no row, and the player and the notices live at the bottom; twenty videos sent to the wrong list need a way back to their places |
|
||||
| 2026-11-01 | Grip on a phone (owner A) | Below a 24rem table a playlist row has no grip: the title keeps its width, and the row menu keeps the moves. | The grip and its drag kept, the title at 44 px | The title gets its width back, and the rail is shut on a phone, so a drag had nowhere to go |
|
||||
| 2026-11-01 | Selection band on a tablet (owner A) | Below `xl` the band is glyphs on one row, the names in tooltip and aria-label, as on the channel header; the replaced row stays in the same grid cell, invisible, so the first tick moves no row under the second tap. | The buttons in words, wrapping to a second row | The first tick pushed the list from under the second tap |
|
||||
| 2026-11-01 | Playlist heading on a narrow window (owner A) | The list's name had no width left: below `xl` Unsynced is its dot (the word read out, the hint in the tooltip); on a phone the cover is gone and Play all is its glyph, named in the tooltip. | Every word kept, the list's name at no width | A heading without its name says nothing |
|
||||
| 2026-11-02 | Dragging onto a list (owner D5 a, D6 a) | The grip carries a row, or every selected row with a selected one, onto a list in the rail; the toast undoes. Many dropped inside the list stay; a coarse pointer's drawer never peeks, so its rail is a target only when pinned. | Only the grabbed row going along; the grip off under a view | One row moving out of three lit ones misleads, and a filter is how the rows to move are found |
|
||||
| 2026-11-02 | Delete with every tick hidden (owner a) | It removes nothing and a warning says the filter hides them | Taking the row under the keyboard | While the band shows a selection, Delete means that selection: a row no one ticked must not go |
|
||||
| 2026-11-02 | Undo of a video gone further (owner a) | A video the move put in the target comes back only if the target still holds it, and a warning says how many | Everything back, whatever it did since | A move skips a video that left the source; the same rule back keeps a video out of lists the reader never put it in |
|
||||
| 2026-11-04 | Tag chip counts (owner D5a) | A Channels tag chip shows how many followed channels it holds: a muted number after the name, no badge | The number in brackets after the name | The owner's pick; the number is a detail of the name, so it takes the name's own text, muted, rather than a `CountBadge` |
|
||||
| 2026-11-05 | Picked tag chips lead the row (owner c) | On the Channels tag row the picked chips come first, in the tags' order, then the rest | Keep the order and fold the unpicked first; mark the "+N" when it holds a pick | The row folds from its end, so a picked tag late in the list sat only inside "+N" with nothing on the row saying a filter was on |
|
||||
| 2026-11-05 | A toggled tag chip keeps the focus (owner a) | A chip that moves when toggled takes the keyboard focus with it: to its new place, or to "+N" when it went into the closed fold | Keep a picked chip in the "+N" popover until it closes | The row always shows the filters as they are; the focus, not the order, was what got lost |
|
||||
| 2026-11-06 | Tag chip count after a dot (owner a) | The count keeps the chip's own colour, set off from the name by a "·" | The number dimmed (it fell under the 4.5:1 text floor, resting and pressed); the number in a tinted inner badge | Contrast holds in every scheme without a new token, and the dot still reads the number as a detail of the name |
|
||||
| 2026-11-06 | A pick in the open fold closes it (owner a) | Toggling a chip inside the "+N" popover closes the popover; the focus goes with the chip | Keep the popover open and focus the chip now in the picked one's place | The focus never sits outside a popover left open, and a picked chip leads the row anyway |
|
||||
| 2026-11-06 | An expired share link offers no Copy (owner a) | An expired row in the share dialog has a warn badge "expired <date>" and its address struck through; Open and Revoke stay, Copy goes | The warn badge alone, Copy kept | An expired link opens nothing, so it is no link to hand out; Open still lets the owner check it |
|
||||
| 2026-11-06 | The shared page's unlock button is a noun in Hungarian (owner a) | "Megtekintés" for a video, "Megnyitás" for a file or folder | Both verbs, "Megnézem" / "Megnyitom" | The app's Hungarian buttons are nouns; one button should not change grammatical form by link kind |
|
||||
| 2026-11-07 | A slow link says so, with both rates (owner b1) | After 5 s of stalling on a link measured slower than the stream: "Slow connection" and "This file needs about X, about Y is arriving" under the spinner; no number measured, no claim | The line without numbers; a pill at the bottom | The viewer can tell their connection from a broken player, and by how much it falls short |
|
||||
| 2026-11-07 | Text over the picture sits on a dark panel (owner a) | The player's loading, stall and error text, the subtitle notice and the track OSD sit on one dark panel (`OVER_VIDEO`, black 75 %, a white 15 % edge) | A shadow behind the letters; bare text as before | Bare white or red text was unreadable over a bright frame (seen live); the panel is the style the notices already had |
|
||||
| 2026-11-07 | The panel over the picture is darker, its error the media red (owner a) | `OVER_VIDEO` is black 85 % under `media-scope`: an error on it is the media red in both themes | Only the red changed (3.75:1 on a white frame); black 90 % | A light theme's red was 1.6:1 on the 75 % panel over a white frame; now 5.47:1 at worst, white text too |
|
||||
| 2026-11-07 | A total's left-out videos are "without length" (owner b) | A summed length names what it left out as "+N without length" / "+N hossz nélkül" | "+N live" as before; "+N stream" | The count holds archives and upcoming streams too, which a card calls a stream or upcoming: "live" there named only one of them (One name per feature) |
|
||||
| 2026-11-08 | The Channels table's Length is the sum alone (owner) | The table's Length cell shows the summed hours only; the "+N without length" stays on a playlist's total | The line wrapped in the column; the column widened to 6.5 rem | The 4.5 rem column is sized for the sum: the longer line spilled into Tags (88 px in 56), and the owner wants no such line there |
|
||||
| 2026-11-11 | A GIF from the clip panel (owner D1 A) | A Format · Video / GIF toggle above the name row; GIF shows the size (long side 320 / 480 / 640), the expected size, a note over 15 MB or 30 MB, and over 30 s the reason with Make GIF unavailable; a GIF is never made from the download dialog | A GIF format in the download dialog with its own range picker; the size as a free number | The range is already marked in the panel; a size people can judge beats pixels to type, and a chat's link preview stops moving over 15 MB |
|
||||
| 2026-11-11 | A GIF from the Downloads editor (owner D2 A) | Its own Format row (a video only); with GIF each kept segment is its own GIF, Join unavailable with its reason, one line in place of precise / fast, a note per segment over a limit | One GIF of the joined segments; GIF as an output mode beside Separate / Join | A GIF longer than 30 s is refused anyway, and a joined GIF hides which segment is over the limit |
|
||||
| 2026-11-11 | A shared GIF shows itself (owner D3) | The share page shows the picture itself, no player; Download beside it, and a tip that the downloaded file moves in any chat; its failure is said only once its retry failed too | The watch player; a file row that only downloads | A GIF plays and repeats on its own and nobody pauses one; a chat's link preview is the chat's to draw, the dragged-in file moves everywhere |
|
||||
|
||||
+2
-1
@@ -144,7 +144,8 @@ npm run typecheck # tsc x3 (app / node / e2e)
|
||||
npm run lint # eslint (typed rules included; warnings fail)
|
||||
npm run format:check # prettier
|
||||
npm run knip # unused files/exports/deps
|
||||
npm run test # vitest (incl. the compiler and gate-register contracts)
|
||||
npm run test # vitest (incl. the compiler and gate-register contracts; the refusal contract runs
|
||||
# the host's `python` on backend/tests/refusal_sites.py, stdlib only)
|
||||
npx vite build # the production bundle (devOnlyGuard fails it on dev-only code, chunkGuard on a lazy page loaded at boot)
|
||||
npm audit --omit=dev --audit-level=high # what ships to the browser, at high or above
|
||||
```
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user