Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
32fe0768b6 | ||
|
|
415a6c16f9 | ||
|
|
8c51062e5b | ||
|
|
63bf1bca78 | ||
|
|
c7a377396b | ||
|
|
8036251589 | ||
|
|
71d7646bf1 | ||
|
|
3eb0f41962 | ||
|
|
a0114deff1 | ||
|
|
d89d887a9b | ||
|
|
f8a31d48cf | ||
|
|
7f17b17c13 | ||
|
|
1f81684b6e | ||
|
|
33fddacee2 | ||
|
|
12121255dd | ||
|
|
4a7b65d187 | ||
|
|
47af4d5c05 | ||
|
|
8a83bc0e9f | ||
|
|
41b652f8fd | ||
|
|
545996f807 | ||
|
|
267fd7f70f | ||
|
|
6d49da17fe | ||
|
|
6a92585098 | ||
|
|
31dc4184f5 | ||
|
|
86971c574d | ||
|
|
785080f3da | ||
|
|
76b5f7c93e | ||
|
|
1cfe6a7f59 | ||
|
|
33fe895539 | ||
|
|
5c5cabbe5a | ||
|
|
4f37c41021 | ||
|
|
f7faa19ee2 | ||
|
|
0b4b268678 | ||
|
|
d349bc9084 | ||
|
|
dcd8a9566a | ||
|
|
6de2cd950d | ||
|
|
645d14f45d | ||
|
|
7360339039 | ||
|
|
473d9f02f2 | ||
|
|
d2e225666b | ||
|
|
efd95fca01 | ||
|
|
060deb89d2 | ||
|
|
4869d6c108 | ||
|
|
d20317e0f1 | ||
|
|
d2ba65e383 | ||
|
|
cf7ec5e9ec | ||
|
|
f76a2afbf1 | ||
|
|
889d2acf52 | ||
|
|
782362b808 | ||
|
|
5a42d76e8a | ||
|
|
ed4ab4e13c | ||
|
|
b8bff504d8 | ||
|
|
1630a3df3c | ||
|
|
14b5fe2e31 | ||
|
|
d260aabfd7 | ||
|
|
d832bb19ce | ||
|
|
59efc8f2e8 | ||
|
|
e4ffd59ca7 | ||
|
|
3709b71a94 | ||
|
|
cedebe0ffb | ||
|
|
fdc1eabbe0 | ||
|
|
84f4b20e5a | ||
|
|
828dad015c | ||
|
|
82914b56ca | ||
|
|
ccd5f3739a | ||
|
|
f3c0b7be23 | ||
|
|
53b50231ff | ||
|
|
5703c4d713 | ||
|
|
013f19faca | ||
|
|
8d9a99ce54 | ||
|
|
45dd4bdda7 | ||
|
|
dac391a890 | ||
|
|
4b33036e7c | ||
|
|
80dea4f307 | ||
|
|
20e17c0ccb | ||
|
|
7d302f71b7 | ||
|
|
f99240d8ca | ||
|
|
cc8cf96c19 | ||
|
|
86163958d6 | ||
|
|
12822a9e04 | ||
|
|
928eac4c89 | ||
|
|
b28be682fe | ||
|
|
7e97559615 | ||
|
|
721467ef69 | ||
|
|
47e2293de3 | ||
|
|
29bb914a95 | ||
|
|
5bfcf5d83a | ||
|
|
31fa554b87 | ||
|
|
6a0efba477 | ||
|
|
9f28f46839 | ||
|
|
14f1c97c0a | ||
|
|
fc56f514b3 | ||
|
|
358d722ce9 | ||
|
|
fe38dfbcca | ||
|
|
f98b060e2b | ||
|
|
bc9e34d0ae | ||
|
|
0cc450d1c9 | ||
|
|
688efb9ed6 | ||
|
|
ca4e559cc7 | ||
|
|
e23128b4a1 | ||
|
|
0d5f8f0708 | ||
|
|
a03bf8d371 | ||
|
|
3dc14402b0 | ||
|
|
540ca244c8 | ||
|
|
563fa0513d | ||
|
|
10c0ca4cf7 | ||
|
|
a76bab9b34 | ||
|
|
7acdf5d577 | ||
|
|
b1086e05dc | ||
|
|
ed91c83dd7 | ||
|
|
a614ee2435 | ||
|
|
c74487fbb3 | ||
|
|
876cece881 | ||
|
|
e5997ca567 | ||
|
|
c134b514c1 | ||
|
|
243869401a | ||
|
|
7339a0b84b | ||
|
|
c960c07a6a | ||
|
|
3cfdb36f06 | ||
|
|
ada95d04a4 | ||
|
|
1b004e6ccf | ||
|
|
95c33e0b0f | ||
|
|
4e6bcbc078 | ||
|
|
e17fb5bbf5 | ||
|
|
a9044ece50 | ||
|
|
473aafd1c8 | ||
|
|
7019e3afdf | ||
|
|
c81b110637 | ||
|
|
2ecc70ca6d | ||
|
|
f8c63e80c2 | ||
|
|
3ac3d382e7 | ||
|
|
94c62ca699 | ||
|
|
1fe24d374a | ||
|
|
2cba6b9fca | ||
|
|
8d7aaad9a8 | ||
|
|
85043013f0 | ||
|
|
44f626f732 | ||
|
|
24c93cbb83 | ||
|
|
cb1f4f00b0 | ||
|
|
f6e1d1e69f | ||
|
|
b64c3f514d | ||
|
|
b5a1002412 | ||
|
|
ea931dca30 | ||
|
|
d954e71a77 | ||
|
|
602f62967a | ||
|
|
ba00b50dea | ||
|
|
92adb593c8 | ||
|
|
66132b6dda | ||
|
|
d09f021f0c | ||
|
|
d7e9240e54 | ||
|
|
f2fa49b778 | ||
|
|
d4ed45b2eb | ||
|
|
375f6e58be | ||
|
|
c1a3b7b935 | ||
|
|
bf98b48517 | ||
|
|
b563155a37 | ||
|
|
915319fa35 | ||
|
|
a2a8f5681d | ||
|
|
4827aebb7f | ||
|
|
840526b1b8 | ||
|
|
10d69cb316 |
+1872
File diff suppressed because it is too large
Load Diff
@@ -26,8 +26,8 @@ The local gate is **ruff (`check` + `format --check`) + typecheck (3 tsc project
|
||||
keep commits logical and Conventional-Commit-styled.
|
||||
- **No AI attribution in commits.** No `Co-Authored-By` trailer, no "generated by" lines. (This file
|
||||
being AI-written is the intended, visible exception; commits are not.)
|
||||
- **Release: user-owned in timing, agent-run in execution.** The user decides *when* to ship; on their
|
||||
go-ahead *you* run `siftlode release` from the feature branch (merge to `main` → publish → deploy →
|
||||
- **Release: user-owned in timing, agent-run in execution.** The user decides _when_ to ship; on their
|
||||
go-ahead _you_ run `siftlode release` from the feature branch (merge to `main` → publish → deploy →
|
||||
verify `/api/version`). Never self-ship — bring work to a green, committed state and stop for the
|
||||
go-ahead first, then run it yourself.
|
||||
- **Public repo.** Keep ops/topology/secrets out — those live in a separate private repo. Nothing here
|
||||
@@ -41,7 +41,7 @@ how it gets built. Concretely:
|
||||
- **Plan before you build.** For anything non-trivial, present a short plan (sprint-level for larger
|
||||
work) and get a go-ahead before writing code.
|
||||
- **Offer choices, mark the optimal — and explain the trade-off.** When several solutions are viable,
|
||||
list them and mark the recommended one. This is the *one* place to spend extra words: give the *why*
|
||||
list them and mark the recommended one. This is the _one_ place to spend extra words: give the _why_
|
||||
behind each option and the weight of the decision. These decision points are where the product owner
|
||||
engages with the build and learns from it, so make them understandable — a reasoned menu, not a bare one.
|
||||
- **Mockups for design/UX decisions.** For any visual or interaction choice, show mockup options to
|
||||
@@ -59,31 +59,38 @@ how it gets built. Concretely:
|
||||
`downloads/`, `youtube/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
|
||||
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`,
|
||||
`security`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
|
||||
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → components; data
|
||||
via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through `lib/api/*`
|
||||
slices over `core.req<T>`.
|
||||
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → module
|
||||
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
|
||||
its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
|
||||
boundary is one-way and **lint-enforced**, not just described: a platform component may not import a
|
||||
feature component or a domain API slice (`no-restricted-imports` in `eslint.config.js`, with its own
|
||||
test). A new generic component goes in `ui/`; see [`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
|
||||
Data via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through
|
||||
`lib/api/*` slices over `core.req<T>`.
|
||||
|
||||
**Not wanted** (reviewed and rejected as over-engineering for this size): microservices, event bus,
|
||||
CQRS, a repository/DI framework, a router library, a generic CRUD-router factory or base serializer.
|
||||
|
||||
## Canonical patterns — when you have a choice, pick THIS
|
||||
|
||||
| Area | Use | Not |
|
||||
|---|---|---|
|
||||
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
|
||||
| Request body | a Pydantic model | `payload: dict` |
|
||||
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
|
||||
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
|
||||
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
|
||||
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
|
||||
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`) | ad-hoc `{"unsubscribed"}`/`{"cleared": n}`/echo |
|
||||
| Transaction commit | the handler/job-wrapper commits; helpers `flush` | helpers that commit (only `quota` owns its session) |
|
||||
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
|
||||
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
|
||||
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
|
||||
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
|
||||
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
|
||||
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
|
||||
| Area | Use | Not |
|
||||
| ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------ |
|
||||
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
|
||||
| Request body | a Pydantic model | `payload: dict` |
|
||||
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
|
||||
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
|
||||
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
|
||||
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
|
||||
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`) | ad-hoc `{"unsubscribed"}`/`{"cleared": n}`/echo |
|
||||
| Transaction commit | the handler/job-wrapper commits; helpers `flush` | helpers that commit (only `quota` owns its session) |
|
||||
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
|
||||
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
|
||||
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
|
||||
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
|
||||
| FE action button | `ui/Button` — pick a `SIZES` rung | a hand-rolled `<button className="px-3 py-2 …">` |
|
||||
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
|
||||
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
|
||||
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
|
||||
|
||||
## House rules that bite (learned from real bugs)
|
||||
|
||||
@@ -100,6 +107,6 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
|
||||
|
||||
---
|
||||
|
||||
*Run-it-locally and the full gate commands live in [`docs/development.md`](docs/development.md); backend
|
||||
_Run-it-locally and the full gate commands live in [`docs/development.md`](docs/development.md); backend
|
||||
code conventions in [`backend/app/README.md`](backend/app/README.md). This file stays the source of
|
||||
truth for the canonical-patterns table and the gate/workflow/collaboration rules above.*
|
||||
truth for the canonical-patterns table and the gate/workflow/collaboration rules above._
|
||||
|
||||
+13
@@ -60,6 +60,19 @@ RUN apt-get update \
|
||||
&& rm /tmp/deno.zip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Intel NEO OpenCL runtime (intel-opencl-icd) — needed by the Plex transcode's HDR/10-bit HW path, which
|
||||
# tone-maps HDR→SDR on the GPU via `tonemap_opencl` (Gen 9.5 / UHD 630 can't tonemap in the VAAPI VPP, so
|
||||
# OpenCL is the only HW route; without this the 10-bit branch drops to CPU libx264). NEO was REMOVED from
|
||||
# Debian trixie (this base = python:3.12-slim = trixie), so it's pulled from **bookworm** — the 22.43 there
|
||||
# still supports Gen 9.5 and its .deb runs on trixie (verified live on the prod UHD 630). The bookworm apt
|
||||
# source is added only for this one install and removed after, so it can't drift the rest of the image.
|
||||
# Harmless on hosts with no /dev/dri or no OpenCL use (the runtime just sits unused).
|
||||
RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nComponents: main\nSigned-By: /usr/share/keyrings/debian-archive-keyring.pgp\n' > /etc/apt/sources.list.d/bookworm.sources \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y --no-install-recommends -t bookworm intel-opencl-icd \
|
||||
&& rm /etc/apt/sources.list.d/bookworm.sources \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
|
||||
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
|
||||
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
|
||||
|
||||
@@ -0,0 +1,661 @@
|
||||
GNU AFFERO GENERAL PUBLIC LICENSE
|
||||
Version 3, 19 November 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU Affero General Public License is a free, copyleft license for
|
||||
software and other kinds of works, specifically designed to ensure
|
||||
cooperation with the community in the case of network server software.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
our General Public Licenses are intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
Developers that use our General Public Licenses protect your rights
|
||||
with two steps: (1) assert copyright on the software, and (2) offer
|
||||
you this License which gives you legal permission to copy, distribute
|
||||
and/or modify the software.
|
||||
|
||||
A secondary benefit of defending all users' freedom is that
|
||||
improvements made in alternate versions of the program, if they
|
||||
receive widespread use, become available for other developers to
|
||||
incorporate. Many developers of free software are heartened and
|
||||
encouraged by the resulting cooperation. However, in the case of
|
||||
software used on network servers, this result may fail to come about.
|
||||
The GNU General Public License permits making a modified version and
|
||||
letting the public access it on a server without ever releasing its
|
||||
source code to the public.
|
||||
|
||||
The GNU Affero General Public License is designed specifically to
|
||||
ensure that, in such cases, the modified source code becomes available
|
||||
to the community. It requires the operator of a network server to
|
||||
provide the source code of the modified version running there to the
|
||||
users of that server. Therefore, public use of a modified version, on
|
||||
a publicly accessible server, gives the public access to the source
|
||||
code of the modified version.
|
||||
|
||||
An older license, called the Affero General Public License and
|
||||
published by Affero, was designed to accomplish similar goals. This is
|
||||
a different license, not a version of the Affero GPL, but Affero has
|
||||
released a new version of the Affero GPL which permits relicensing under
|
||||
this license.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, if you modify the
|
||||
Program, your modified version must prominently offer all users
|
||||
interacting with it remotely through a computer network (if your version
|
||||
supports such interaction) an opportunity to receive the Corresponding
|
||||
Source of your version by providing access to the Corresponding Source
|
||||
from a network server at no charge, through some standard or customary
|
||||
means of facilitating copying of software. This Corresponding Source
|
||||
shall include the Corresponding Source for any work covered by version 3
|
||||
of the GNU General Public License that is incorporated pursuant to the
|
||||
following paragraph.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the work with which it is combined will remain governed by version
|
||||
3 of the GNU General Public License.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU Affero General Public License from time to time. Such new versions
|
||||
will be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU Affero General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU Affero General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU Affero General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If your software can interact with users remotely through a computer
|
||||
network, you should also make sure that it provides a way for users to
|
||||
get its source. For example, if your program is a web application, its
|
||||
interface could display a "Source" link that leads users to an archive
|
||||
of the code. There are many ways you could offer source, and different
|
||||
solutions will be better for different programs; see section 13 for the
|
||||
specific requirements.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
@@ -94,9 +94,18 @@ docker compose -f docker-compose.selfhost.yml pull # or: docker compose pull
|
||||
docker compose -f docker-compose.selfhost.yml up -d
|
||||
```
|
||||
|
||||
Database migrations run automatically on startup. Your data (accounts, subscriptions, playlists, the
|
||||
video catalog) lives in a Postgres volume — back it up with `scripts/backup.sh` (or `backup.ps1` on
|
||||
Windows) and restore with `scripts/restore.sh`.
|
||||
That follows `latest`, which moves with every release. Set `IMAGE_TAG=0.80.0` in `.env` to pin a
|
||||
version and upgrade on your own schedule; released versions are listed in [`CHANGELOG.md`](CHANGELOG.md)
|
||||
and in the app's Release Notes. The registry keeps only the newest few of those, so a pin decides
|
||||
*when* you take the next hop rather than letting you stay put forever — see
|
||||
[docs/self-hosting.md](docs/self-hosting.md#upgrading).
|
||||
|
||||
**Back up before upgrading.** Migrations run automatically on startup and are not undone by pulling
|
||||
the old image back. A backup is the database dump **plus your `.env`** — `TOKEN_ENCRYPTION_KEY` is
|
||||
what decrypts the Google, SMTP and Plex secrets stored in that database, so a dump restored without it
|
||||
is only half an instance. Use `scripts/backup.sh` (or `backup.ps1` on Windows) and `scripts/restore.sh`;
|
||||
the equivalent one-liners, the pinning recipe and the rollback steps are in
|
||||
[docs/self-hosting.md](docs/self-hosting.md#backups).
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -111,6 +120,12 @@ Windows) and restore with `scripts/restore.sh`.
|
||||
FastAPI + PostgreSQL (SQLAlchemy, Alembic) backend; React + Vite + Tailwind + TanStack Query
|
||||
frontend; packaged as a single Docker image with Docker Compose.
|
||||
|
||||
## License
|
||||
|
||||
[GNU AGPL-3.0](LICENSE). You can run, study, modify and share Siftlode; if you modify it and let other
|
||||
people use it over a network, they are entitled to your modified source. Copyright © 2026 Siftlode
|
||||
contributors.
|
||||
|
||||
## Note
|
||||
|
||||
This project is developed with AI assistance.
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
"""plex_items playback-class facet (E7: filter the feed by Direct Play / Direct Stream / Transcode)
|
||||
|
||||
Adds the columns the playback-class filter needs:
|
||||
* transcode_hw — for a `playable="transcode"` item, whether the prod iGPU can hardware-transcode it
|
||||
(codec HW-decodable AND 8-bit SDR) vs software-only. NULL for non-transcode items.
|
||||
* video_bit_depth — the source video bit depth (from the Plex media Stream), NULL until probed.
|
||||
* media_probed_at — watermark for the bit-depth backfill (NULL = not yet probed).
|
||||
|
||||
Backfill for existing rows: `transcode_hw` is derived from the ALREADY-stored `codec_video` alone (the
|
||||
same HW-decodable allow-list `plex.sync.HW_DECODABLE_CODECS` / `plex.stream` use), so the filter works
|
||||
immediately after migration. Bit depth is unknown for existing rows (the cheap Plex listing doesn't
|
||||
carry it), so a 10-bit/HDR HEVC is provisionally 'hw' here and later refined to 'sw' by the bit-depth
|
||||
backfill (`plex.sync._probe_media_batch`) once its Stream is probed. `video_bit_depth`/`media_probed_at`
|
||||
start NULL.
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0077_plex_playback_class"
|
||||
down_revision: Union[str, None] = "0076_media_asset_kind"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("plex_items", sa.Column("transcode_hw", sa.Boolean(), nullable=True))
|
||||
op.add_column("plex_items", sa.Column("video_bit_depth", sa.Integer(), nullable=True))
|
||||
op.add_column(
|
||||
"plex_items", sa.Column("media_probed_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
op.create_index("ix_plex_items_transcode_hw", "plex_items", ["transcode_hw"])
|
||||
# Codec-only derivation for existing rows — keep this codec set identical to
|
||||
# plex.sync.HW_DECODABLE_CODECS. Bit depth is unknown here, so provisionally 8-bit (→ hw for a
|
||||
# decodable codec); the bit-depth backfill downgrades 10-bit/HDR HEVC to sw later.
|
||||
op.execute(
|
||||
"""
|
||||
UPDATE plex_items
|
||||
SET transcode_hw = CASE
|
||||
WHEN playable = 'transcode'
|
||||
THEN lower(codec_video) IN ('h264','hevc','h265','vp9','vc1','mpeg2video')
|
||||
ELSE NULL
|
||||
END
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_plex_items_transcode_hw", table_name="plex_items")
|
||||
op.drop_column("plex_items", "media_probed_at")
|
||||
op.drop_column("plex_items", "video_bit_depth")
|
||||
op.drop_column("plex_items", "transcode_hw")
|
||||
@@ -0,0 +1,58 @@
|
||||
"""users.password_revoked_at — remember that SEC-10 dropped a preset password
|
||||
|
||||
Revision ID: 0078_password_revoked_at
|
||||
Revises: 0077_plex_playback_class
|
||||
Create Date: 2026-09-02
|
||||
|
||||
SEC-10 (`auth.revoke_preset_password_if_unverified`) nulls `password_hash` when an account whose email
|
||||
was never verified is attested (Google adoption, or admin approval). That left NO trace, and
|
||||
password-reset eligibility keyed off `password_hash IS NOT NULL` — so such an account could neither
|
||||
sign in (no password) nor request a reset link (not "eligible"): a permanent lockout with a silent,
|
||||
uniform response at every step.
|
||||
|
||||
This column is that trace. A row with `password_revoked_at` set is a local account that HAD a password
|
||||
and had it revoked, so the reset flow treats it as eligible again. Setting a password (reset-confirm or
|
||||
/auth/set-password) clears the stamp. A Google-only account that never had a password keeps it NULL and
|
||||
stays reset-ineligible by design — its owner adds a password from Settings while signed in.
|
||||
|
||||
Backfill: the signature is exact and narrow — a local (google_sub IS NULL), non-demo account with NO
|
||||
password could ONLY have got there through the revoke. Registration always sets a hash, the setup
|
||||
wizard sets one too, every Google row carries a google_sub (the adoption path assigns it BEFORE
|
||||
revoking), and nothing in the codebase ever clears google_sub. Stamp those with the current time so
|
||||
they become recoverable without operator intervention.
|
||||
|
||||
Deliberately NOT also requiring `email_verified = false`, even though the revoke only fires on an
|
||||
unverified account: verifying is exactly what such a victim does NEXT. The revoke does not invalidate
|
||||
the outstanding verification token, so clicking the emailed link sets email_verified=True while
|
||||
password_hash stays NULL — and that row has no route back at all (reset ineligible, resend refuses an
|
||||
already-verified address, and set-password needs a signed-in session the account can no longer obtain:
|
||||
no password to sign in with, and no Google identity). Those are the victims who most need the stamp,
|
||||
so the predicate must not exclude them.
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0078_password_revoked_at"
|
||||
down_revision: Union[str, None] = "0077_plex_playback_class"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"users",
|
||||
sa.Column("password_revoked_at", sa.DateTime(timezone=True), nullable=True),
|
||||
)
|
||||
op.execute(
|
||||
"UPDATE users SET password_revoked_at = now() "
|
||||
"WHERE password_hash IS NULL "
|
||||
"AND google_sub IS NULL "
|
||||
"AND is_demo = false"
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("users", "password_revoked_at")
|
||||
@@ -0,0 +1,65 @@
|
||||
"""email_canonical on users + invites, for matching one mailbox across Gmail spellings
|
||||
|
||||
Revision ID: 0079_email_canonical
|
||||
Revises: 0078_password_revoked_at
|
||||
Create Date: 2026-09-04
|
||||
|
||||
Gmail delivers `x.y@gmail.com`, `xy@gmail.com` and `xy+tag@gmail.com` to ONE mailbox; the invite gate
|
||||
only lowercased. On 2026-09-03 that turned an approved user's Google sign-in into a silent second
|
||||
access request (OAuth returns the account's stored primary, which need not be the spelling they typed)
|
||||
and left prod with two user rows for one mailbox.
|
||||
|
||||
The backfill below reproduces `app.utils.canonical_email` in SQL. That duplication is deliberate — a
|
||||
migration must not import application code that will keep changing — but it means the two have to
|
||||
agree; `test_canonical_email_backfill_matches_the_helper` pins them together.
|
||||
|
||||
Deliberately NOT unique. Prod already carries canonical duplicates: two approved invites for one
|
||||
mailbox dating back to 2026-06-16, and two active user rows from the 2026-09-03 walkthrough. Merging
|
||||
accounts is a judgement call about people's data, not something a migration gets to do on its own, so
|
||||
this migration only records the canonical form and leaves the duplicates in place for a human.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0079_email_canonical"
|
||||
down_revision: str | None = "0078_password_revoked_at"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
# Mirrors app.utils.canonical_email: Gmail (and its googlemail alias) loses dots and any `+tag` from
|
||||
# the local part and lands on gmail.com; every other domain is only lowercased, because the dot rule
|
||||
# is not guaranteed elsewhere — and does not hold for Workspace custom domains unless their admin
|
||||
# enabled it.
|
||||
_CANON_SQL = """
|
||||
CASE
|
||||
WHEN split_part(lower({col}), '@', 2) IN ('gmail.com', 'googlemail.com')
|
||||
-- an empty local part after stripping falls through to plain lowercase, as the helper does
|
||||
AND replace(split_part(split_part(lower({col}), '@', 1), '+', 1), '.', '') <> ''
|
||||
THEN replace(split_part(split_part(lower({col}), '@', 1), '+', 1), '.', '') || '@gmail.com'
|
||||
ELSE lower({col})
|
||||
END
|
||||
"""
|
||||
|
||||
|
||||
def _backfill(table: str) -> None:
|
||||
op.execute(
|
||||
f"UPDATE {table} SET email_canonical = {_CANON_SQL.format(col='email')} "
|
||||
f"WHERE email IS NOT NULL"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for table in ("users", "invites"):
|
||||
op.add_column(
|
||||
table,
|
||||
sa.Column("email_canonical", sa.String(length=320), nullable=False, server_default=""),
|
||||
)
|
||||
_backfill(table)
|
||||
op.create_index(f"ix_{table}_email_canonical", table, ["email_canonical"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for table in ("users", "invites"):
|
||||
op.drop_index(f"ix_{table}_email_canonical", table_name=table)
|
||||
op.drop_column(table, "email_canonical")
|
||||
@@ -0,0 +1,32 @@
|
||||
"""users.google_email — the address Google asserts, kept beside the one the user registered with
|
||||
|
||||
Revision ID: 0080_google_email
|
||||
Revises: 0079_email_canonical
|
||||
Create Date: 2026-09-04
|
||||
|
||||
Since 0079's follow-up a Google sign-in no longer overwrites `users.email`, so a deliberate
|
||||
`me+siftlode@gmail.com` survives being linked to a Google account whose primary is `me@gmail.com`.
|
||||
That leaves two true addresses for one account and only one of them stored. This column holds the
|
||||
other, so the account menu and the Settings sign-in row can show which address Google actually
|
||||
answers with.
|
||||
|
||||
No backfill is possible: Google's answer is only known at sign-in. Accounts linked before this
|
||||
column existed carry NULL until their owner next signs in with Google — and for the majority the two
|
||||
addresses are identical anyway, where the UI shows nothing either way.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0080_google_email"
|
||||
down_revision: str | None = "0079_email_canonical"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("users", sa.Column("google_email", sa.String(length=320), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("users", "google_email")
|
||||
@@ -0,0 +1,34 @@
|
||||
"""invites.invited_at — when the admin actually told this person the door was open
|
||||
|
||||
Revision ID: 0081_invite_invited_at
|
||||
Revises: 0080_google_email
|
||||
Create Date: 2026-09-05
|
||||
|
||||
Whitelisting an address stays SILENT by design: the operator is opening a door as a possibility,
|
||||
usually after agreeing it with the person, and a row can sit approved for weeks without anyone
|
||||
registering. Telling them is therefore a separate, deliberate act — a send button on the row.
|
||||
|
||||
Which raises the one question the admin cannot otherwise answer: "did I already write to them?"
|
||||
Without a record the only failure mode is writing to a real person twice, which is exactly the
|
||||
nudging the silent-by-default design exists to avoid. The audit log holds the event, but not where
|
||||
the decision is made.
|
||||
|
||||
NULL means "never invited from here", which is true of every existing row: no backfill is possible
|
||||
or wanted, since the historic approval mails answered a REQUEST rather than inviting anyone.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0081_invite_invited_at"
|
||||
down_revision: str | None = "0080_google_email"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("invites", sa.Column("invited_at", sa.DateTime(timezone=True), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("invites", "invited_at")
|
||||
@@ -0,0 +1,34 @@
|
||||
"""invites.lang — the language this person asked us in
|
||||
|
||||
Revision ID: 0082_invite_lang
|
||||
Revises: 0081_invite_invited_at
|
||||
Create Date: 2026-09-05
|
||||
|
||||
The mails an access request produces — approved, denied, invited — go to an address that often has
|
||||
no account yet, so there is no `preferences.language` to read. The signal does exist, though, and
|
||||
only at the moment of the request: the language of the page the form was submitted from, or the
|
||||
locale Google reported for a sign-in we refused. Recording it there is what lets the decision mail
|
||||
answer in the requester's own language instead of an admin's guess about a stranger.
|
||||
|
||||
Captured rather than derived, for the third time in this epic: `preferences.language` (S6) and
|
||||
`google_email` (S3) both taught that a signal which exists only in one moment cannot be backfilled
|
||||
later. So no backfill here either — NULL means "we were never told", which is the honest state for
|
||||
every historic row, and for every address an admin whitelists by hand. Those fall back to English
|
||||
unless the admin picks otherwise on the invitation send.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0082_invite_lang"
|
||||
down_revision: str | None = "0081_invite_invited_at"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("invites", sa.Column("lang", sa.String(length=8), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("invites", "lang")
|
||||
@@ -0,0 +1,78 @@
|
||||
"""comfort_pins — the shelf you fall back to when you cannot decide
|
||||
|
||||
Revision ID: 0083_comfort_pins
|
||||
Revises: 0082_invite_lang
|
||||
Create Date: 2026-09-06
|
||||
|
||||
E24's one new table. The shelf holds five different things — a video, a YouTube channel, a Plex show,
|
||||
a Plex movie, a playlist — because comfort media is usually "put on X", a series or a channel, and
|
||||
almost never one particular video.
|
||||
|
||||
Three shapes were possible and two are wrong here:
|
||||
|
||||
- a boolean on `video_states` / `plex_states` cannot ORDER a shelf, and has nowhere to put a channel
|
||||
or a show (neither has a per-user state row at all);
|
||||
- a polymorphic `(kind, ref)` string pair orders and holds anything, but drops the foreign key — and a
|
||||
Plex item removed from the library, or a channel the user unsubscribes and prunes, would then leave
|
||||
a dead tile on the one surface whose entire job is to be reliable when nothing else is.
|
||||
|
||||
So: one nullable CASCADE-ing FK per kind, plus `num_nonnulls(...) = 1`. The union invariant lives in
|
||||
the schema rather than in a comment at every insert site, which is the same rule that the
|
||||
`cancelled`/`canceled` bug taught this codebase.
|
||||
|
||||
NULLs are DISTINCT in a Postgres unique index, so the five per-kind `(user_id, <ref>)` uniques each
|
||||
stop a double-pin without colliding with the other four kinds' NULL rows.
|
||||
|
||||
No backfill: an empty shelf is the honest state for every existing account. The surface renders its
|
||||
own empty state and invites the first pin.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0083_comfort_pins"
|
||||
down_revision: str | None = "0082_invite_lang"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"comfort_pins",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column(
|
||||
"user_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("users.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column("video_id", sa.String(length=16), sa.ForeignKey("videos.id", ondelete="CASCADE")),
|
||||
sa.Column(
|
||||
"channel_id", sa.String(length=32), sa.ForeignKey("channels.id", ondelete="CASCADE")
|
||||
),
|
||||
sa.Column("plex_item_id", sa.Integer(), sa.ForeignKey("plex_items.id", ondelete="CASCADE")),
|
||||
sa.Column("plex_show_id", sa.Integer(), sa.ForeignKey("plex_shows.id", ondelete="CASCADE")),
|
||||
sa.Column("playlist_id", sa.Integer(), sa.ForeignKey("playlists.id", ondelete="CASCADE")),
|
||||
sa.Column("position", sa.Integer(), nullable=False, server_default="0"),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.DateTime(timezone=True),
|
||||
server_default=sa.func.now(),
|
||||
nullable=False,
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"num_nonnulls(video_id, channel_id, plex_item_id, plex_show_id, playlist_id) = 1",
|
||||
name="ck_comfort_pin_one_ref",
|
||||
),
|
||||
sa.UniqueConstraint("user_id", "video_id", name="uq_comfort_pin_video"),
|
||||
sa.UniqueConstraint("user_id", "channel_id", name="uq_comfort_pin_channel"),
|
||||
sa.UniqueConstraint("user_id", "plex_item_id", name="uq_comfort_pin_plex_item"),
|
||||
sa.UniqueConstraint("user_id", "plex_show_id", name="uq_comfort_pin_plex_show"),
|
||||
sa.UniqueConstraint("user_id", "playlist_id", name="uq_comfort_pin_playlist"),
|
||||
)
|
||||
op.create_index("ix_comfort_pins_user_id", "comfort_pins", ["user_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_comfort_pins_user_id", table_name="comfort_pins")
|
||||
op.drop_table("comfort_pins")
|
||||
@@ -27,9 +27,18 @@ class AuditAction:
|
||||
USER_SUSPEND = "user_suspend"
|
||||
USER_UNSUSPEND = "user_unsuspend"
|
||||
USER_DELETE = "user_delete"
|
||||
# E9.5 S5: an admin let a stuck registrant in, or re-sent their confirmation link. Both are
|
||||
# recorded because both are the admin acting ON someone's access, not merely reading a list.
|
||||
USER_ACTIVATE = "user_activate"
|
||||
USER_VERIFY_RESEND = "user_verify_resend"
|
||||
USER_SETPW_LINK = "user_setpw_link"
|
||||
INVITE_APPROVE = "invite_approve"
|
||||
INVITE_DENY = "invite_deny"
|
||||
INVITE_ADD = "invite_add"
|
||||
INVITE_DELETE = "invite_delete"
|
||||
# E9.5 S6: the admin chose to tell someone their address is whitelisted. Adding the address is
|
||||
# silent, so this is the act that actually reaches a person — and the one worth a record.
|
||||
INVITE_SEND = "invite_send"
|
||||
DEMO_ADD = "demo_add"
|
||||
DEMO_REMOVE = "demo_remove"
|
||||
DEMO_RESET = "demo_reset"
|
||||
|
||||
+140
-9
@@ -7,11 +7,13 @@ from sqlalchemy.orm import Session
|
||||
from starlette.requests import HTTPConnection
|
||||
|
||||
from app import email as email_mod
|
||||
from app import mailtext
|
||||
from app import sysconfig
|
||||
from app.config import settings
|
||||
from app.db import get_db
|
||||
from app.models import Invite, User
|
||||
from app.security import decrypt_optional
|
||||
from app.utils import canonical_email
|
||||
|
||||
PASSWORD_MIN_LEN = 10
|
||||
|
||||
@@ -63,16 +65,126 @@ def has_write_scope(user: User) -> bool:
|
||||
return WRITE_SCOPE in tok.scopes.split()
|
||||
|
||||
|
||||
def env_allowed_canonicals() -> set[str]:
|
||||
"""The env bootstrap allow-list, in CANONICAL form — the only shape it may be compared in.
|
||||
|
||||
S3 canonicalised the Invite branch of `is_allowed` and left this one matching the literal string,
|
||||
which is the 2026-09-03 lockout with a different source of truth: an operator who writes
|
||||
`first.last@gmail.com` into ALLOWED_EMAILS is refused when Google answers with the undotted
|
||||
primary, and a `+tag`ged entry can never match at all, because a Google account cannot contain a
|
||||
`+`. The addresses themselves stay literal wherever we MAIL them (`admin_notify_groups`); only
|
||||
the comparison is canonical.
|
||||
"""
|
||||
return env_admin_canonicals() | {canonical_email(e) for e in settings.allowed_email_set}
|
||||
|
||||
|
||||
def env_admin_canonicals() -> set[str]:
|
||||
"""The env ADMIN_EMAILS bootstrap list, canonical — for the role check only. Kept separate from
|
||||
`env_allowed_canonicals` because being listed as an admin also grants access, but being allowed
|
||||
does not make anyone an admin."""
|
||||
return {canonical_email(e) for e in settings.admin_email_set}
|
||||
|
||||
|
||||
def is_allowed(db: Session, email: str) -> bool:
|
||||
"""May this email sign in? An approved Invite is the source of truth; the env
|
||||
ALLOWED_EMAILS/ADMIN_EMAILS remain a bootstrap fallback (e.g. a fresh DB)."""
|
||||
ALLOWED_EMAILS/ADMIN_EMAILS remain a bootstrap fallback (e.g. a fresh DB).
|
||||
|
||||
Matched on the CANONICAL form (utils.canonical_email), not the literal string. Gmail delivers
|
||||
every dotted / `+tagged` spelling to one mailbox, so approving one spelling always meant approving
|
||||
that mailbox — matching literally only produced the 2026-09-03 lockout, where an approved user's
|
||||
Google sign-in became a silent second access request because OAuth returned the undotted primary.
|
||||
|
||||
One canonical form can match SEVERAL invite rows (prod carries approved pairs for one mailbox,
|
||||
the oldest from 2026-06-16). An approved row among them grants access: the API cannot flip an
|
||||
approved invite to denied (`_decide` only accepts a pending one), so approved-beside-denied means
|
||||
the admin approved that mailbox under a different spelling, not that they revoked it. Revoking is
|
||||
deleting the invite row or suspending the account — both still work, since a deleted row is no
|
||||
longer here to match and suspension is enforced at sign-in.
|
||||
"""
|
||||
if not email:
|
||||
return False
|
||||
email = email.lower()
|
||||
inv = db.execute(select(Invite).where(Invite.email == email)).scalar_one_or_none()
|
||||
if inv is not None:
|
||||
return inv.status == "approved"
|
||||
return email in settings.allowed_email_set or email in settings.admin_email_set
|
||||
canon = canonical_email(email)
|
||||
statuses = (
|
||||
db.execute(select(Invite.status).where(Invite.email_canonical == canon)).scalars().all()
|
||||
)
|
||||
if statuses:
|
||||
return "approved" in statuses
|
||||
return canon in env_allowed_canonicals()
|
||||
|
||||
|
||||
def can_activate(db: Session, user: User) -> bool:
|
||||
"""Is this account merely waiting for an admin to finish letting it in?
|
||||
|
||||
True when it is inactive but its address is ALREADY allowed — the shape roadmap UX-25 describes:
|
||||
the admin approved the address, the person signed up afterwards, and nothing connects the two
|
||||
(the Access-requests queue only offers Approve on a still-pending invite).
|
||||
|
||||
Gated on `is_allowed`, the same function the sign-in paths consult, because activating an address
|
||||
that is NOT allowed manufactures an inconsistent account: password sign-in only checks
|
||||
`is_active` and would let them in, while their Google sign-in would be turned away and silently
|
||||
reopened as a fresh access request. Letting in someone with no approved invite is the whitelist
|
||||
box's job — that action says so out loud.
|
||||
|
||||
Lives in core so the admin surface and the /api/me badge count answer from ONE definition; a
|
||||
button that appears and a number that counts must never disagree about who is stuck.
|
||||
"""
|
||||
if user.is_demo or user.is_suspended or user.is_active:
|
||||
return False
|
||||
return is_allowed(db, user.email)
|
||||
|
||||
|
||||
def count_pending_activations(db: Session) -> int:
|
||||
"""How many sign-ups are waiting on an admin click (see can_activate). Drives the admin badge.
|
||||
|
||||
Deliberately NOT "everyone who cannot sign in": an account that has simply not confirmed its
|
||||
address yet is waiting on its owner, not on the admin, and badging that would nag about something
|
||||
the admin did not cause. It is still listed on the admin screen — with a re-send — because the
|
||||
mail may never have arrived.
|
||||
"""
|
||||
# Narrowed in SQL, then decided by can_activate itself — so the number the badge shows and the
|
||||
# button the admin sees cannot drift apart by one of them re-deriving the rule.
|
||||
rows = (
|
||||
db.execute(
|
||||
select(User).where(
|
||||
User.is_demo.is_(False),
|
||||
User.is_suspended.is_(False),
|
||||
User.is_active.is_(False),
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
return sum(1 for u in rows if can_activate(db, u))
|
||||
|
||||
|
||||
def find_user_by_email(db: Session, email: str) -> User | None:
|
||||
"""The account someone means when they type an address at sign-in, password-reset or
|
||||
verify-resend. None when there isn't exactly one answer.
|
||||
|
||||
Exact spelling wins. Failing that, the address is resolved to its mailbox
|
||||
(utils.canonical_email) — so a password chosen at registration still works when the person types
|
||||
another spelling of their own Gmail, and so the three flows that look an account up by typed
|
||||
address stop being dead ends. Before this, requesting a reset for `xy@gmail.com` on an account
|
||||
registered as `x.y@gmail.com` issued nothing and said nothing: the endpoint answers uniformly by
|
||||
design, so the user got silence and no way in.
|
||||
|
||||
SEVERAL canonical candidates return None rather than a guess. That case is real — prod carries two
|
||||
accounts for one mailbox — and picking one of them for a password sign-in would hand someone the
|
||||
wrong account. The callers all treat None as "no such account", which is the uniform,
|
||||
non-enumerating answer they already gave.
|
||||
"""
|
||||
if not email:
|
||||
return None
|
||||
email = email.strip().lower()
|
||||
exact = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
|
||||
if exact is not None:
|
||||
return exact
|
||||
candidates = (
|
||||
db.execute(select(User).where(User.email_canonical == canonical_email(email)))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
return candidates[0] if len(candidates) == 1 else None
|
||||
|
||||
|
||||
def get_or_create_demo_user(db: Session) -> User:
|
||||
@@ -119,6 +231,9 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
|
||||
touch any browser session — each caller handles its own."""
|
||||
email = user.email.lower()
|
||||
user_id = user.id
|
||||
# Read the language off the row while it still exists — the confirmation below is queued as a
|
||||
# background task that runs after this session (and this user) are gone.
|
||||
lang = mailtext.lang_for_user(user)
|
||||
tok = user.token
|
||||
# Both are stored encrypted — decrypt for the revoke call (the access-token fallback was passing
|
||||
# ciphertext, so a token row with no refresh token silently failed to revoke). decrypt_optional so a
|
||||
@@ -130,13 +245,25 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
|
||||
)
|
||||
# Raw delete so Postgres applies ON DELETE CASCADE / SET NULL on every dependent table.
|
||||
db.execute(delete(User).where(User.id == user_id))
|
||||
db.execute(delete(Invite).where(Invite.email == email))
|
||||
# Every canonically-equal invite goes, not just the literal spelling: is_allowed matches on the
|
||||
# canonical form, so leaving a sibling row behind would keep granting access to a mailbox whose
|
||||
# account was just erased — and prod really does carry approved pairs for one mailbox.
|
||||
#
|
||||
# Unless ANOTHER account still lives on that mailbox — prod carries that too (two user rows for
|
||||
# one Gmail mailbox). Then NO invite goes, not even the literal one: is_allowed matches
|
||||
# canonically, so every row here is what keeps the survivor allowed, and removing any of them
|
||||
# would refuse their next Google sign-in and reopen it as a fresh access request — this epic's
|
||||
# founding bug, caused by its own cleanup. `delete_invite` refuses for exactly this reason; the
|
||||
# user row is already gone by now, so this query sees only the others.
|
||||
canon = canonical_email(email)
|
||||
if db.execute(select(User.id).where(User.email_canonical == canon)).first() is None:
|
||||
db.execute(delete(Invite).where(Invite.email_canonical == canon))
|
||||
db.commit()
|
||||
if google_token:
|
||||
background.add_task(revoke_google_token, google_token)
|
||||
# Confirm the erasure to the (now former) user — GDPR good practice, and covers both the
|
||||
# self-service and admin deletion paths since both funnel through here.
|
||||
background.add_task(email_mod.send_account_deleted, email, operator_contact())
|
||||
background.add_task(email_mod.send_account_deleted, email, lang, operator_contact())
|
||||
|
||||
|
||||
ACTIVE_ACCOUNT_HEADER = "x-siftlode-account"
|
||||
@@ -214,7 +341,11 @@ def require_human(user: User = Depends(current_user)) -> User:
|
||||
no token, so has_read_scope/has_write_scope are False); this makes the boundary explicit
|
||||
where there's no scope check to lean on."""
|
||||
if user.is_demo:
|
||||
raise HTTPException(status_code=403, detail="Not available in the demo account.")
|
||||
# Structured, because `req` leaves 403 to the caller and `notifyActionError` only shows a
|
||||
# STRUCTURED detail — a plain string here reached the demo user as their caller's generic
|
||||
# "couldn't do that", with no hint that the demo account is the reason (round 6). Affordances
|
||||
# this gate sits behind are not all UI-gated on `is_demo` (the comfort shelf is not).
|
||||
raise HTTPException(status_code=403, detail={"code": "demo"})
|
||||
return user
|
||||
|
||||
|
||||
|
||||
@@ -265,12 +265,17 @@ class Settings(BaseSettings):
|
||||
# --- Transcode (playable="transcode") re-encode tuning (E6 S1) ---
|
||||
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
|
||||
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
|
||||
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_want_hw`.
|
||||
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
|
||||
plex_transcode_hw: str = "auto"
|
||||
# Software-fallback libx264 tuning (IGNORED on the VAAPI path). `preset` trades CPU for size; `crf`
|
||||
# trades quality for size (lower = better + bigger). `max_height` caps the SOFTWARE transcode's output
|
||||
# height (0 = keep source) so a CPU re-encode of a tall source stays affordable — the GPU path always
|
||||
# encodes at source resolution.
|
||||
# Loudness normalisation of the downmixed stereo audio on the HLS path (EBU R128 `loudnorm`).
|
||||
# OFF by default: it makes a film ~11 dB louder and narrows its dynamic range, which is a taste
|
||||
# call, not a fix. Only affects files that go through remux/transcode — a direct-played file is
|
||||
# served untouched, but those are browser-native stereo anyway, not the 5.1 mixes this is for.
|
||||
plex_audio_loudnorm: bool = False
|
||||
plex_transcode_preset: str = "veryfast"
|
||||
plex_transcode_crf: int = 20
|
||||
plex_transcode_max_height: int = 1080
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
"""E24 — the read-model behind the Home surface ("Mit nézzek most?").
|
||||
|
||||
The anti-paralysis answer layer: one hero suggestion, a cross-source resume rail, and the user's own
|
||||
comfort shelf, all served from our own Postgres with no upstream call at serve time (the same rule
|
||||
the Plex Discover surface follows).
|
||||
|
||||
Layering, mirroring `app/plex/`: `budget` (the time bands) and `cards` (the unified row shape) are
|
||||
pure; `queries` holds the SQL; `rails`, `pins` and `hero` compose. Nothing here imports from
|
||||
`app.routes` — `routes/discover.py` is a thin HTTP shell over these.
|
||||
"""
|
||||
@@ -0,0 +1,62 @@
|
||||
"""The time budget — E24's replacement for "what mood are you in?".
|
||||
|
||||
`random-gondolatok` 15/D asked for a mood questionnaire before the recommendations. The same file's
|
||||
later BA-interview note (#23) says the mood half is not what makes this work: the friction is
|
||||
deciding at all. A mood→genre mapping would also be invented data — we store no mood signal and never
|
||||
will — whereas "how much time do you have" is a question the library can actually answer, from a
|
||||
column we already have.
|
||||
|
||||
So the surface asks about TIME, and the answer is deterministic: four bands, no model, no latency.
|
||||
The band also picks the source for free — ten minutes is a YouTube video, an evening is a movie —
|
||||
which is exactly the cross-source behaviour the Home surface exists to provide.
|
||||
|
||||
The bands overlap on purpose. "I have 30 minutes" should not be answered with a two-minute Short just
|
||||
because it fits; a lower bound is as much a part of the promise as the upper one.
|
||||
"""
|
||||
|
||||
from typing import Literal
|
||||
|
||||
Budget = Literal["short", "medium", "long", "any"]
|
||||
|
||||
DEFAULT_BUDGET: Budget = "any"
|
||||
|
||||
# budget -> (min_seconds, max_seconds); None = unbounded on that side.
|
||||
BANDS: dict[str, tuple[int | None, int | None]] = {
|
||||
"short": (None, 12 * 60), # "10 perc" — a bit of slack over the round number the chip shows
|
||||
"medium": (8 * 60, 40 * 60), # "30 perc"
|
||||
"long": (40 * 60, None), # "Egy este"
|
||||
"any": (None, None), # "Mindegy"
|
||||
}
|
||||
|
||||
|
||||
def band(budget: str) -> tuple[int | None, int | None]:
|
||||
"""The (min, max) seconds for a budget. Unknown budgets fall back to the unbounded band rather
|
||||
than raising: the route validates the vocabulary with a Literal, so anything reaching here with a
|
||||
bad value is our own bug, and an unfiltered shelf is a better failure than a 500 on the landing
|
||||
page."""
|
||||
return BANDS.get(budget, BANDS["any"])
|
||||
|
||||
|
||||
def remaining_seconds(duration_seconds: int | None, position_seconds: int) -> int | None:
|
||||
"""What is actually LEFT to watch. The question is "how much time do you have", so a 90-minute
|
||||
film with 8 minutes to go belongs in the ten-minute band, not the evening one. Unknown duration
|
||||
stays unknown — see `fits`."""
|
||||
if duration_seconds is None:
|
||||
return None
|
||||
return max(0, duration_seconds - max(0, position_seconds))
|
||||
|
||||
|
||||
def fits(seconds: int | None, budget: str) -> bool:
|
||||
"""Does a runtime fit the budget?
|
||||
|
||||
An unknown duration fits only the unbounded band. We cannot promise an item finishes inside ten
|
||||
minutes when we do not know how long it is, and quietly including it would make the one control
|
||||
on this page dishonest — the failure mode the house rules call an un-honest state."""
|
||||
lo, hi = band(budget)
|
||||
if seconds is None:
|
||||
return lo is None and hi is None
|
||||
if lo is not None and seconds < lo:
|
||||
return False
|
||||
if hi is not None and seconds > hi:
|
||||
return False
|
||||
return True
|
||||
@@ -0,0 +1,141 @@
|
||||
"""The unified discover card: one row shape the Home surface can render without knowing which
|
||||
module it came from.
|
||||
|
||||
This is deliberately a THIN mapping, not a new serializer layer. The per-source projections already
|
||||
exist and are already the shapes the frontend's `VideoCard` and `PlexPosterCard` consume —
|
||||
`feedquery.serialize` for YouTube, `plex.cards.movie_card`/`show_card`/`leaf_card` for Plex — so each
|
||||
unified card carries the native payload under `video` / `plex` alongside the handful of fields the
|
||||
rail chrome itself needs (title, thumbnail, progress, runtime). The Home page therefore renders with
|
||||
the existing card components rather than a third one, and a field added to a feed row or a Plex card
|
||||
still gets added in exactly one place. CLAUDE.md's "not wanted" list names a base serializer for
|
||||
precisely this reason.
|
||||
|
||||
`aspect` is the one genuinely new field. A rail has to know whether a tile is a 16:9 still or a 2:3
|
||||
poster, and that is a property of the ARTWORK, not of the module: a Plex episode ships a landscape
|
||||
still exactly like a YouTube video does, while a movie and a show ship posters. Emitting it lets the
|
||||
rail layer stay domain-free instead of branching on `source`.
|
||||
|
||||
`subtitle` only ever carries DATA — a channel name, a show title, a year. Nothing here composes an
|
||||
English phrase: this repo is HU+EN with zero key drift, and a sentence assembled in the backend is a
|
||||
string no translation file can reach. Anything that needs wording (season counts, "N videó") is left
|
||||
to the client, which has both the native payload and the translations.
|
||||
"""
|
||||
|
||||
from app.discover.budget import remaining_seconds
|
||||
from app.models import Channel, Playlist
|
||||
|
||||
# Plex artwork is a poster for the things you browse and a landscape still for the things you play.
|
||||
_POSTER_KINDS = ("movie", "show")
|
||||
|
||||
|
||||
def _base(
|
||||
*,
|
||||
source: str, # "youtube" | "plex"
|
||||
kind: str, # "video" | "movie" | "show" | "episode" | "channel" | "playlist"
|
||||
id: str,
|
||||
title: str | None,
|
||||
aspect: str, # "wide" (16:9) | "poster" (2:3)
|
||||
subtitle: str | None = None,
|
||||
thumb: str | None = None,
|
||||
duration_seconds: int | None = None,
|
||||
position_seconds: int = 0,
|
||||
status: str = "new",
|
||||
) -> dict:
|
||||
return {
|
||||
"source": source,
|
||||
"kind": kind,
|
||||
"id": id,
|
||||
"title": title or "",
|
||||
"subtitle": subtitle,
|
||||
"thumb": thumb,
|
||||
"aspect": aspect,
|
||||
"duration_seconds": duration_seconds,
|
||||
"position_seconds": position_seconds,
|
||||
"remaining_seconds": remaining_seconds(duration_seconds, position_seconds),
|
||||
"status": status,
|
||||
# Native payloads — at most one is ever set. The client opens the player from these.
|
||||
"video": None,
|
||||
"plex": None,
|
||||
}
|
||||
|
||||
|
||||
def key(card: dict) -> str:
|
||||
"""The identity of a card ACROSS sources — `<source>:<id>`. A Plex rating_key and a YouTube video
|
||||
id could in principle collide as bare strings, and the reroll's exclusion list has to be able to
|
||||
say "not that one" about either."""
|
||||
return f"{card['source']}:{card['id']}"
|
||||
|
||||
|
||||
def from_video(row: dict) -> dict:
|
||||
"""`row` is a `feedquery.serialize` dict — the same object the feed grid renders."""
|
||||
card = _base(
|
||||
source="youtube",
|
||||
kind="video",
|
||||
id=row["id"],
|
||||
title=row["title"],
|
||||
aspect="wide",
|
||||
subtitle=row.get("channel_title"),
|
||||
thumb=row.get("thumbnail_url"),
|
||||
duration_seconds=row.get("duration_seconds"),
|
||||
position_seconds=row.get("position_seconds") or 0,
|
||||
status=row.get("status") or "new",
|
||||
)
|
||||
card["video"] = row
|
||||
return card
|
||||
|
||||
|
||||
def from_plex(row: dict) -> dict:
|
||||
"""`row` is a `plex.cards` dict (movie / show / leaf). Its own `type` is the kind."""
|
||||
kind = row.get("type") or "movie"
|
||||
if kind == "episode":
|
||||
subtitle = row.get("show_title")
|
||||
elif kind == "show":
|
||||
subtitle = None # season count lives in the native payload; the client words it
|
||||
else:
|
||||
subtitle = str(row["year"]) if row.get("year") else None
|
||||
|
||||
card = _base(
|
||||
source="plex",
|
||||
kind=kind,
|
||||
id=row["id"],
|
||||
title=row.get("title"),
|
||||
aspect="poster" if kind in _POSTER_KINDS else "wide",
|
||||
subtitle=subtitle,
|
||||
thumb=row.get("thumb"),
|
||||
duration_seconds=row.get("duration_seconds"),
|
||||
position_seconds=row.get("position_seconds") or 0,
|
||||
status=row.get("status") or "new",
|
||||
)
|
||||
card["plex"] = row
|
||||
return card
|
||||
|
||||
|
||||
def from_channel(ch: Channel, unwatched: int) -> dict:
|
||||
"""A pinned channel. It is not itself playable — the shelf resolves it to its newest unwatched
|
||||
video on click (see `pins._shelf_card`) — so the tile advertises what is waiting there instead of a
|
||||
runtime it does not have."""
|
||||
card = _base(
|
||||
source="youtube",
|
||||
kind="channel",
|
||||
id=ch.id,
|
||||
title=ch.title,
|
||||
aspect="wide",
|
||||
thumb=ch.thumbnail_url,
|
||||
)
|
||||
card["unwatched_count"] = unwatched
|
||||
return card
|
||||
|
||||
|
||||
def from_playlist(pl: Playlist, item_count: int, thumb: str | None) -> dict:
|
||||
"""A pinned playlist. `thumb` is its first item's thumbnail — a shelf of blank tiles would defeat
|
||||
the point of a surface you are supposed to be able to scan at a glance."""
|
||||
card = _base(
|
||||
source="youtube",
|
||||
kind="playlist",
|
||||
id=str(pl.id),
|
||||
title=pl.name,
|
||||
aspect="wide",
|
||||
thumb=thumb,
|
||||
)
|
||||
card["item_count"] = item_count
|
||||
return card
|
||||
@@ -0,0 +1,104 @@
|
||||
"""The hero: the single suggestion the Home surface leads with.
|
||||
|
||||
One rule, in one function, because the page's entire promise is that it answers rather than offers.
|
||||
Three tiers, tried in order:
|
||||
|
||||
1. **resume** — something already started whose REMAINING time fits the budget. Almost always the
|
||||
right answer: it is the thing the user themselves chose most recently, and finishing it costs no
|
||||
decision at all.
|
||||
2. **comfort** — a pinned shelf entry that fits. The user's own declared fallback, which beats
|
||||
anything we could infer.
|
||||
3. **surprise** — a seeded draw from what is unseen and fits.
|
||||
|
||||
That order is the PAGE's answer. "Mást kérek" rotates it (`start`), so each press offers a different
|
||||
KIND of answer rather than the next row of the same list — the user's call, 2026-09-06, after the
|
||||
button was measured cycling nine resume cards and never once reaching the shelf.
|
||||
|
||||
The tier travels with the card so the client can label it honestly ("Ott hagytad abba" vs "A
|
||||
polcodról" vs "Meglepetés") instead of guessing from the shape of the payload — a kicker that says
|
||||
"where you left off" over a film the user has never opened is exactly the kind of quiet lie the house
|
||||
rules call a dishonest state.
|
||||
|
||||
Budgets are matched against REMAINING time throughout (`budget.remaining_seconds`, which equals the
|
||||
runtime for anything not yet started): with eight minutes left, a 90-minute film is a ten-minute
|
||||
answer.
|
||||
"""
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.discover import cards, pins, rails
|
||||
from app.discover.budget import fits
|
||||
from app.models import User
|
||||
|
||||
HERO_TIERS = ("resume", "comfort", "surprise")
|
||||
|
||||
# How deep the surprise tier draws before giving up. Deliberately deeper than one, so an excluded
|
||||
# pick moves the tier along instead of emptying it (the shuffle is seeded — the same seed always
|
||||
# hands back the same order, so without the headroom "already shown" would mean "nothing left").
|
||||
SURPRISE_DRAW = 12
|
||||
|
||||
|
||||
def pick(
|
||||
db: Session,
|
||||
user: User,
|
||||
budget: str,
|
||||
seed: int,
|
||||
*,
|
||||
resume: list[dict] | None = None,
|
||||
shelf: list[dict] | None = None,
|
||||
exclude: set[str] | None = None,
|
||||
start: int = 0,
|
||||
) -> dict | None:
|
||||
"""`{"tier": ..., "card": ...}`, or None when nothing at all fits.
|
||||
|
||||
`resume` and `shelf` may be passed pre-computed so the consolidated `/api/discover` response does
|
||||
not build the same two rails twice; the reroll endpoint passes neither and pays only for what it
|
||||
reaches.
|
||||
|
||||
`exclude` is what makes "Mást kérek" mean something. A new seed alone only re-draws the SURPRISE
|
||||
tier: the resume tier deterministically returns the most recently progressed thing that fits, so
|
||||
pressing reroll on a user who has something half-finished handed back the identical card and the
|
||||
button looked dead (measured in the browser, 2026-09-06). Skipping the cards already shown makes
|
||||
every press produce a different answer, at every tier.
|
||||
|
||||
`start` ROTATES which tier is tried first — 0 keeps the priority order (the page's own first
|
||||
answer), 1 leads with comfort, 2 with surprise. Exclusion alone was not enough: the resume rail
|
||||
holds up to 20 rows, so the button walked down it and never reached the shelf or a surprise at
|
||||
all (measured, 2026-09-06: twelve presses returned nine cards, every one of them `resume`). The
|
||||
fallthrough is kept, so a press does not come back empty just because the tier it led with has
|
||||
nothing to offer."""
|
||||
skip = exclude or set()
|
||||
|
||||
def _resume() -> dict | None:
|
||||
for card in rails.continue_watching(db, user) if resume is None else resume:
|
||||
playable = rails.playable(db, user.id, card)
|
||||
# Excluded on the card the USER saw (the show), not only on what it resolves to, so a
|
||||
# rerolled-past show does not come straight back as its own next episode.
|
||||
if cards.key(card) in skip or (playable is not None and cards.key(playable) in skip):
|
||||
continue
|
||||
if playable is not None and fits(playable["remaining_seconds"], budget):
|
||||
return playable
|
||||
return None
|
||||
|
||||
def _comfort() -> dict | None:
|
||||
for card in pins.playables(pins.shelf(db, user) if shelf is None else shelf):
|
||||
if cards.key(card) not in skip and fits(card["remaining_seconds"], budget):
|
||||
return card
|
||||
return None
|
||||
|
||||
def _surprise() -> dict | None:
|
||||
# A wider draw than the one card we need: the shuffle is seeded, so without the headroom an
|
||||
# excluded pick would leave the tier empty rather than move past it.
|
||||
for card in rails.surprise(db, user, budget, seed, limit=SURPRISE_DRAW):
|
||||
if cards.key(card) not in skip:
|
||||
return card
|
||||
return None
|
||||
|
||||
tiers = {"resume": _resume, "comfort": _comfort, "surprise": _surprise}
|
||||
offset = start % len(HERO_TIERS)
|
||||
for tier in HERO_TIERS[offset:] + HERO_TIERS[:offset]:
|
||||
card = tiers[tier]()
|
||||
if card is not None:
|
||||
return {"tier": tier, "card": card}
|
||||
|
||||
return None
|
||||
@@ -0,0 +1,173 @@
|
||||
"""The comfort shelf: pin a thing you can always fall back to, and have it be one click from playing.
|
||||
|
||||
Five kinds are pinnable — a video, a YouTube channel, a Plex show, a Plex movie, a playlist — because
|
||||
comfort media is almost always "put on X", a series or a channel, and only sometimes one particular
|
||||
video. `models.ComfortPin` carries that union as five nullable FKs under a `num_nonnulls(...) = 1`
|
||||
check; this module is where the union is spoken about in one vocabulary, so no call site invents a
|
||||
second one.
|
||||
|
||||
Every shelf card carries a resolved `next`: the thing that actually plays when you click the tile —
|
||||
a show's in-progress-or-next episode, a channel's newest unwatched video, a playlist's first
|
||||
unwatched item. It costs a query per non-leaf pin, which is affordable precisely because this shelf is
|
||||
short and hand-curated, and it buys the property the whole surface exists for: one click, no
|
||||
intermediate page, no second decision.
|
||||
|
||||
Domain module, so nothing here raises HTTP. `resolve_ref` returns None for "no such thing, or not
|
||||
yours" and the route turns that into its 404 — core never imports routes.
|
||||
"""
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.discover import cards, queries
|
||||
from app.models import Channel, ComfortPin, PlexItem, PlexShow, Playlist, User, Video
|
||||
from app.plex import rollup
|
||||
from app.plex.cards import leaf_card, show_card
|
||||
from app.plex.lookups import by_rating_key
|
||||
|
||||
# kind -> the ComfortPin column that holds it. The client's vocabulary and the schema's, in one place.
|
||||
PIN_COLUMNS: dict[str, str] = {
|
||||
"video": "video_id",
|
||||
"channel": "channel_id",
|
||||
"plex_item": "plex_item_id",
|
||||
"plex_show": "plex_show_id",
|
||||
"playlist": "playlist_id",
|
||||
}
|
||||
PIN_KINDS = tuple(PIN_COLUMNS)
|
||||
|
||||
|
||||
def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, object] | None:
|
||||
"""Translate the client's (kind, ref) into the (column, value) the row stores, or None.
|
||||
|
||||
The client speaks the ids the rest of the API speaks: a video id, a channel id, a Plex
|
||||
`rating_key`, a playlist id. Plex's internal integer keys never leave the backend, so the
|
||||
translation belongs here rather than in five call sites.
|
||||
|
||||
The COLUMN half of every answer comes from `PIN_COLUMNS`; only the LOOKUP differs per kind
|
||||
(a direct `db.get`, a rating-key translation, an int parse plus an ownership check). Before the
|
||||
E24 review the branches spelled the column names out again and `PIN_COLUMNS` had no readers at
|
||||
all — a constant that called itself the one vocabulary while nothing spoke it."""
|
||||
col = PIN_COLUMNS.get(kind)
|
||||
if col is None:
|
||||
return None
|
||||
if kind == "video":
|
||||
return (col, ref) if db.get(Video, ref) is not None else None
|
||||
if kind == "channel":
|
||||
return (col, ref) if db.get(Channel, ref) is not None else None
|
||||
if kind == "plex_item":
|
||||
it = by_rating_key(db, PlexItem, ref)
|
||||
return (col, it.id) if it is not None else None
|
||||
if kind == "plex_show":
|
||||
sh = by_rating_key(db, PlexShow, ref)
|
||||
return (col, sh.id) if sh is not None else None
|
||||
try:
|
||||
pid = int(ref)
|
||||
except ValueError:
|
||||
return None
|
||||
pl = db.get(Playlist, pid)
|
||||
# A playlist is per-user, unlike the other four (which are shared catalog rows whose PRIVACY
|
||||
# lives in the state tables). Someone else's list must read as "no such thing", not as a 403 —
|
||||
# the same reason `owned_or_404` 404s rather than 403s.
|
||||
return (col, pl.id) if pl is not None and pl.user_id == user.id else None
|
||||
|
||||
|
||||
def list_pins(db: Session, user: User) -> list[ComfortPin]:
|
||||
return list(
|
||||
db.execute(
|
||||
select(ComfortPin)
|
||||
.where(ComfortPin.user_id == user.id)
|
||||
.order_by(ComfortPin.position.asc(), ComfortPin.id.asc())
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def next_position(db: Session, user: User) -> int:
|
||||
"""New pins land at the end of the shelf.
|
||||
|
||||
`max(position) + 1`, not the row count. Positions are never renumbered on removal — `remove_pin`
|
||||
deletes the row, and every FK cascade silently removes a pin when the thing it points at goes
|
||||
away — so past the first hole a count under-counts and would drop the new pin into the MIDDLE of
|
||||
the shelf, or onto a position another pin already holds (nothing in the schema forbids that).
|
||||
Same idiom as `routes/playlists._next_playlist_position`."""
|
||||
return (
|
||||
db.scalar(
|
||||
select(func.coalesce(func.max(ComfortPin.position), -1)).where(
|
||||
ComfortPin.user_id == user.id
|
||||
)
|
||||
)
|
||||
+ 1
|
||||
)
|
||||
|
||||
|
||||
def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
|
||||
"""One pin as a unified card with its resolved `next`. None when the pinned row has gone (the FK
|
||||
cascade removes the pin itself, so this is only the window between a delete and a refetch)."""
|
||||
card: dict | None = None
|
||||
nxt: dict | None = None
|
||||
|
||||
if pin.video_id is not None:
|
||||
row = queries.yt_video_row(db, user, pin.video_id)
|
||||
if row is None:
|
||||
return None
|
||||
card = cards.from_video(row)
|
||||
nxt = dict(card) # a video plays itself
|
||||
|
||||
elif pin.channel_id is not None:
|
||||
ch = db.get(Channel, pin.channel_id)
|
||||
if ch is None:
|
||||
return None
|
||||
card = cards.from_channel(ch, queries.channel_unwatched_count(db, user, ch.id))
|
||||
row = queries.channel_next_video(db, user, ch.id)
|
||||
nxt = cards.from_video(row) if row else None
|
||||
|
||||
elif pin.plex_item_id is not None:
|
||||
it = db.get(PlexItem, pin.plex_item_id)
|
||||
if it is None:
|
||||
return None
|
||||
st = rollup.item_state_map(db, user.id, [it.id]).get(it.id)
|
||||
card = cards.from_plex(leaf_card(it, st))
|
||||
nxt = dict(card) # a movie or a single episode plays itself
|
||||
|
||||
elif pin.plex_show_id is not None:
|
||||
sh = db.get(PlexShow, pin.plex_show_id)
|
||||
if sh is None:
|
||||
return None
|
||||
status = rollup.state_map(db, user.id, [sh.id]).get(sh.id, "new")
|
||||
card = cards.from_plex(show_card(sh, status))
|
||||
ep = queries.next_episode(db, user.id, sh.id)
|
||||
nxt = cards.from_plex(ep) if ep else None
|
||||
|
||||
elif pin.playlist_id is not None:
|
||||
pl = db.get(Playlist, pin.playlist_id)
|
||||
if pl is None:
|
||||
return None
|
||||
count, row = queries.playlist_summary(db, user, pl.id)
|
||||
card = cards.from_playlist(pl, count, row.get("thumbnail_url") if row else None)
|
||||
nxt = cards.from_video(row) if row else None
|
||||
|
||||
if card is None:
|
||||
return None
|
||||
# A COPY, never the card itself: a self-playing pin (a video, a movie) would otherwise make
|
||||
# `card["next"] is card`, which serializes as a circular reference and 500s the whole shelf. The
|
||||
# copy is taken before `pin_id`/`next` are attached, so `next` stays a plain playable card.
|
||||
card["pin_id"] = pin.id
|
||||
card["next"] = nxt
|
||||
return card
|
||||
|
||||
|
||||
def shelf(db: Session, user: User) -> list[dict]:
|
||||
"""The whole shelf, in the user's own order."""
|
||||
out = []
|
||||
for pin in list_pins(db, user):
|
||||
card = _shelf_card(db, user, pin)
|
||||
if card is not None:
|
||||
out.append(card)
|
||||
return out
|
||||
|
||||
|
||||
def playables(shelf_cards: list[dict]) -> list[dict]:
|
||||
"""The `next` of every shelf entry that has one — what the hero may draw from. A pinned channel
|
||||
with nothing unwatched left, or an empty playlist, simply is not an answer right now."""
|
||||
return [c["next"] for c in shelf_cards if c.get("next")]
|
||||
@@ -0,0 +1,348 @@
|
||||
"""The row fetchers behind the Home surface — one place where "what is startable right now" is
|
||||
expressed as SQL, for both sources.
|
||||
|
||||
Two rules shape this module:
|
||||
|
||||
**Reuse the query layer, not the route.** The YouTube half calls `feedquery.filtered_query` /
|
||||
`sort_keys`, so the Home surface inherits the feed's own definitions of subscribed, blocked,
|
||||
explore-visible and in-progress for free, and cannot drift from them. The Plex half calls
|
||||
`plex.rollup` and the `plex.cards` serializers rather than `routes/plex/discover.py`: that route
|
||||
narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
|
||||
sidebar and wrong for a landing page that does not.
|
||||
|
||||
**One seed drives both halves.** The feed orders a shuffle by `md5(id || seed)` so a keyset cursor
|
||||
stays valid across pages; the Plex surprise rail uses `ORDER BY random()`, which is fine for a single
|
||||
un-paged draw but cannot be reproduced. A cross-source reroll needs both halves to move together —
|
||||
otherwise "Mást kérek" re-draws the Plex picks while the YouTube ones stand still — so the Plex side
|
||||
here takes the feed's seeded-md5 idiom instead of `random()`.
|
||||
"""
|
||||
|
||||
from sqlalchemy import and_, func, or_, select
|
||||
from sqlalchemy.orm import Session, aliased
|
||||
|
||||
from app.discover.budget import band
|
||||
from app.feedquery import (
|
||||
feed_columns,
|
||||
filtered_query,
|
||||
not_blocked_cond,
|
||||
order_by,
|
||||
serialize,
|
||||
sort_keys,
|
||||
)
|
||||
from app.models import (
|
||||
Channel,
|
||||
PlaylistItem,
|
||||
PlexItem,
|
||||
PlexShow,
|
||||
PlexState,
|
||||
User,
|
||||
Video,
|
||||
VideoState,
|
||||
)
|
||||
from app.plex import rollup
|
||||
from app.plex.cards import leaf_card, movie_card, show_card
|
||||
from app.plex.query import apply_show_state_filter, content_libs
|
||||
|
||||
# How many rows a single rail may carry. Matches `routes/plex/discover.py`'s RAIL_LIMIT on purpose —
|
||||
# the two surfaces should feel like the same app, and a rail is a rail.
|
||||
RAIL_LIMIT = 20
|
||||
|
||||
# The feed parameters a Home rail runs with. Spelled out rather than borrowed from the route's
|
||||
# `_feed_params` defaults, because those are an HTTP contract that may change for the feed's own
|
||||
# reasons; the landing page's scope should not move with them silently.
|
||||
_FEED_BASE: dict = {
|
||||
"tags": [],
|
||||
"tag_modes": {},
|
||||
"untagged": False,
|
||||
"channel_ids": [],
|
||||
"q": None,
|
||||
"min_duration": None,
|
||||
"max_duration": None,
|
||||
"max_age_days": None,
|
||||
"published_after": None,
|
||||
"published_before": None,
|
||||
"show_normal": True,
|
||||
"include_shorts": False,
|
||||
"include_live": False,
|
||||
"show": "unwatched",
|
||||
"scope": "my",
|
||||
"library_source": "organic",
|
||||
}
|
||||
|
||||
|
||||
def _seeded(col, seed: int):
|
||||
"""The feed's deterministic shuffle expression, reused for Plex so one seed moves both halves."""
|
||||
return func.md5(func.concat(col, str(seed)))
|
||||
|
||||
|
||||
# ── YouTube ───────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def yt_in_progress(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[tuple[object, dict]]:
|
||||
"""Started-but-not-finished videos, most recently progressed first, as (progressed_at, row).
|
||||
|
||||
The predicate is `feedquery`'s `show="in_progress"` verbatim (a stored resume position, status not
|
||||
watched/hidden) — but the query is built here rather than through `filtered_query` because this
|
||||
rail wants an ordering by `progress_updated_at`, and that column lives on the state alias the feed
|
||||
query keeps private. Driving from `video_states` is also the faster plan: it is a per-user table of
|
||||
a few hundred rows, against a catalog of many thousands.
|
||||
|
||||
Building it here means the feed's OTHER predicates have to be chosen deliberately rather than
|
||||
inherited. `not_blocked_cond` is shared, because "never show me this channel" outranks "you
|
||||
started it" — blocking leaves the state row untouched, so without this the video the user just
|
||||
banned heads the resume rail. The feed's SCOPE clauses (subscribed / via_explore accessibility)
|
||||
are deliberately NOT applied, for the reason `yt_video_row` gives: something you have already
|
||||
started must stay resumable even after you unsubscribe or an exploration expires."""
|
||||
st = aliased(VideoState)
|
||||
q = (
|
||||
select(*feed_columns(user, st), st.progress_updated_at.label("progressed_at"))
|
||||
.join(Channel, Channel.id == Video.channel_id)
|
||||
.join(st, and_(st.video_id == Video.id, st.user_id == user.id))
|
||||
.where(
|
||||
st.position_seconds > 0,
|
||||
st.status.notin_(("watched", "hidden")),
|
||||
Video.unavailable_since.is_(None),
|
||||
not_blocked_cond(user),
|
||||
)
|
||||
.order_by(st.progress_updated_at.desc().nullslast())
|
||||
.limit(limit)
|
||||
)
|
||||
return [(r.progressed_at, serialize(r)) for r in db.execute(q).all()]
|
||||
|
||||
|
||||
def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) -> list[dict]:
|
||||
"""Unwatched videos from the user's own feed scope, in seeded-shuffle order, inside the budget.
|
||||
|
||||
Shorts are opened up only for the shortest band: "I have ten minutes" is the one answer a Short is
|
||||
actually a good response to, and the feed's own default keeps them out otherwise."""
|
||||
lo, hi = band(budget)
|
||||
params = dict(_FEED_BASE, min_duration=lo, max_duration=hi, include_shorts=budget == "short")
|
||||
query, rank_expr = filtered_query(db, user, **params)
|
||||
keys = sort_keys("shuffle", seed, rank_expr)
|
||||
# Over-fetch, then drop anything already started. The feed's `unwatched` scope keeps watched and
|
||||
# hidden out but deliberately keeps in-progress rows in (they belong in the feed). Here they must
|
||||
# go: the resume tier has already had its chance at them, and it may have passed precisely because
|
||||
# the REMAINING time did not fit the budget — offering the same item back as a surprise, whose
|
||||
# runtime does fit, would label a half-finished thing "Meglepetés" and contradict the question.
|
||||
rows = db.execute(query.order_by(*order_by(keys)).limit(limit * 4)).all()
|
||||
out = [serialize(r) for r in rows]
|
||||
return [r for r in out if not r["position_seconds"]][:limit]
|
||||
|
||||
|
||||
def channel_next_video(db: Session, user: User, channel_id: str) -> dict | None:
|
||||
"""The newest unwatched video of a pinned channel — what "put on that channel" resolves to."""
|
||||
# `library_source="all"`: a pinned channel means the whole channel, including videos that only
|
||||
# ever arrived through a search. The feed's "organic" default is a feed-scoping choice, not a
|
||||
# statement about what belongs to a channel.
|
||||
params = dict(_FEED_BASE, channel_ids=[channel_id], scope="all", library_source="all")
|
||||
query, rank_expr = filtered_query(db, user, **params)
|
||||
keys = sort_keys("newest", 0, rank_expr)
|
||||
row = db.execute(query.order_by(*order_by(keys)).limit(1)).first()
|
||||
return serialize(row) if row is not None else None
|
||||
|
||||
|
||||
def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
|
||||
"""One video as a feed row, with this user's watch state.
|
||||
|
||||
Built directly rather than through `filtered_query` on purpose: a PINNED video must keep working
|
||||
even when it falls outside the feed's scope — the user unsubscribed from the channel, or blocked
|
||||
it, or the video only ever arrived through a search. The shelf is the surface you fall back to;
|
||||
it silently losing an entry because a filter moved is exactly the failure it must not have."""
|
||||
st = aliased(VideoState)
|
||||
row = db.execute(
|
||||
select(*feed_columns(user, st))
|
||||
.join(Channel, Channel.id == Video.channel_id)
|
||||
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
|
||||
.where(Video.id == video_id)
|
||||
).first()
|
||||
return serialize(row) if row is not None else None
|
||||
|
||||
|
||||
def channel_unwatched_count(db: Session, user: User, channel_id: str) -> int:
|
||||
"""How many unwatched videos a pinned channel is holding — the tile's only honest promise, since a
|
||||
channel has no runtime of its own."""
|
||||
params = dict(_FEED_BASE, channel_ids=[channel_id], scope="all", library_source="all")
|
||||
query, _ = filtered_query(db, user, **params)
|
||||
return int(db.execute(select(func.count()).select_from(query.subquery())).scalar() or 0)
|
||||
|
||||
|
||||
def playlist_summary(db: Session, user: User, playlist_id: int) -> tuple[int, dict | None]:
|
||||
"""(item count, the row to play next) for a pinned playlist — its first not-yet-watched item, or
|
||||
its first item once the whole list is watched. The same row doubles as the tile's artwork, so a
|
||||
shelf of playlists is scannable instead of a row of blank boxes."""
|
||||
st = aliased(VideoState)
|
||||
rows = db.execute(
|
||||
select(*feed_columns(user, st), PlaylistItem.position)
|
||||
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
|
||||
.join(Channel, Channel.id == Video.channel_id)
|
||||
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
|
||||
.where(PlaylistItem.playlist_id == playlist_id)
|
||||
.order_by(PlaylistItem.position.asc())
|
||||
).all()
|
||||
if not rows:
|
||||
return 0, None
|
||||
nxt = next((r for r in rows if (r.status or "new") not in ("watched", "hidden")), rows[0])
|
||||
return len(rows), serialize(nxt)
|
||||
|
||||
|
||||
# ── Plex ──────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _unseen_movie_conds(uid: int):
|
||||
"""A movie the user has neither finished nor hidden. A LEFT-JOIN-IS-NULL would also drop rows that
|
||||
merely carry a resume position, which are exactly the ones Continue watching wants back."""
|
||||
seen = (
|
||||
select(1)
|
||||
.select_from(PlexState)
|
||||
.where(
|
||||
PlexState.item_id == PlexItem.id,
|
||||
PlexState.user_id == uid,
|
||||
or_(
|
||||
PlexState.status.in_(("watched", "hidden")),
|
||||
# Started counts as "not a surprise" too — see the note in `yt_surprise`.
|
||||
PlexState.position_seconds > 0,
|
||||
),
|
||||
)
|
||||
.exists()
|
||||
)
|
||||
return ~seen
|
||||
|
||||
|
||||
def plex_in_progress(db: Session, uid: int, limit: int = RAIL_LIMIT) -> list[tuple[object, dict]]:
|
||||
"""In-progress movies + partially-watched shows as (progressed_at, card).
|
||||
|
||||
Mirrors `routes/plex/discover.py::_continue_watching` — same `PlexState` predicate for movies (a
|
||||
NULL status still counts as started, so `notin_` alone would drop it), and the same shared
|
||||
`rollup.agg_subquery` + `apply_show_state_filter` for shows, so the two rails cannot disagree on
|
||||
what "in progress" means. What it does NOT inherit is that route's `LibraryFilters` narrowing:
|
||||
this surface has no Plex sidebar to read a filter from."""
|
||||
movie_libs, show_libs = content_libs(db)
|
||||
out: list[tuple[object, dict]] = []
|
||||
|
||||
if movie_libs:
|
||||
mq = (
|
||||
select(PlexItem, PlexState)
|
||||
.join(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
|
||||
.where(
|
||||
PlexItem.kind == "movie",
|
||||
PlexItem.library_id.in_(movie_libs),
|
||||
PlexState.position_seconds > 0,
|
||||
or_(
|
||||
PlexState.status.is_(None),
|
||||
PlexState.status.notin_(("watched", "hidden")),
|
||||
),
|
||||
)
|
||||
.order_by(PlexState.progress_updated_at.desc().nullslast())
|
||||
.limit(limit)
|
||||
)
|
||||
for it, st in db.execute(mq).all():
|
||||
out.append((st.progress_updated_at, movie_card(it, st)))
|
||||
|
||||
if show_libs:
|
||||
prog = rollup.agg_subquery(db, uid, include_resumed=True).subquery()
|
||||
sq = apply_show_state_filter(
|
||||
select(PlexShow, prog.c.resumed_at).join(prog, prog.c.show_id == PlexShow.id),
|
||||
prog.c.total,
|
||||
prog.c.watched,
|
||||
prog.c.inprog,
|
||||
"in_progress",
|
||||
).where(PlexShow.library_id.in_(show_libs))
|
||||
for sh, resumed_at in db.execute(
|
||||
sq.order_by(prog.c.resumed_at.desc().nullslast()).limit(limit)
|
||||
).all():
|
||||
out.append((resumed_at, show_card(sh, "in_progress")))
|
||||
|
||||
return out
|
||||
|
||||
|
||||
def plex_surprise(db: Session, uid: int, budget: str, seed: int, limit: int) -> list[dict]:
|
||||
"""Unseen playable movies inside the budget, in seeded-shuffle order.
|
||||
|
||||
Movies only. A show is not a thing you can start playing, and resolving every candidate show to
|
||||
its next episode just to draw three cards would be a query per row for a rail whose whole value is
|
||||
that it answers instantly — shows reach this surface through Continue watching and the comfort
|
||||
shelf, which is where "put the next episode on" actually belongs.
|
||||
|
||||
No `playable` filter, matching the Plex surprise rail: `PlexItem.playable` is a playback CLASS
|
||||
("direct" / "remux" / "transcode"), not a yes/no, and NULL there means "not probed yet" rather than
|
||||
"will not play". Excluding un-probed items would quietly hide a fresh library from the one surface
|
||||
that is supposed to have an answer. The class travels on the card, so the client still shows it."""
|
||||
movie_libs, _ = content_libs(db)
|
||||
if not movie_libs:
|
||||
return []
|
||||
lo, hi = band(budget)
|
||||
q = (
|
||||
select(PlexItem)
|
||||
.where(
|
||||
PlexItem.kind == "movie",
|
||||
PlexItem.library_id.in_(movie_libs),
|
||||
_unseen_movie_conds(uid),
|
||||
)
|
||||
.order_by(_seeded(PlexItem.rating_key, seed))
|
||||
.limit(limit)
|
||||
)
|
||||
if lo is not None:
|
||||
q = q.where(PlexItem.duration_s >= lo)
|
||||
if hi is not None:
|
||||
q = q.where(PlexItem.duration_s <= hi)
|
||||
items = db.execute(q).scalars().all()
|
||||
states = rollup.item_state_map(db, uid, [it.id for it in items])
|
||||
return [movie_card(it, states.get(it.id)) for it in items]
|
||||
|
||||
|
||||
# Watch states that take an episode out of the running for "what would play next".
|
||||
_DONE = ("watched", "hidden")
|
||||
|
||||
|
||||
def next_episode(db: Session, uid: int, show_id: int) -> dict | None:
|
||||
"""What "put that show on" means: the episode in progress, else the first unwatched one, else the
|
||||
first — over the same `(season, episode)` broadcast ordering `detail.py::_episode_nav` uses.
|
||||
|
||||
Three ordered `LIMIT 1` probes rather than one scan picked apart in Python. The Python version
|
||||
loaded EVERY episode of the show to choose one of them — 236 rows for Friends, 227 for JAG — which
|
||||
the comfort shelf could just about afford (a handful of hand-picked pins) and the resume rail
|
||||
could not, since it resolves a show per rail row on the landing page.
|
||||
|
||||
Returned as a `leaf_card` so it carries the show title the episode row itself does not."""
|
||||
asc = (
|
||||
PlexItem.season_number.asc().nullsfirst(),
|
||||
PlexItem.episode_number.asc().nullsfirst(),
|
||||
)
|
||||
base = (
|
||||
select(PlexItem, PlexState)
|
||||
.outerjoin(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
|
||||
.where(PlexItem.show_id == show_id, PlexItem.kind == "episode")
|
||||
.limit(1)
|
||||
)
|
||||
# In progress — the LAST one in broadcast order, which is `plex.rollup.rollup`'s `inprog[-1]`,
|
||||
# i.e. what the show page's own Resume button plays. Two episodes of one show are in progress at
|
||||
# once more often than it sounds (`watch_sync._reconcile_inprogress_movies` documents that a
|
||||
# scrub to the end leaves a stale `status='new', position>0` row until the daily reconcile), and
|
||||
# taking opposite ends of that set is how one show came to have two different "continue" answers:
|
||||
# the Home tile ranked it by its LATEST touch and then offered the EARLIEST started episode.
|
||||
# `position_seconds > 0` already requires the state row to exist, so no null guard.
|
||||
pick = db.execute(
|
||||
base.where(PlexState.position_seconds > 0, PlexState.status.notin_(_DONE)).order_by(
|
||||
PlexItem.season_number.desc().nullslast(),
|
||||
PlexItem.episode_number.desc().nullslast(),
|
||||
)
|
||||
).first()
|
||||
# Unwatched — INCLUDING an episode with no state row at all, which is the common case and the one
|
||||
# a bare `status NOT IN (...)` silently drops (NULL is not false, it is unknown, so the row fails
|
||||
# the filter).
|
||||
pick = (
|
||||
pick
|
||||
or db.execute(
|
||||
base.where(or_(PlexState.id.is_(None), PlexState.status.notin_(_DONE))).order_by(*asc)
|
||||
).first()
|
||||
)
|
||||
# Fully watched show: start it again from the top.
|
||||
pick = pick or db.execute(base.order_by(*asc)).first()
|
||||
if pick is None:
|
||||
return None
|
||||
|
||||
ep, st = pick
|
||||
show = db.get(PlexShow, show_id)
|
||||
return leaf_card(
|
||||
ep, st, show_title=show.title if show else None, show_rk=show.rating_key if show else None
|
||||
)
|
||||
@@ -0,0 +1,97 @@
|
||||
"""The Home surface's rails: cross-source lists, assembled from `queries.py` and emitted as unified
|
||||
cards.
|
||||
|
||||
The whole point of this module is that a rail is ONE list, not a YouTube list beside a Plex list. Two
|
||||
side-by-side rails would just move the choice the user could not make ("which half do I look at?") one
|
||||
level up, which is the opposite of what this epic is for.
|
||||
"""
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import sysconfig
|
||||
from app.discover import cards, queries
|
||||
from app.discover.queries import RAIL_LIMIT
|
||||
from app.models import PlexShow, User
|
||||
from app.plex.lookups import by_rating_key
|
||||
from app.plex.rollup import NO_TS
|
||||
|
||||
|
||||
def plex_available(db: Session) -> bool:
|
||||
"""Plex is a deployment-level toggle, not a per-user one (see `routes/me.py`). With it off, the
|
||||
Home rails are YouTube-only and say nothing about Plex."""
|
||||
return sysconfig.get_bool(db, "plex_enabled")
|
||||
|
||||
|
||||
def playable(db: Session, uid: int, card: dict) -> dict | None:
|
||||
"""The thing that would actually START from this card — itself for anything with a runtime, and
|
||||
for a Plex SHOW the episode that is next (in progress, else the first unwatched, else the first).
|
||||
|
||||
ONE function, because three surfaces need the same answer and a second implementation is how the
|
||||
shelf and the rail came to disagree in the first place: the hero calls it to decide what it is
|
||||
offering, `continue_watching` calls it to fill each show tile's `next`, and the comfort shelf
|
||||
resolves the same way in `pins`. None when the show has no episodes mirrored yet.
|
||||
|
||||
A card that already carries a resolved `next` (a rail row) is taken at its word rather than
|
||||
re-queried — the hero reads the rail it was handed."""
|
||||
if not (card.get("source") == "plex" and card.get("kind") == "show"):
|
||||
return card
|
||||
if "next" in card:
|
||||
return card["next"]
|
||||
sh = by_rating_key(db, PlexShow, card["id"])
|
||||
if sh is None:
|
||||
return None
|
||||
ep = queries.next_episode(db, uid, sh.id)
|
||||
return cards.from_plex(ep) if ep else None
|
||||
|
||||
|
||||
def continue_watching(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[dict]:
|
||||
"""One resume rail across both sources, most recently progressed first.
|
||||
|
||||
A Plex SHOW enters as a show CARD — that is what the Plex rail shows, and it is the artwork the
|
||||
user recognises — but it carries `next`: the episode that would actually start. Without it a
|
||||
click on a half-watched series landed on the show page with six seasons to choose between, which
|
||||
is the decision this whole surface exists to remove (the comfort shelf and the hero already
|
||||
resolved; only this rail did not). The client plays `next ?? card`, so the tile stays a poster and
|
||||
the click stays one click."""
|
||||
rows: list[tuple[object, dict]] = [
|
||||
(ts, cards.from_video(r)) for ts, r in queries.yt_in_progress(db, user, limit)
|
||||
]
|
||||
if plex_available(db):
|
||||
rows += [(ts, cards.from_plex(c)) for ts, c in queries.plex_in_progress(db, user.id, limit)]
|
||||
rows.sort(key=lambda r: r[0] or NO_TS, reverse=True)
|
||||
out = [card for _, card in rows[:limit]]
|
||||
# Resolve AFTER the slice: only the rows that survive it are worth a query.
|
||||
for card in out:
|
||||
if card["kind"] == "show":
|
||||
card["next"] = playable(db, user.id, card)
|
||||
return out
|
||||
|
||||
|
||||
def surprise(db: Session, user: User, budget: str, seed: int, limit: int = 3) -> list[dict]:
|
||||
"""The hero's draw pool: unseen things that fit the budget, from both sources under ONE seed.
|
||||
|
||||
Interleaved rather than concatenated — the same alternation `routes/plex/discover.py` uses for its
|
||||
movie/show split — so a short pool never comes back all-YouTube or all-Plex just because one side
|
||||
happened to be queried first.
|
||||
|
||||
WHICH source leads is decided by the seed, not by a hardcoded order. The hero draws a pool of ONE,
|
||||
so a fixed order would mean the Plex library could never answer "egy este" while a single YouTube
|
||||
video also fit — the reroll would look broken and the cross-source promise would quietly be a
|
||||
YouTube promise. Parity on the same seed that orders the rows keeps it deterministic: the same
|
||||
seed always gives the same answer, and pressing "Mást kérek" moves both halves together."""
|
||||
halves = [
|
||||
(cards.from_video, queries.yt_surprise(db, user, budget, seed, limit)),
|
||||
(
|
||||
cards.from_plex,
|
||||
queries.plex_surprise(db, user.id, budget, seed, limit) if plex_available(db) else [],
|
||||
),
|
||||
]
|
||||
if seed % 2:
|
||||
halves.reverse()
|
||||
|
||||
out: list[dict] = []
|
||||
for i in range(max(len(rows) for _, rows in halves)):
|
||||
for to_card, rows in halves:
|
||||
if i < len(rows):
|
||||
out.append(to_card(rows[i]))
|
||||
return out[:limit]
|
||||
@@ -54,6 +54,10 @@ class EditAborted(Exception):
|
||||
|
||||
# --- edit spec ------------------------------------------------------------------------------
|
||||
|
||||
# SEC-13: ceiling on a join's cut-list. An accurate join emits two filters per segment into a single
|
||||
# ffmpeg `filter_complex`, so "however many the caller sent" is an unbounded command line.
|
||||
_MAX_SEGMENTS = 100
|
||||
|
||||
|
||||
def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
"""Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
|
||||
@@ -88,6 +92,12 @@ def normalize_edit_spec(spec: dict | None) -> dict:
|
||||
if en > st:
|
||||
norm.append({"start_s": round(st, 3), "end_s": round(en, 3)})
|
||||
norm.sort(key=lambda x: x["start_s"])
|
||||
# SEC-13: an accurate join builds one ffmpeg `filter_complex` with two filters per segment,
|
||||
# so an unbounded list becomes an unbounded command line and an ffmpeg that never finishes.
|
||||
# The cap is far above any real cut-list (the UI's own splitter produces a handful) and the
|
||||
# excess is DROPPED rather than raising: the surplus segments are the attack, the first
|
||||
# _MAX_SEGMENTS are a legitimate edit.
|
||||
del norm[_MAX_SEGMENTS:]
|
||||
if norm:
|
||||
out["segments"] = norm
|
||||
# A join always chooses a codec path: accurate=filter-concat re-encode,
|
||||
|
||||
@@ -8,9 +8,28 @@ explanation (and decides whether a retry makes sense). The raw text stays availa
|
||||
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
|
||||
"""
|
||||
|
||||
# Two worker-authored strings that this module owns because both are coupled to `_RULES`.
|
||||
#
|
||||
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
|
||||
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
|
||||
# rule drifting apart is exactly how a category silently becomes unreachable.
|
||||
#
|
||||
# NETWORK_REDACTED_MESSAGE is the opposite direction: it REPLACES a message after classification, so
|
||||
# it deliberately has no rule. Nothing re-classifies a stored `job.error` today; if anything ever
|
||||
# does, it must carry the original code rather than re-deriving one from this text.
|
||||
#
|
||||
# Other worker-raised texts are still inline literals at their raise sites. The ffmpeg ones and
|
||||
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
|
||||
# classifies as `unknown` — it is a refusal, not a diagnosis, and no category would say more.
|
||||
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
|
||||
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
|
||||
|
||||
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
|
||||
# case-insensitive on a substring, against the raw yt-dlp/ffmpeg message.
|
||||
_RULES: tuple[tuple[str, str], ...] = (
|
||||
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
|
||||
# first — it is our wording, so it can never be a false match on an upstream message.
|
||||
("download storage is full", "quota_exceeded"),
|
||||
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
|
||||
("sign in to confirm your age", "login_required"),
|
||||
("confirm your age", "login_required"),
|
||||
@@ -91,3 +110,20 @@ def classify(msg: str | None) -> str:
|
||||
if marker in low:
|
||||
return category
|
||||
return "unknown"
|
||||
|
||||
|
||||
def user_error_text(source_kind: str, msg: str, code: str) -> str:
|
||||
"""The failure text to STORE on the job, given the raw message and its category.
|
||||
|
||||
SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection
|
||||
refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner with
|
||||
extra steps, since the raw text is shown to the user behind the details disclosure. `urlguard`
|
||||
stops the fetch aiming at our own network; this stops the remaining readout.
|
||||
|
||||
Only the network class, and only for ``url`` jobs: a YouTube job has no user-chosen host, and
|
||||
every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis. A separate
|
||||
function rather than an inline branch in the worker so it can be pinned by a test — the raw
|
||||
text leaking back is a security regression that would otherwise pass a green suite."""
|
||||
if source_kind == "url" and code == "network":
|
||||
return NETWORK_REDACTED_MESSAGE
|
||||
return msg
|
||||
|
||||
@@ -116,6 +116,7 @@ def build_ydl_opts(
|
||||
progress_hook,
|
||||
extra: dict | None = None,
|
||||
cookiefile: str | None = None,
|
||||
max_bytes: int | None = None,
|
||||
) -> dict:
|
||||
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
|
||||
|
||||
@@ -125,12 +126,20 @@ def build_ydl_opts(
|
||||
|
||||
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
|
||||
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
|
||||
merge so a source adapter can never clobber it."""
|
||||
merge so a source adapter can never clobber it.
|
||||
|
||||
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp refuses a format larger
|
||||
than it rather than filling the disk and only then being caught by the quota accounting."""
|
||||
s = normalize(spec)
|
||||
h = s["max_height"]
|
||||
opts: dict = {
|
||||
"outtmpl": outtmpl,
|
||||
"noplaylist": True,
|
||||
# SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to the single video. A
|
||||
# URL that is PURELY a playlist or channel has no such video to fall back to, so it still
|
||||
# expands into every entry — one enqueue, an unbounded number of downloads. One job means
|
||||
# one item; a user who wants the whole list enqueues it item by item, under their quota.
|
||||
"playlist_items": "1",
|
||||
"quiet": True,
|
||||
"no_warnings": True,
|
||||
"noprogress": True,
|
||||
@@ -154,6 +163,16 @@ def build_ydl_opts(
|
||||
if extra:
|
||||
opts.update(extra)
|
||||
|
||||
# SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
|
||||
# `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set.
|
||||
#
|
||||
# A non-positive budget is left unset because 0 is not an expressible yt-dlp filter, NOT because
|
||||
# nothing needs to happen: the worker refuses that job outright before extraction (`_OwnerRefused`
|
||||
# → `quota_exceeded`). Enqueue cannot make that call — it ran while the user was still under the
|
||||
# cap. Delete the worker's refusal and this branch silently uncaps the one user already over it.
|
||||
if max_bytes and max_bytes > 0:
|
||||
opts["max_filesize"] = max_bytes
|
||||
|
||||
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
|
||||
# be able to override the authenticated session the worker resolved for this specific user.
|
||||
if cookiefile:
|
||||
|
||||
@@ -15,7 +15,8 @@ from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import errors as dlerrors
|
||||
from app.downloads import adapters, credentials, formats, mediakind, probe, service, storage
|
||||
from app.downloads import adapters, credentials, formats, mediakind, probe, quota, service, storage
|
||||
from app.downloads import urlguard
|
||||
from app.downloads.jobdb import (
|
||||
_edit_staging_dir,
|
||||
_job_status,
|
||||
@@ -36,6 +37,18 @@ class _Aborted(Exception):
|
||||
"""Raised inside the progress hook when the job was paused/canceled mid-download."""
|
||||
|
||||
|
||||
class _OwnerRefused(Exception):
|
||||
"""A refusal that belongs to THIS job's owner, not to the source.
|
||||
|
||||
MediaAssets are shared: two users asking for the same video and format get one row (see
|
||||
`service.get_or_create_asset`), and the generic failure handler writes the message onto it —
|
||||
correctly, because an upstream failure really is everyone's. A per-user refusal is not. Written
|
||||
to the shared row it would fail a second user's queued job for the same video with the first
|
||||
user's private reason (`worker._process_job` reads `asset.error` verbatim), telling someone with
|
||||
an empty library to free up space. So this is caught separately: the job fails, the asset goes
|
||||
back to `pending` for whoever else is holding it."""
|
||||
|
||||
|
||||
# Both progress sources call back FAR faster than the database should be touched: yt-dlp fires its
|
||||
# hook per chunk, and ffmpeg's `-progress pipe:1` emits a block of ~12 lines every ~0.5s while
|
||||
# run_ffmpeg calls its callbacks once per LINE. Every callback here is a round-trip (`_job_status`
|
||||
@@ -305,13 +318,17 @@ def _media_info(info: dict) -> dict:
|
||||
return {**info, **chosen}
|
||||
|
||||
|
||||
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter, cookiefile=None):
|
||||
def _extract_with_fallback(
|
||||
job_id, asset_id, url, spec, staging, adapter, cookiefile=None, max_bytes=None
|
||||
):
|
||||
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
|
||||
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
|
||||
error); other sources make a single attempt. Returns the info dict of the successful attempt.
|
||||
|
||||
`cookiefile` (S6) is the acting user's per-host cookies.txt, already gated by
|
||||
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated."""
|
||||
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated.
|
||||
|
||||
`max_bytes` (SEC-13) is the owner's remaining storage budget, applied to every attempt."""
|
||||
client_sets = adapter.client_sets()
|
||||
outtmpl = str(staging / "%(id)s.%(ext)s")
|
||||
last_exc: Exception | None = None
|
||||
@@ -322,6 +339,7 @@ def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter, cookie
|
||||
_make_progress_hook(job_id, asset_id),
|
||||
extra=adapter.ydl_opts_extra(clients),
|
||||
cookiefile=cookiefile,
|
||||
max_bytes=max_bytes,
|
||||
)
|
||||
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
|
||||
_touch_job(job_id) # stall-watchdog heartbeat: extraction writes no progress, so a slow /
|
||||
@@ -415,8 +433,17 @@ def _download(
|
||||
_set_job(job_id, phase="downloading", progress=0)
|
||||
adapter = adapters.resolve_adapter(source_kind, source_ref)
|
||||
url = service.source_url(source_kind, source_ref)
|
||||
# SEC-12, second half: the enqueue guard ran when the job was created, which may have been
|
||||
# hours ago and against a different DNS answer. This is where the socket actually gets
|
||||
# opened, so the check is repeated here and the job fails rather than fetching.
|
||||
if source_kind == "url":
|
||||
try:
|
||||
urlguard.check_url(url)
|
||||
except urlguard.BlockedUrl as e:
|
||||
raise RuntimeError(f"Refusing to download from this address ({e.reason}).") from e
|
||||
# S6: attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
|
||||
# this asset is that user's private, salted identity (the double gate lives in credentials).
|
||||
remaining_bytes: int | None = None
|
||||
with SessionLocal() as cdb:
|
||||
asset_sig = cdb.execute(
|
||||
select(MediaAsset.format_sig).where(MediaAsset.id == asset_id)
|
||||
@@ -425,6 +452,19 @@ def _download(
|
||||
cookiefile = credentials.resolve_cookiefile(
|
||||
cdb, user_id, url, asset_sig, spec, staging
|
||||
)
|
||||
# SEC-13: what this user has left. `check_enqueue` only compared the footprint as it
|
||||
# stood when the job was QUEUED — a download's size is unknown until it finishes, so a
|
||||
# single job could previously blow through the cap by any amount. Handing the remainder
|
||||
# to yt-dlp as `max_filesize` makes the format selector refuse an oversized file up
|
||||
# front instead.
|
||||
remaining_bytes = quota.remaining_bytes(cdb, user_id)
|
||||
# …and zero is a REFUSAL, not "no limit". Earlier jobs of this user may have filled the cap
|
||||
# while this one sat in the queue, and 0 is not an expressible yt-dlp `max_filesize`, so
|
||||
# passing it down would leave the ceiling unset — the one user already at their cap would be
|
||||
# the only one downloading unbounded. Fail the job here instead. Not retryable until they
|
||||
# free space, which is what the `quota_exceeded` category tells them.
|
||||
if remaining_bytes == 0:
|
||||
raise _OwnerRefused(dlerrors.QUOTA_FULL_MESSAGE)
|
||||
info = _media_info(
|
||||
_extract_with_fallback(
|
||||
job_id,
|
||||
@@ -434,6 +474,7 @@ def _download(
|
||||
staging,
|
||||
adapter,
|
||||
cookiefile=str(cookiefile) if cookiefile else None,
|
||||
max_bytes=remaining_bytes,
|
||||
)
|
||||
)
|
||||
# Re-resolve now that yt-dlp's info reveals the true source (a bare/redirected url only
|
||||
@@ -525,8 +566,31 @@ def _download(
|
||||
# job's status (the route set it to paused/canceled). ref_count is adjusted by the route.
|
||||
_reset_asset_pending(asset_id)
|
||||
log.info("job %s aborted (pause/cancel)", job_id)
|
||||
except _OwnerRefused as exc:
|
||||
# Fail this job only. The asset is never marked `error`; it goes back to `pending` (below),
|
||||
# so another user's job for the same video is claimed and downloaded normally instead of
|
||||
# inheriting a reason that is about someone else's account.
|
||||
msg = str(exc)[:500]
|
||||
with SessionLocal() as db:
|
||||
service.finalize_if_running(
|
||||
db,
|
||||
job_id,
|
||||
status=JobStatus.ERROR,
|
||||
error=msg,
|
||||
error_code=dlerrors.classify(msg),
|
||||
phase=None,
|
||||
)
|
||||
db.commit()
|
||||
# Released AFTER the job is terminal: the moment the asset is `pending` another job can claim
|
||||
# it, and this one must already be out of the way. (The staging tree is still torn down in
|
||||
# `finally` below — the same narrow window the pause/cancel path above has always had.)
|
||||
_reset_asset_pending(asset_id)
|
||||
log.info("job %s refused for its owner: %s", job_id, msg)
|
||||
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
|
||||
msg = str(exc)[:500]
|
||||
code = dlerrors.classify(msg)
|
||||
# SEC-12: redact a transport-level readout for user-supplied URLs (see user_error_text).
|
||||
msg = dlerrors.user_error_text(source_kind, msg, code)
|
||||
with SessionLocal() as db:
|
||||
asset = db.get(MediaAsset, asset_id)
|
||||
if asset:
|
||||
@@ -539,7 +603,7 @@ def _download(
|
||||
job_id,
|
||||
status=JobStatus.ERROR,
|
||||
error=msg,
|
||||
error_code=dlerrors.classify(msg),
|
||||
error_code=code,
|
||||
phase=None,
|
||||
)
|
||||
db.commit()
|
||||
|
||||
@@ -7,6 +7,13 @@ Limits come from a per-user DownloadQuota row if the admin set one, else the sys
|
||||
|
||||
Enforcement split: max_jobs + max_bytes are checked at enqueue (they bound holdings); per-user
|
||||
max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads).
|
||||
|
||||
DOWNLOAD jobs additionally consult `remaining_bytes` in the worker, because enqueue can only compare
|
||||
the footprint as it stood when the job was created and earlier jobs may have filled the cap since.
|
||||
That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user each read the same
|
||||
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
|
||||
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
|
||||
is unknown until it finishes. Edit jobs do not consult it at all (tracked as SEC-19).
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
@@ -122,6 +129,21 @@ def check_enqueue(db: Session, user_id: int) -> None:
|
||||
raise QuotaExceeded("max_bytes", lim.max_bytes, used)
|
||||
|
||||
|
||||
def remaining_bytes(db: Session, user_id: int) -> int | None:
|
||||
"""Storage this user may still spend on ONE download: ``None`` for an unlimited user, else the
|
||||
cap minus the current footprint, floored at 0.
|
||||
|
||||
Zero is a real answer, not "no limit" — it means the cap is already met and the caller must
|
||||
REFUSE rather than proceed unbounded. `check_enqueue` cannot make that refusal on the caller's
|
||||
behalf: it ran when the job was queued, while the user was still under the cap, and the jobs
|
||||
that filled it have landed since. Keeping the arithmetic here (rather than inline in the worker)
|
||||
is what keeps every max_bytes decision in this module."""
|
||||
lim = resolve(db, user_id)
|
||||
if lim.unlimited:
|
||||
return None
|
||||
return max(0, lim.max_bytes - footprint(db, user_id))
|
||||
|
||||
|
||||
def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
|
||||
"""Like check_enqueue, but for an upload whose size is KNOWN at ingest: also reject when adding
|
||||
this file would push the footprint over max_bytes. A download can't pre-check its size (unknown
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Refuse a user-supplied download URL that points back into our own network (SEC-12).
|
||||
|
||||
The Download Center deliberately accepts an arbitrary ``http(s)`` URL — that is the feature — and
|
||||
hands it to yt-dlp running on the WORKER. Without a guard, any approved user can make the server
|
||||
fetch anything the server can reach: other containers on the compose network, the LAN, a cloud
|
||||
metadata endpoint. The failure text then comes back to them in ``job.error``, so "connection
|
||||
refused" vs "HTTP Error 401" vs "timed out" turns the download box into an internal port scanner.
|
||||
|
||||
What this closes: a URL whose host resolves to a loopback / private / link-local / reserved address
|
||||
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
|
||||
differently between the two moments, and the worker is where the socket is actually opened.
|
||||
|
||||
What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
|
||||
the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
|
||||
|
||||
* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
|
||||
with an internal address.
|
||||
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
|
||||
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
|
||||
open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
|
||||
response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
|
||||
the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
|
||||
class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
|
||||
one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
|
||||
kept). Tracked as SEC-17.
|
||||
|
||||
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
|
||||
hostname into the box), which is the reachable one.
|
||||
|
||||
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
|
||||
user-controlled destination to guard.
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import socket
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# Reason codes travel to the client inside the structured 422 detail, so they are part of the API
|
||||
# vocabulary (frontend `localizeDetail`) — keep them stable.
|
||||
REASON_SCHEME = "scheme" # not http(s)
|
||||
REASON_NO_HOST = "no_host" # URL carries no hostname
|
||||
REASON_MALFORMED = "malformed" # unparseable authority or port; urlparse raises late, so guard it
|
||||
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
|
||||
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
|
||||
|
||||
|
||||
class BlockedUrl(Exception):
|
||||
"""A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
|
||||
REASON_* codes above."""
|
||||
|
||||
def __init__(self, reason: str):
|
||||
super().__init__(reason)
|
||||
self.reason = reason
|
||||
|
||||
|
||||
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
"""Every address family we must not open a connection to. Checked explicitly rather than
|
||||
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
|
||||
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
|
||||
mapped = getattr(ip, "ipv4_mapped", None)
|
||||
if mapped is not None:
|
||||
ip = mapped
|
||||
return bool(
|
||||
ip.is_private
|
||||
or ip.is_loopback
|
||||
or ip.is_link_local
|
||||
or ip.is_multicast
|
||||
or ip.is_reserved
|
||||
or ip.is_unspecified
|
||||
)
|
||||
|
||||
|
||||
def _resolve(host: str, port: int | None) -> list[str]:
|
||||
"""Every address ``host`` resolves to. Raises BlockedUrl(REASON_UNRESOLVABLE) when DNS has no
|
||||
answer — reported as its own outcome, because "we could not look that up" and "we refuse to
|
||||
fetch that" are different things to tell a user."""
|
||||
try:
|
||||
infos = socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
|
||||
except socket.gaierror as exc:
|
||||
raise BlockedUrl(REASON_UNRESOLVABLE) from exc
|
||||
return [info[4][0] for info in infos]
|
||||
|
||||
|
||||
def check_url(raw: str) -> None:
|
||||
"""Raise BlockedUrl unless ``raw`` is an http(s) URL whose host resolves only to addresses the
|
||||
server is allowed to fetch from. A host with even ONE forbidden address is refused: a
|
||||
round-robin answer must not become a coin flip."""
|
||||
# `urlparse` itself raises ValueError on a malformed authority ("http://[::1/x" — an unclosed
|
||||
# bracket), and `.port` raises on ":99999" / ":abc" when READ rather than at parse time. Both
|
||||
# escape the caller's `except BlockedUrl` as a 500 unless caught here, which is the opposite of
|
||||
# what this module exists to do: turn a bad URL into a structured, localizable refusal.
|
||||
try:
|
||||
u = urlparse((raw or "").strip())
|
||||
port = u.port
|
||||
except ValueError as exc:
|
||||
raise BlockedUrl(REASON_MALFORMED) from exc
|
||||
if u.scheme not in ("http", "https"):
|
||||
raise BlockedUrl(REASON_SCHEME)
|
||||
host = u.hostname
|
||||
if not host:
|
||||
raise BlockedUrl(REASON_NO_HOST)
|
||||
for addr in _resolve(host, port):
|
||||
try:
|
||||
ip = ipaddress.ip_address(addr)
|
||||
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
|
||||
raise BlockedUrl(REASON_PRIVATE) from None
|
||||
if _is_forbidden(ip):
|
||||
raise BlockedUrl(REASON_PRIVATE)
|
||||
+252
-114
@@ -3,36 +3,103 @@
|
||||
Gmail SMTP + App Password by default. Deliberately fail-soft: if SMTP isn't configured
|
||||
or a send errors, we log and return False so the caller can fall back to in-app
|
||||
notifications — email is never load-bearing for the app's core flow.
|
||||
|
||||
Fail-soft must not mean invisible, though: every drop also leaves an in-app notice in the admins'
|
||||
inbox (see `_alert_admins`). An instance whose SMTP was never filled in silently swallows
|
||||
verification links, password resets and access-request notices, and the operator's first clue used
|
||||
to be a person telling them nothing arrived.
|
||||
"""
|
||||
|
||||
import html
|
||||
import logging
|
||||
import re
|
||||
import smtplib
|
||||
import ssl
|
||||
from email.message import EmailMessage
|
||||
from email.utils import formatdate
|
||||
|
||||
from app import sysconfig
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import mailtext, sysconfig
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import User
|
||||
from app.notifications import create_notification
|
||||
from app.ratelimit import RateLimiter
|
||||
|
||||
log = logging.getLogger("siftlode.email")
|
||||
|
||||
|
||||
def _smtp() -> dict:
|
||||
"""Effective SMTP config (admin DB override over env defaults). Opens its own short
|
||||
session because email is often sent from a BackgroundTask, outside a request's db."""
|
||||
with SessionLocal() as db:
|
||||
return {
|
||||
"host": sysconfig.get_str(db, "smtp_host"),
|
||||
"port": sysconfig.get_int(db, "smtp_port"),
|
||||
"user": sysconfig.get_str(db, "smtp_user"),
|
||||
"password": sysconfig.get_str(db, "smtp_password"),
|
||||
"from": sysconfig.get_str(db, "smtp_from"),
|
||||
}
|
||||
# One admin notice an hour per reason. A relay that is down fails every send, and the inbox must
|
||||
# stay readable; the log keeps the full record either way.
|
||||
_alert_throttle = RateLimiter(max_events=1, window_seconds=3600)
|
||||
|
||||
|
||||
def email_enabled() -> bool:
|
||||
c = _smtp()
|
||||
def _alert_admins(reason: str, subject: str, recipients: list[str]) -> None:
|
||||
"""Record an in-app notice that a message did NOT go out. Best effort in every direction: it
|
||||
opens its own session (callers are often background tasks), swallows its own errors, and never
|
||||
reports failure to the caller — this is the fallback path, it must not become a new way to
|
||||
break a send."""
|
||||
if not _alert_throttle.allow(reason):
|
||||
return
|
||||
try:
|
||||
with SessionLocal() as db:
|
||||
admins = (
|
||||
db.execute(
|
||||
select(User.id).where(
|
||||
User.role == "admin",
|
||||
User.is_active.is_(True),
|
||||
User.is_suspended.is_(False),
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
if not admins:
|
||||
# Nothing was recorded, so don't let this cost the next hour of alerts on the very
|
||||
# path whose job is to stop drops going unseen. Same on the failure branch below.
|
||||
_alert_throttle.reset()
|
||||
return
|
||||
for uid in admins:
|
||||
create_notification(
|
||||
db,
|
||||
user_id=uid,
|
||||
type="email_undelivered",
|
||||
# English fallback; the inbox renders a translated version from `data`.
|
||||
title="Email could not be sent",
|
||||
body=subject,
|
||||
data={"reason": reason, "subject": subject, "recipients": recipients},
|
||||
commit=False,
|
||||
)
|
||||
db.commit()
|
||||
except Exception:
|
||||
_alert_throttle.reset()
|
||||
log.exception("Could not record the undelivered-email notice (%r)", subject)
|
||||
|
||||
|
||||
def _read_smtp(db: Session) -> dict:
|
||||
return {
|
||||
"host": sysconfig.get_str(db, "smtp_host"),
|
||||
"port": sysconfig.get_int(db, "smtp_port"),
|
||||
"user": sysconfig.get_str(db, "smtp_user"),
|
||||
"password": sysconfig.get_str(db, "smtp_password"),
|
||||
"from": sysconfig.get_str(db, "smtp_from"),
|
||||
}
|
||||
|
||||
|
||||
def _smtp(db: Session | None = None) -> dict:
|
||||
"""Effective SMTP config (admin DB override over env defaults). Opens its own short session when
|
||||
the caller has none, because email is usually sent from a BackgroundTask outside a request's db.
|
||||
A caller that IS inside a request should pass its session: reading the password decrypts a Fernet
|
||||
value, and /api/me polls this for every admin tab."""
|
||||
if db is not None:
|
||||
return _read_smtp(db)
|
||||
with SessionLocal() as own:
|
||||
return _read_smtp(own)
|
||||
|
||||
|
||||
def email_enabled(db: Session | None = None) -> bool:
|
||||
c = _smtp(db)
|
||||
return bool(c["host"] and c["user"] and c["password"])
|
||||
|
||||
|
||||
@@ -40,6 +107,48 @@ def _admin_contact() -> str | None:
|
||||
return next(iter(sorted(settings.admin_email_set)), None)
|
||||
|
||||
|
||||
# A bare URL in a text/plain mail is only clickable if the CLIENT decides to linkify it. Gmail,
|
||||
# Outlook and Apple Mail do; plenty of lighter web clients render the body verbatim and leave the
|
||||
# link as dead text — which, for a verification or password-reset mail, is the whole message. So
|
||||
# every mail also carries an HTML alternative built from the SAME plain body (below), rather than
|
||||
# per-template HTML that would have to be kept in sync with it.
|
||||
#
|
||||
# `[^\s<]` stops at whitespace and at the `<` of any following markup; the trailing-punctuation trim
|
||||
# keeps a sentence-final period out of the href. `;` and `:` are deliberately NOT trimmed — escaping
|
||||
# turns `&` into `&`, so a query string can legitimately end in a semicolon.
|
||||
_URL_RE = re.compile(r"https?://[^\s<]+")
|
||||
|
||||
|
||||
def _html_body(text: str) -> str:
|
||||
"""Render a plain-text mail body as minimal HTML: escaped, blank-line-separated paragraphs, with
|
||||
real links. Escaping happens FIRST and the linkifier then runs over the escaped text — bodies
|
||||
interpolate untrusted values (a requester's email address, the operator contact), so no caller
|
||||
can inject markup, and `&` inside a URL is already `&` by the time it reaches the href, which
|
||||
is what an HTML attribute wants anyway."""
|
||||
|
||||
def _anchor(m: re.Match[str]) -> str:
|
||||
url = m.group(0)
|
||||
trail = ""
|
||||
while url and url[-1] in ".,!?)":
|
||||
url, trail = url[:-1], url[-1] + trail
|
||||
if not url:
|
||||
return trail
|
||||
return f'<a href="{url}">{url}</a>{trail}'
|
||||
|
||||
paragraphs = [p for p in html.escape(text).split("\n\n") if p.strip()]
|
||||
rendered = "".join(
|
||||
f"<p>{_URL_RE.sub(_anchor, p).replace(chr(10), '<br>')}</p>" for p in paragraphs
|
||||
)
|
||||
# Inline styles only — many clients drop <style> blocks. No images, no external CSS, no tracking:
|
||||
# the HTML part exists to make the link clickable, not to become a template to maintain.
|
||||
return (
|
||||
'<html><body style="margin:0;padding:16px;font-family:system-ui,-apple-system,'
|
||||
"'Segoe UI',sans-serif;font-size:15px;line-height:1.6\">"
|
||||
f'<div style="max-width:560px">{rendered}</div>'
|
||||
"</body></html>"
|
||||
)
|
||||
|
||||
|
||||
def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -> bool:
|
||||
recipients = [e for e in to if e]
|
||||
if not recipients:
|
||||
@@ -47,6 +156,7 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
|
||||
c = _smtp()
|
||||
if not (c["host"] and c["user"] and c["password"]):
|
||||
log.info("SMTP not configured; skipping email %r to %s", subject, recipients)
|
||||
_alert_admins("not_configured", subject, recipients)
|
||||
return False
|
||||
msg = EmailMessage()
|
||||
msg["Subject"] = subject
|
||||
@@ -57,7 +167,10 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
|
||||
msg["Date"] = formatdate(localtime=True)
|
||||
if reply_to:
|
||||
msg["Reply-To"] = reply_to
|
||||
# text/plain stays the primary part (it is the source of truth, and a text-only client still
|
||||
# gets the full message); the HTML alternative below is what makes the link clickable everywhere.
|
||||
msg.set_content(body)
|
||||
msg.add_alternative(_html_body(body), subtype="html")
|
||||
try:
|
||||
with smtplib.SMTP(c["host"], c["port"], timeout=20) as s:
|
||||
s.starttls(context=ssl.create_default_context())
|
||||
@@ -67,128 +180,153 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
|
||||
return True
|
||||
except Exception:
|
||||
log.exception("SMTP send failed (%r to %s)", subject, recipients)
|
||||
_alert_admins("send_failed", subject, recipients)
|
||||
return False
|
||||
|
||||
|
||||
def send_access_approved(email: str) -> bool:
|
||||
def _next_step(verify_link: str | None, has_account: bool, hours: int, lang: str) -> str:
|
||||
"""What this person actually has to do next, in the one paragraph that carries the whole flow.
|
||||
|
||||
Three states, because a mail that guesses wrong sends someone looking for a link that isn't
|
||||
coming — or worse, tells them to sign in when their address is still unconfirmed and sign-in will
|
||||
refuse them (E9.5 S4 named that state on screen; the mail must not contradict it).
|
||||
|
||||
The link branch is the point of the C option: in the common ordering — approve first, the person
|
||||
signs up later, or signs up and waits — they would otherwise have to find and open TWO separate
|
||||
messages. One mail, one click.
|
||||
"""
|
||||
if verify_link:
|
||||
return mailtext.text(
|
||||
"next_step.verify", lang, link=verify_link, expiry=mailtext.link_expiry(hours, lang)
|
||||
)
|
||||
if has_account:
|
||||
return mailtext.text("next_step.sign_in", lang, app=settings.app_base)
|
||||
return mailtext.text("next_step.register", lang, app=settings.app_base)
|
||||
|
||||
|
||||
def _reply_contact(lang: str) -> tuple[str | None, str]:
|
||||
"""The admin address to answer to, and the closing line that names it — empty when no admin
|
||||
contact is configured, so the mail never promises a reply nobody will read."""
|
||||
admin = _admin_contact()
|
||||
body = (
|
||||
"Hi,\n\n"
|
||||
"Your request to use Siftlode has been approved — welcome aboard.\n\n"
|
||||
"Open Siftlode and sign in, and your YouTube subscriptions feed will be ready:\n\n"
|
||||
f"{settings.app_base}\n\n"
|
||||
"If you weren't expecting this, you can ignore the message.\n\n"
|
||||
"— Siftlode" + (f"\n\nQuestions? Just reply to this email ({admin})." if admin else "")
|
||||
return admin, (mailtext.text("contact.reply", lang, admin=admin) if admin else "")
|
||||
|
||||
|
||||
def send_access_approved(
|
||||
email: str,
|
||||
lang: str,
|
||||
verify_link: str | None = None,
|
||||
has_account: bool = False,
|
||||
hours: int = 24,
|
||||
) -> bool:
|
||||
"""Answer someone who ASKED for access. Sent automatically on approval, because a person who
|
||||
made a request is owed a reply — unlike the admin-initiated invitation below."""
|
||||
admin, contact = _reply_contact(lang)
|
||||
body = mailtext.text(
|
||||
"access_approved.body",
|
||||
lang,
|
||||
next_step=_next_step(verify_link, has_account, hours, lang),
|
||||
contact=contact,
|
||||
)
|
||||
return _send([email], "You're in — your Siftlode access is approved", body, reply_to=admin)
|
||||
return _send([email], mailtext.text("access_approved.subject", lang), body, reply_to=admin)
|
||||
|
||||
|
||||
def send_access_denied(email: str) -> bool:
|
||||
admin = _admin_contact()
|
||||
body = (
|
||||
"Hi,\n\n"
|
||||
"Thanks for your interest in Siftlode. After review, your access request wasn't approved "
|
||||
"this time.\n\n"
|
||||
"If you think this was a mistake, you're welcome to reply and let us know.\n\n"
|
||||
"— Siftlode" + (f"\n\nQuestions? Just reply to this email ({admin})." if admin else "")
|
||||
def send_access_invitation(
|
||||
email: str,
|
||||
lang: str,
|
||||
verify_link: str | None = None,
|
||||
has_account: bool = False,
|
||||
hours: int = 24,
|
||||
) -> bool:
|
||||
"""Invite someone the admin let in WITHOUT them asking — sent only when the admin presses send.
|
||||
|
||||
Whitelisting an address is opening a door, not knocking on one: the operator has usually already
|
||||
agreed it with the person, and an address can sit approved for weeks without anyone registering.
|
||||
So adding to the whitelist stays silent and this is a separate, deliberate act. That is also why
|
||||
the closing line offers no deadline — nudging is the one thing this mail must not do.
|
||||
|
||||
Its language is the one place the app has no signal to read: the admin typed this address in, so
|
||||
nobody has ever told us how to address its owner. That is why the send button asks (E9.5 S7).
|
||||
"""
|
||||
admin, contact = _reply_contact(lang)
|
||||
body = mailtext.text(
|
||||
"access_invitation.body",
|
||||
lang,
|
||||
next_step=_next_step(verify_link, has_account, hours, lang),
|
||||
contact=contact,
|
||||
)
|
||||
return _send([email], "About your Siftlode access request", body, reply_to=admin)
|
||||
return _send([email], mailtext.text("access_invitation.subject", lang), body, reply_to=admin)
|
||||
|
||||
|
||||
def send_role_changed(to: str, role: str, operator: str | None) -> bool:
|
||||
line = (
|
||||
"You've been granted admin access on Siftlode — you can now manage settings, the "
|
||||
"scheduler and users."
|
||||
if role == "admin"
|
||||
else "Your admin access on Siftlode has been removed — you're now a regular user."
|
||||
def send_access_denied(email: str, lang: str) -> bool:
|
||||
admin, contact = _reply_contact(lang)
|
||||
body = mailtext.text("access_denied.body", lang, contact=contact)
|
||||
return _send([email], mailtext.text("access_denied.subject", lang), body, reply_to=admin)
|
||||
|
||||
|
||||
def send_role_changed(to: str, lang: str, role: str, operator: str | None) -> bool:
|
||||
line = mailtext.text(
|
||||
"role_changed.granted" if role == "admin" else "role_changed.revoked", lang
|
||||
)
|
||||
contact = f"\n\nQuestions? Contact the operator at {operator}." if operator else ""
|
||||
body = "Hi,\n\n" + line + contact + "\n\n— Siftlode"
|
||||
return _send([to], "Your Siftlode role has changed", body, reply_to=operator)
|
||||
contact = mailtext.text("contact.operator", lang, operator=operator) if operator else ""
|
||||
body = mailtext.text("role_changed.body", lang, line=line, contact=contact)
|
||||
return _send([to], mailtext.text("role_changed.subject", lang), body, reply_to=operator)
|
||||
|
||||
|
||||
def send_account_unsuspended(to: str, operator: str | None) -> bool:
|
||||
contact = f"\n\nQuestions? Contact the operator at {operator}." if operator else ""
|
||||
body = (
|
||||
"Hi,\n\n"
|
||||
"Your Siftlode account has been reinstated — the suspension was lifted and you can sign "
|
||||
"in again." + contact + "\n\n— Siftlode"
|
||||
)
|
||||
return _send([to], "Your Siftlode account has been reinstated", body, reply_to=operator)
|
||||
def send_account_unsuspended(to: str, lang: str, operator: str | None) -> bool:
|
||||
contact = mailtext.text("contact.operator", lang, operator=operator) if operator else ""
|
||||
body = mailtext.text("account_unsuspended.body", lang, contact=contact)
|
||||
return _send([to], mailtext.text("account_unsuspended.subject", lang), body, reply_to=operator)
|
||||
|
||||
|
||||
def send_account_deleted(to: str, operator: str | None) -> bool:
|
||||
def send_account_deleted(to: str, lang: str, operator: str | None) -> bool:
|
||||
contact = mailtext.text("contact.deleted", lang, operator=operator) if operator else ""
|
||||
body = mailtext.text("account_deleted.body", lang, contact=contact)
|
||||
return _send([to], mailtext.text("account_deleted.subject", lang), body, reply_to=operator)
|
||||
|
||||
|
||||
def send_account_suspended(to: str, lang: str, operator: str | None) -> bool:
|
||||
contact = (
|
||||
f"\n\nIf you didn't request this, contact the Siftlode operator at {operator}."
|
||||
mailtext.text("contact.suspended", lang, operator=operator)
|
||||
if operator
|
||||
else ""
|
||||
else mailtext.text("contact.suspended_unknown", lang)
|
||||
)
|
||||
body = (
|
||||
"Hi,\n\n"
|
||||
"Your Siftlode account and all associated data — subscriptions, tags, watch history, "
|
||||
"playlists and settings — have been permanently deleted. Nothing is retained.\n\n"
|
||||
"Thanks for having used Siftlode." + contact + "\n\n— Siftlode"
|
||||
body = mailtext.text("account_suspended.body", lang, contact=contact)
|
||||
return _send([to], mailtext.text("account_suspended.subject", lang), body, reply_to=operator)
|
||||
|
||||
|
||||
def send_admin_new_request(admins: list[str], lang: str, requester: str) -> bool:
|
||||
"""Tell the admins someone is waiting. One call per language group (see
|
||||
`routes.auth.notify_admins_of_request`) — admins are people with their own UI language, and a
|
||||
single blast to all of them would have to pick one and be wrong for the rest.
|
||||
|
||||
The link deep-links straight to the admin Access-requests page (the SPA reads ?admin=…). The
|
||||
requester address is spelled out (mail clients auto-link both the URL and the address), and the
|
||||
reply note is accurate: _send sets Reply-To to the requester, so a reply reaches THEM — not the
|
||||
sending mailbox. (The old copy said "Settings → Account", which was the wrong menu.)
|
||||
"""
|
||||
body = mailtext.text(
|
||||
"admin_new_request.body",
|
||||
lang,
|
||||
requester=requester,
|
||||
link=f"{settings.app_base}/?admin=access-requests",
|
||||
)
|
||||
return _send([to], "Your Siftlode account has been deleted", body, reply_to=operator)
|
||||
subject = mailtext.text("admin_new_request.subject", lang, requester=requester)
|
||||
return _send(admins, subject, body, reply_to=requester)
|
||||
|
||||
|
||||
def send_account_suspended(to: str, operator: str | None) -> bool:
|
||||
contact = (
|
||||
f"If you think this is a mistake or have questions, contact the Siftlode operator "
|
||||
f"at {operator}.\n\n"
|
||||
if operator
|
||||
else "If you think this is a mistake, please contact the Siftlode operator.\n\n"
|
||||
def send_verify_email(to: str, lang: str, link: str, hours: int = 24) -> bool:
|
||||
# The body says what the link's rules are. The reset mail always has; this one left the reader to
|
||||
# discover both by meeting a dead link, which is precisely the silence this epic removes.
|
||||
body = mailtext.text(
|
||||
"verify_email.body", lang, link=link, expiry=mailtext.link_expiry(hours, lang)
|
||||
)
|
||||
body = (
|
||||
"Hi,\n\n"
|
||||
"A sign-in to your Siftlode account was just attempted, but the account is currently "
|
||||
"suspended, so access was denied.\n\n" + contact + "— Siftlode"
|
||||
)
|
||||
return _send([to], "Your Siftlode account is suspended", body, reply_to=operator)
|
||||
return _send([to], mailtext.text("verify_email.subject", lang), body)
|
||||
|
||||
|
||||
def send_admin_new_request(admins: list[str], requester: str) -> bool:
|
||||
# The link deep-links straight to the admin Access-requests page (the SPA reads ?admin=…). The
|
||||
# requester address is spelled out (mail clients auto-link both the URL and the address), and the
|
||||
# reply note is accurate: _send sets Reply-To to the requester, so a reply reaches THEM — not the
|
||||
# sending mailbox. (The old copy said "Settings → Account", which was the wrong menu.)
|
||||
approve_url = f"{settings.app_base}/?admin=access-requests"
|
||||
body = (
|
||||
f"{requester} just requested access to Siftlode.\n\n"
|
||||
f"Approve or deny it here:\n{approve_url}\n"
|
||||
f"(or in Siftlode: open Users → Access requests)\n\n"
|
||||
f"Requester's email: {requester}\n"
|
||||
f"Replying to this message goes straight to the requester (it's the Reply-To address).\n"
|
||||
)
|
||||
return _send(admins, f"Siftlode access request from {requester}", body, reply_to=requester)
|
||||
def send_password_reset(to: str, lang: str, link: str) -> bool:
|
||||
body = mailtext.text("password_reset.body", lang, link=link)
|
||||
return _send([to], mailtext.text("password_reset.subject", lang), body)
|
||||
|
||||
|
||||
def send_verify_email(to: str, link: str) -> bool:
|
||||
body = (
|
||||
"Hi,\n\n"
|
||||
"Confirm your email to finish creating your Siftlode account:\n\n"
|
||||
f"{link}\n\n"
|
||||
"If you didn't sign up, you can ignore this message.\n\n"
|
||||
"— Siftlode"
|
||||
)
|
||||
return _send([to], "Confirm your Siftlode email", body)
|
||||
|
||||
|
||||
def send_password_reset(to: str, link: str) -> bool:
|
||||
body = (
|
||||
"Hi,\n\n"
|
||||
"Use this link to set a new Siftlode password (it expires in an hour):\n\n"
|
||||
f"{link}\n\n"
|
||||
"If you didn't request this, you can ignore this message — your password is unchanged.\n\n"
|
||||
"— Siftlode"
|
||||
)
|
||||
return _send([to], "Reset your Siftlode password", body)
|
||||
|
||||
|
||||
def send_test(to: str) -> bool:
|
||||
body = (
|
||||
"This is a test email from Siftlode.\n\n"
|
||||
"If you're reading this, your SMTP settings work.\n\n"
|
||||
"— Siftlode"
|
||||
)
|
||||
return _send([to], "Siftlode SMTP test", body)
|
||||
def send_test(to: str, lang: str) -> bool:
|
||||
return _send([to], mailtext.text("test.subject", lang), mailtext.text("test.body", lang))
|
||||
|
||||
@@ -117,6 +117,19 @@ def tags_by_category(db: Session, tag_ids: list[int]) -> dict[str, list[int]]:
|
||||
return out
|
||||
|
||||
|
||||
def not_blocked_cond(user: User):
|
||||
"""Per-user channel blocklist: never show this user videos from a channel they blocked
|
||||
(applies to feed, Library, explore, the channel page and the Home resume rail alike).
|
||||
|
||||
A shared expression rather than a clause each surface writes for itself. Blocking does not touch
|
||||
`video_states`, so a half-watched video from a newly blocked channel survives with its resume
|
||||
position — and the one query that copied only PART of this predicate set is exactly how that
|
||||
video came back as the landing page's suggestion after the user had said "never again"."""
|
||||
return Video.channel_id.notin_(
|
||||
select(BlockedChannel.channel_id).where(BlockedChannel.user_id == user.id)
|
||||
)
|
||||
|
||||
|
||||
def filtered_query(
|
||||
db: Session,
|
||||
user: User,
|
||||
@@ -207,13 +220,7 @@ def filtered_query(
|
||||
)
|
||||
)
|
||||
|
||||
# Per-user channel blocklist: never show this user videos from a channel they blocked
|
||||
# (applies to feed, Library, explore, and the channel page alike).
|
||||
query = query.where(
|
||||
Video.channel_id.notin_(
|
||||
select(BlockedChannel.channel_id).where(BlockedChannel.user_id == user.id)
|
||||
)
|
||||
)
|
||||
query = query.where(not_blocked_cond(user))
|
||||
|
||||
# Source filter. In the shared Library ("all" scope) it uses the GLOBAL via_search flag
|
||||
# (search-discovered by anyone). In "my" scope it's the user's OWN provenance:
|
||||
|
||||
@@ -0,0 +1,373 @@
|
||||
"""Bilingual text for every outbound mail, and the one resolver that decides which language a
|
||||
recipient gets (E9.5 S7).
|
||||
|
||||
Why a catalog instead of literals in `email.py`: HU/EN parity is a house rule for every user-facing
|
||||
string, and a mail is the one user-facing surface the reader CANNOT re-render by switching the UI
|
||||
language — it arrives once, in whatever language we picked, and stays that way. `test_mail_language.py`
|
||||
enforces that every key carries both languages *and* the same `{placeholders}`, so a half-translated
|
||||
mail fails the gate instead of reaching a person.
|
||||
|
||||
The language is resolved by the REQUEST handler and handed to `send_*` as a plain code. It is never
|
||||
resolved inside the mail: every send runs as a `BackgroundTasks` job, after the request's session is
|
||||
gone — the same reason `_welcome_mail_parts` computes its link in the handler and not in the task.
|
||||
"""
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import Invite, User
|
||||
from app.utils import canonical_email
|
||||
|
||||
SUPPORTED: tuple[str, ...] = ("en", "hu")
|
||||
DEFAULT = "en"
|
||||
|
||||
|
||||
def normalize(lang: str | None) -> str:
|
||||
"""Coerce anything to a language we can actually render. Storage is not a guarantee: a
|
||||
`preferences.language` written by an older build (or a locale we no longer ship) must degrade to
|
||||
English rather than raise inside a background task nobody is watching."""
|
||||
return lang if lang in SUPPORTED else DEFAULT
|
||||
|
||||
|
||||
def text(key: str, lang: str | None, **values: object) -> str:
|
||||
"""Render one catalog entry. Unknown languages fall back to English (see `normalize`); an
|
||||
unknown KEY raises, because that is a programming error the tests must surface, not something to
|
||||
paper over with an empty line in someone's inbox."""
|
||||
return MAIL_TEXT[key][normalize(lang)].format(**values)
|
||||
|
||||
|
||||
def link_expiry(hours: int, lang: str) -> str:
|
||||
"""How long a link lasts, in a unit a reader converts on sight. A whole number of days is
|
||||
said in days — the approval mail's token lives seven of them, and "168 hours" is not an answer
|
||||
to "how long have I got". A single day stays "24 hours", which is how everyone writes it."""
|
||||
if hours >= 48 and hours % 24 == 0:
|
||||
return text("duration.days", lang, n=hours // 24)
|
||||
return text("duration.hours", lang, n=hours)
|
||||
|
||||
|
||||
def lang_for_user(user: User | None) -> str:
|
||||
"""The language a signed-up recipient gets: the one they picked in the app."""
|
||||
prefs = (user.preferences or {}) if user is not None else {}
|
||||
return normalize(prefs.get("language"))
|
||||
|
||||
|
||||
def lang_for_email(db: Session, email: str) -> str:
|
||||
"""The language for an address that may not have an account yet: their own account preference,
|
||||
else the language they filled our access-request form in, else English.
|
||||
|
||||
Matched CANONICALLY like every other identity lookup in this epic, and it walks past rows that
|
||||
carry no usable code instead of stopping at the first one — prod really does hold two accounts
|
||||
for a single mailbox, and only one of them may have ever picked a language.
|
||||
"""
|
||||
canon = canonical_email(email)
|
||||
for prefs in (
|
||||
db.execute(select(User.preferences).where(User.email_canonical == canon).order_by(User.id))
|
||||
.scalars()
|
||||
.all()
|
||||
):
|
||||
code = (prefs or {}).get("language")
|
||||
if code in SUPPORTED:
|
||||
return code
|
||||
for code in (
|
||||
db.execute(select(Invite.lang).where(Invite.email_canonical == canon).order_by(Invite.id))
|
||||
.scalars()
|
||||
.all()
|
||||
):
|
||||
if code in SUPPORTED:
|
||||
return code
|
||||
return DEFAULT
|
||||
|
||||
|
||||
# The brand line ("— Siftlode") and the product name are deliberately NOT catalog entries: they are
|
||||
# a name, not a translated string, and giving them keys would invite someone to "translate" them.
|
||||
MAIL_TEXT: dict[str, dict[str, str]] = {
|
||||
# ── How long a link lasts ───────────────────────────────────────────────────────────────
|
||||
# Rendered, not interpolated as a raw hour count: the approval mail's token lives seven days
|
||||
# (E9.5 S6), and "expires in 168 hours" is a unit no reader converts.
|
||||
"duration.hours": {"en": "{n} hours", "hu": "{n} óra"},
|
||||
"duration.days": {"en": "{n} days", "hu": "{n} nap"},
|
||||
# ── Shared closing fragments ────────────────────────────────────────────────────────────────
|
||||
"contact.reply": {
|
||||
"en": "\n\nQuestions? Just reply to this email ({admin}).",
|
||||
"hu": "\n\nKérdésed van? Csak válaszolj erre a levélre ({admin}).",
|
||||
},
|
||||
"contact.operator": {
|
||||
"en": "\n\nQuestions? Contact the operator at {operator}.",
|
||||
"hu": "\n\nKérdésed van? Írj az üzemeltetőnek: {operator}.",
|
||||
},
|
||||
"contact.deleted": {
|
||||
"en": "\n\nIf you didn't request this, contact the Siftlode operator at {operator}.",
|
||||
"hu": "\n\nHa nem te kérted, írj a Siftlode üzemeltetőjének: {operator}.",
|
||||
},
|
||||
"contact.suspended": {
|
||||
"en": (
|
||||
"If you think this is a mistake or have questions, contact the Siftlode operator "
|
||||
"at {operator}.\n\n"
|
||||
),
|
||||
"hu": (
|
||||
"Ha szerinted ez tévedés, vagy kérdésed van, írj a Siftlode üzemeltetőjének: "
|
||||
"{operator}.\n\n"
|
||||
),
|
||||
},
|
||||
"contact.suspended_unknown": {
|
||||
"en": "If you think this is a mistake, please contact the Siftlode operator.\n\n",
|
||||
"hu": "Ha szerinted ez tévedés, keresd a Siftlode üzemeltetőjét.\n\n",
|
||||
},
|
||||
# ── The one paragraph that carries the whole onboarding flow (see email._next_step) ──────────
|
||||
"next_step.verify": {
|
||||
"en": (
|
||||
"You already started an account, so there is just one step left. Confirm your email "
|
||||
"address and you're in:\n\n"
|
||||
"{link}\n\n"
|
||||
"The link works once and expires in {expiry}. If it does, you can ask for a fresh "
|
||||
"one from the sign-in page."
|
||||
),
|
||||
"hu": (
|
||||
"A fiókodat már elkezdted, úgyhogy egyetlen lépés maradt. Erősítsd meg az "
|
||||
"e-mail-címed, és bent is vagy:\n\n"
|
||||
"{link}\n\n"
|
||||
"A link egyszer használható, és {expiry} múlva lejár. Ha lejárt, a belépési oldalon "
|
||||
"kérhetsz újat."
|
||||
),
|
||||
},
|
||||
"next_step.sign_in": {
|
||||
"en": "Your account is ready — just sign in:\n\n{app}",
|
||||
"hu": "A fiókod készen áll — csak lépj be:\n\n{app}",
|
||||
},
|
||||
"next_step.register": {
|
||||
"en": (
|
||||
"Create your account whenever you feel like it:\n\n"
|
||||
"{app}\n\n"
|
||||
"We'll send you a confirmation link to finish, and then you're in."
|
||||
),
|
||||
"hu": (
|
||||
"Hozd létre a fiókod, amikor kedved tartja:\n\n"
|
||||
"{app}\n\n"
|
||||
"Küldünk egy megerősítő linket a befejezéshez, és már bent is vagy."
|
||||
),
|
||||
},
|
||||
# ── Access decision ─────────────────────────────────────────────────────────────────────────
|
||||
"access_approved.subject": {
|
||||
"en": "You're in — your Siftlode access is approved",
|
||||
"hu": "Bent vagy — jóváhagytuk a Siftlode-hozzáférésed",
|
||||
},
|
||||
"access_approved.body": {
|
||||
"en": (
|
||||
"Hi,\n\n"
|
||||
"Your request to use Siftlode has been approved — welcome aboard.\n\n"
|
||||
"{next_step}\n\n"
|
||||
"If you weren't expecting this, you can ignore the message.\n\n"
|
||||
"— Siftlode{contact}"
|
||||
),
|
||||
"hu": (
|
||||
"Szia,\n\n"
|
||||
"Jóváhagytuk a Siftlode-hozzáférési kérelmed — üdv a fedélzeten.\n\n"
|
||||
"{next_step}\n\n"
|
||||
"Ha nem számítottál erre, nyugodtan hagyd figyelmen kívül ezt a levelet.\n\n"
|
||||
"— Siftlode{contact}"
|
||||
),
|
||||
},
|
||||
"access_invitation.subject": {
|
||||
"en": "You're invited to Siftlode",
|
||||
"hu": "Meghívtak a Siftlode-ra",
|
||||
},
|
||||
"access_invitation.body": {
|
||||
"en": (
|
||||
"Hi,\n\n"
|
||||
"Someone has opened up Siftlode for you — a quiet, self-hosted reader for your YouTube "
|
||||
"subscriptions. Every upload from the channels you follow lands in one feed you can "
|
||||
"actually filter and sort, with no algorithm deciding what you see, and no Shorts or "
|
||||
"livestreams unless you want them.\n\n"
|
||||
"{next_step}\n\n"
|
||||
"No rush — the door stays open.\n\n"
|
||||
"— Siftlode{contact}"
|
||||
),
|
||||
"hu": (
|
||||
"Szia,\n\n"
|
||||
"Valaki megnyitotta előtted a Siftlode-ot — ez egy csendes, saját üzemeltetésű olvasó "
|
||||
"a YouTube-feliratkozásaidhoz. A követett csatornák minden feltöltése egyetlen "
|
||||
"hírfolyamba érkezik, amit tényleg tudsz szűrni és rendezni; nincs algoritmus, ami "
|
||||
"eldönti, mit látsz, és nincs Shorts vagy élő adás, hacsak nem kéred.\n\n"
|
||||
"{next_step}\n\n"
|
||||
"Nem sürget semmi — az ajtó nyitva marad.\n\n"
|
||||
"— Siftlode{contact}"
|
||||
),
|
||||
},
|
||||
"access_denied.subject": {
|
||||
"en": "About your Siftlode access request",
|
||||
"hu": "A Siftlode-hozzáférési kérelmedről",
|
||||
},
|
||||
"access_denied.body": {
|
||||
"en": (
|
||||
"Hi,\n\n"
|
||||
"Thanks for your interest in Siftlode. After review, your access request wasn't "
|
||||
"approved this time.\n\n"
|
||||
"If you think this was a mistake, you're welcome to reply and let us know.\n\n"
|
||||
"— Siftlode{contact}"
|
||||
),
|
||||
"hu": (
|
||||
"Szia,\n\n"
|
||||
"Köszönjük az érdeklődésed a Siftlode iránt. Átnéztük a kérelmed, de most nem hagytuk "
|
||||
"jóvá.\n\n"
|
||||
"Ha szerinted ez tévedés, válaszolj nyugodtan erre a levélre.\n\n"
|
||||
"— Siftlode{contact}"
|
||||
),
|
||||
},
|
||||
# ── Account lifecycle ───────────────────────────────────────────────────────────────────────
|
||||
"role_changed.subject": {
|
||||
"en": "Your Siftlode role has changed",
|
||||
"hu": "Megváltozott a szereped a Siftlode-on",
|
||||
},
|
||||
"role_changed.granted": {
|
||||
"en": (
|
||||
"You've been granted admin access on Siftlode — you can now manage settings, the "
|
||||
"scheduler and users."
|
||||
),
|
||||
"hu": (
|
||||
"Admin jogot kaptál a Siftlode-on — mostantól kezelheted a beállításokat, az ütemezőt "
|
||||
"és a felhasználókat."
|
||||
),
|
||||
},
|
||||
"role_changed.revoked": {
|
||||
"en": "Your admin access on Siftlode has been removed — you're now a regular user.",
|
||||
"hu": "Elvettük az admin jogod a Siftlode-on — mostantól sima felhasználó vagy.",
|
||||
},
|
||||
"role_changed.body": {
|
||||
"en": "Hi,\n\n{line}{contact}\n\n— Siftlode",
|
||||
"hu": "Szia,\n\n{line}{contact}\n\n— Siftlode",
|
||||
},
|
||||
"account_unsuspended.subject": {
|
||||
"en": "Your Siftlode account has been reinstated",
|
||||
"hu": "Visszaállítottuk a Siftlode-fiókod",
|
||||
},
|
||||
"account_unsuspended.body": {
|
||||
"en": (
|
||||
"Hi,\n\n"
|
||||
"Your Siftlode account has been reinstated — the suspension was lifted and you can "
|
||||
"sign in again.{contact}\n\n— Siftlode"
|
||||
),
|
||||
"hu": (
|
||||
"Szia,\n\n"
|
||||
"Visszaállítottuk a Siftlode-fiókod — a felfüggesztést feloldottuk, újra be tudsz "
|
||||
"lépni.{contact}\n\n— Siftlode"
|
||||
),
|
||||
},
|
||||
"account_deleted.subject": {
|
||||
"en": "Your Siftlode account has been deleted",
|
||||
"hu": "Töröltük a Siftlode-fiókod",
|
||||
},
|
||||
"account_deleted.body": {
|
||||
"en": (
|
||||
"Hi,\n\n"
|
||||
"Your Siftlode account and all associated data — subscriptions, tags, watch history, "
|
||||
"playlists and settings — have been permanently deleted. Nothing is retained.\n\n"
|
||||
"Thanks for having used Siftlode.{contact}\n\n— Siftlode"
|
||||
),
|
||||
"hu": (
|
||||
"Szia,\n\n"
|
||||
"A Siftlode-fiókod és minden hozzá tartozó adat — feliratkozások, címkék, megtekintési "
|
||||
"előzmények, lejátszási listák és beállítások — véglegesen törölve. Semmit nem őrzünk "
|
||||
"meg.\n\n"
|
||||
"Köszönjük, hogy használtad a Siftlode-ot.{contact}\n\n— Siftlode"
|
||||
),
|
||||
},
|
||||
"account_suspended.subject": {
|
||||
"en": "Your Siftlode account is suspended",
|
||||
"hu": "A Siftlode-fiókod fel van függesztve",
|
||||
},
|
||||
"account_suspended.body": {
|
||||
"en": (
|
||||
"Hi,\n\n"
|
||||
"A sign-in to your Siftlode account was just attempted, but the account is currently "
|
||||
"suspended, so access was denied.\n\n{contact}— Siftlode"
|
||||
),
|
||||
"hu": (
|
||||
"Szia,\n\n"
|
||||
"Valaki most próbált belépni a Siftlode-fiókodba, de a fiók jelenleg fel van "
|
||||
"függesztve, ezért a belépést elutasítottuk.\n\n{contact}— Siftlode"
|
||||
),
|
||||
},
|
||||
# ── To the admins ───────────────────────────────────────────────────────────────────────────
|
||||
"admin_new_request.subject": {
|
||||
"en": "Siftlode access request from {requester}",
|
||||
"hu": "Siftlode hozzáférési kérelem: {requester}",
|
||||
},
|
||||
"admin_new_request.body": {
|
||||
"en": (
|
||||
"{requester} just requested access to Siftlode.\n\n"
|
||||
"Approve or deny it here:\n{link}\n"
|
||||
"(or in Siftlode: open Users → Access requests)\n\n"
|
||||
"Requester's email: {requester}\n"
|
||||
"Replying to this message goes straight to the requester (it's the Reply-To address).\n"
|
||||
),
|
||||
"hu": (
|
||||
"{requester} most kért hozzáférést a Siftlode-hoz.\n\n"
|
||||
"Itt hagyhatod jóvá vagy utasíthatod el:\n{link}\n"
|
||||
"(vagy a Siftlode-ban: Felhasználók → Hozzáférési kérések)\n\n"
|
||||
"A kérelmező e-mail-címe: {requester}\n"
|
||||
"Ha erre a levélre válaszolsz, az egyenesen a kérelmezőhöz megy (ő a Reply-To cím).\n"
|
||||
),
|
||||
},
|
||||
# ── Link mails ──────────────────────────────────────────────────────────────────────────────
|
||||
"verify_email.subject": {
|
||||
"en": "Confirm your Siftlode email",
|
||||
"hu": "Erősítsd meg a Siftlode e-mail-címed",
|
||||
},
|
||||
"verify_email.body": {
|
||||
"en": (
|
||||
"Hi,\n\n"
|
||||
"Confirm your email to finish creating your Siftlode account:\n\n"
|
||||
"{link}\n\n"
|
||||
"The link works once and expires in {expiry}. If it does, you can ask for a fresh "
|
||||
"one from the sign-in page.\n\n"
|
||||
"If you didn't sign up, you can ignore this message.\n\n"
|
||||
"— Siftlode"
|
||||
),
|
||||
"hu": (
|
||||
"Szia,\n\n"
|
||||
"Erősítsd meg az e-mail-címed, hogy elkészüljön a Siftlode-fiókod:\n\n"
|
||||
"{link}\n\n"
|
||||
"A link egyszer használható, és {expiry} múlva lejár. Ha lejárt, a belépési oldalon "
|
||||
"kérhetsz újat.\n\n"
|
||||
"Ha nem te regisztráltál, nyugodtan hagyd figyelmen kívül ezt a levelet.\n\n"
|
||||
"— Siftlode"
|
||||
),
|
||||
},
|
||||
"password_reset.subject": {
|
||||
"en": "Reset your Siftlode password",
|
||||
"hu": "Állíts be új Siftlode-jelszót",
|
||||
},
|
||||
"password_reset.body": {
|
||||
"en": (
|
||||
"Hi,\n\n"
|
||||
"Use this link to set a new Siftlode password (it expires in an hour):\n\n"
|
||||
"{link}\n\n"
|
||||
"If you didn't request this, you can ignore this message — your password is "
|
||||
"unchanged.\n\n"
|
||||
"— Siftlode"
|
||||
),
|
||||
"hu": (
|
||||
"Szia,\n\n"
|
||||
"Ezzel a linkkel állíthatsz be új Siftlode-jelszót (egy óra múlva lejár):\n\n"
|
||||
"{link}\n\n"
|
||||
"Ha nem te kérted, hagyd figyelmen kívül ezt a levelet — a jelszavad változatlan.\n\n"
|
||||
"— Siftlode"
|
||||
),
|
||||
},
|
||||
"test.subject": {
|
||||
"en": "Siftlode SMTP test",
|
||||
"hu": "Siftlode SMTP-teszt",
|
||||
},
|
||||
"test.body": {
|
||||
"en": (
|
||||
"This is a test email from Siftlode.\n\n"
|
||||
"If you're reading this, your SMTP settings work.\n\n"
|
||||
"— Siftlode"
|
||||
),
|
||||
"hu": (
|
||||
"Ez egy teszt e-mail a Siftlode-tól.\n\n"
|
||||
"Ha ezt olvasod, az SMTP-beállításaid működnek.\n\n"
|
||||
"— Siftlode"
|
||||
),
|
||||
},
|
||||
}
|
||||
+66
-4
@@ -1,10 +1,13 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import math
|
||||
import sys
|
||||
from contextlib import asynccontextmanager, suppress
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.encoders import jsonable_encoder
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
|
||||
# When started via uvicorn --log-config the "siftlode" logger is already configured
|
||||
# (see log_config.json). This block is a timestamped fallback for other entrypoints
|
||||
@@ -21,18 +24,21 @@ log = logging.getLogger("siftlode.app")
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from sqlalchemy.exc import DataError
|
||||
from starlette.middleware.sessions import SessionMiddleware
|
||||
|
||||
from app import state
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.plex import stream as plex_stream
|
||||
from app.plex import subtitles as plex_subtitles
|
||||
from app.routes import (
|
||||
admin,
|
||||
audit as audit_routes,
|
||||
auth as auth_routes,
|
||||
channels,
|
||||
config as config_routes,
|
||||
discover as discover_routes,
|
||||
downloads,
|
||||
feed,
|
||||
health,
|
||||
@@ -76,6 +82,7 @@ async def lifespan(app: FastAPI):
|
||||
# alone doesn't buy that; not waiting does. The sweep only touches directories no live session
|
||||
# owns, so racing it against early playback is safe.
|
||||
sweep = asyncio.create_task(_sweep_hls_orphans())
|
||||
reaper = asyncio.create_task(_reap_hls_sessions())
|
||||
start_scheduler()
|
||||
try:
|
||||
yield
|
||||
@@ -85,10 +92,36 @@ async def lifespan(app: FastAPI):
|
||||
# pending ("Task was destroyed but it is pending!" — noise that reads like a bug during
|
||||
# triage). Cancelling does NOT interrupt the rmtree thread itself; this only makes our own
|
||||
# bookkeeping honest.
|
||||
sweep.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await sweep
|
||||
for task in (sweep, reaper):
|
||||
task.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await task
|
||||
shutdown_scheduler()
|
||||
# Kills a running subtitle extraction: its worker would otherwise hold interpreter exit for
|
||||
# the rest of the read (see plex_subtitles.shutdown).
|
||||
plex_subtitles.shutdown()
|
||||
|
||||
|
||||
# How often idle HLS sessions are reaped (see plex/stream.py `reap_idle`).
|
||||
_HLS_REAP_EVERY_S = 60.0
|
||||
|
||||
|
||||
async def _reap_hls_sessions() -> None:
|
||||
"""The periodic reaper plex/stream.py's docstring promises. `reap_idle` used to run only inside
|
||||
the NEXT session request, so after the last playback of the day an abandoned session — its
|
||||
ffmpeg still working through the film, its directory a whole segmented copy of it — lived until
|
||||
somebody pressed play again or the process restarted.
|
||||
|
||||
A FINISHED session is reaped on the short rule here, not kept for the full idle window: an open
|
||||
player checkpoints every 10 s and that route touches the session, so a paused viewer is no longer
|
||||
invisible to this pass (see `reap_idle`). Keeping every finished session for ten minutes instead
|
||||
held one whole segmented copy of a film per item on the download volume (round 5)."""
|
||||
while True:
|
||||
await asyncio.sleep(_HLS_REAP_EVERY_S)
|
||||
try:
|
||||
await asyncio.to_thread(plex_stream.reap_idle)
|
||||
except Exception:
|
||||
log.exception("HLS session reap failed")
|
||||
|
||||
|
||||
async def _sweep_hls_orphans() -> None:
|
||||
@@ -107,6 +140,34 @@ async def _sweep_hls_orphans() -> None:
|
||||
|
||||
app = FastAPI(title=settings.app_name, lifespan=lifespan)
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def _validation_error(request: Request, exc: RequestValidationError) -> JSONResponse:
|
||||
"""FastAPI's own 422, in its own shape — except that it can be ENCODED. The JSON parser accepts
|
||||
`NaN`/`Infinity`, a float field refuses them, and the refusal echoes the value back: the default
|
||||
handler then failed to serialise its own error (JSON has no NaN) and the request became a 500
|
||||
with a traceback. A non-finite value is echoed as its name instead."""
|
||||
finite = {float: lambda x: x if math.isfinite(x) else str(x)}
|
||||
return JSONResponse(
|
||||
status_code=422, content={"detail": jsonable_encoder(exc.errors(), custom_encoder=finite)}
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(DataError)
|
||||
async def _nul_in_text(request: Request, exc: DataError) -> JSONResponse:
|
||||
"""A NUL byte in a string the request sent is a 422, on every route at once. Postgres cannot hold
|
||||
NUL in text, so the driver refuses to even send the query — and a `%00` in a query parameter or
|
||||
an escaped NUL in a JSON body (a playlist's `contains=`, a reorder's key list, a collection
|
||||
search) was a 500 with a traceback wherever it reached a comparison. Guarding each site did not
|
||||
hold: round 3 guarded the by-rating_key lookup and missed the rest. Any other DataError is left
|
||||
alone — re-raised, it stays the 500 it always was."""
|
||||
if "NUL (0x00)" not in str(exc.orig):
|
||||
raise exc
|
||||
return JSONResponse(
|
||||
status_code=422, content={"detail": {"code": "invalid_input", "reason": "nul"}}
|
||||
)
|
||||
|
||||
|
||||
app.add_middleware(
|
||||
SessionMiddleware,
|
||||
secret_key=settings.secret_key,
|
||||
@@ -187,6 +248,7 @@ app.include_router(auth_routes.router)
|
||||
app.include_router(sync.router)
|
||||
app.include_router(tags.router)
|
||||
app.include_router(feed.router)
|
||||
app.include_router(discover_routes.router)
|
||||
app.include_router(search_routes.router)
|
||||
app.include_router(me.router)
|
||||
app.include_router(notifications.router)
|
||||
|
||||
+123
-3
@@ -3,6 +3,7 @@ from datetime import date, datetime
|
||||
from sqlalchemy import (
|
||||
BigInteger,
|
||||
Boolean,
|
||||
CheckConstraint,
|
||||
Computed,
|
||||
Date,
|
||||
DateTime,
|
||||
@@ -17,9 +18,15 @@ from sqlalchemy import (
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
|
||||
|
||||
from app.db import Base
|
||||
from app.utils import canonical_email
|
||||
|
||||
|
||||
# The resume-position columns (VideoState / PlexState.position_seconds) are int4. The checkpoint
|
||||
# routes bound their bodies by this, so an out-of-range position is a 422 rather than a 500.
|
||||
POSITION_MAX_S = 2**31 - 1
|
||||
|
||||
|
||||
class TimestampMixin:
|
||||
@@ -29,6 +36,21 @@ class TimestampMixin:
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class CanonicalEmailMixin:
|
||||
"""Keeps `email_canonical` derived from `email` on every write.
|
||||
|
||||
The invariant — "identity is matched on the canonical form, mail is sent to the typed one" —
|
||||
lives HERE, in code, instead of in a comment every write site has to remember. That distinction
|
||||
is what the 2026-09-03 lockout was made of: one gate lowercased, nothing else did, and an
|
||||
approved user's Google sign-in silently became a second access request.
|
||||
"""
|
||||
|
||||
@validates("email")
|
||||
def _derive_email_canonical(self, _key: str, value: str | None) -> str | None:
|
||||
self.email_canonical = canonical_email(value)
|
||||
return value
|
||||
|
||||
|
||||
class UpdatedAtMixin:
|
||||
"""Adds an `updated_at` that server-defaults on insert and bumps on every UPDATE."""
|
||||
|
||||
@@ -37,7 +59,7 @@ class UpdatedAtMixin:
|
||||
)
|
||||
|
||||
|
||||
class User(Base, TimestampMixin):
|
||||
class User(Base, TimestampMixin, CanonicalEmailMixin):
|
||||
__tablename__ = "users"
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
@@ -45,8 +67,26 @@ class User(Base, TimestampMixin):
|
||||
# multiple NULLs under a unique index). Set on Google sign-in / in-app linking.
|
||||
google_sub: Mapped[str | None] = mapped_column(String(64), unique=True, index=True)
|
||||
email: Mapped[str] = mapped_column(String(320), unique=True, index=True)
|
||||
# The same mailbox, in the form used to MATCH identities (utils.canonical_email) — Gmail dots and
|
||||
# `+tags` folded away. Kept by the validator below so no write site can forget it. Deliberately
|
||||
# NOT unique: prod already carries two rows for one mailbox (a password registration and the
|
||||
# Google sign-in that failed to find it), and a migration must not merge accounts on its own.
|
||||
email_canonical: Mapped[str] = mapped_column(String(320), index=True, server_default="")
|
||||
# The address Google asserted for this account, recorded at sign-in / link time. NOT the same
|
||||
# thing as `email`: since E9.5 S3 a Google sign-in no longer overwrites the address the user
|
||||
# registered with, so a deliberate `me+siftlode@gmail.com` survives — and this is where the other
|
||||
# half of the truth lives, for the UI to show when the two differ. NULL until the next Google
|
||||
# sign-in for accounts linked before this column existed.
|
||||
google_email: Mapped[str | None] = mapped_column(String(320))
|
||||
# argon2id hash for email+password login; NULL for Google-only / demo accounts.
|
||||
password_hash: Mapped[str | None] = mapped_column(String(255))
|
||||
# When SEC-10 dropped a preset password on Google adoption (see the single caller of
|
||||
# auth.revoke_preset_password_if_unverified; admin approval no longer revokes — E9.5 S1). Without
|
||||
# this trace a revoked account was indistinguishable from a Google-only one, so it failed the
|
||||
# password-reset eligibility check and could never be recovered — it could neither sign in nor
|
||||
# reset. Set = "had a password, lost it to the guard" -> reset-eligible. Cleared whenever a
|
||||
# password is set again (reset-confirm / set-password).
|
||||
password_revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# Email ownership proven (Google sign-in is implicitly verified; password registration
|
||||
# verifies via a link). Login requires this AND is_active.
|
||||
email_verified: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
|
||||
@@ -114,7 +154,7 @@ class AuthToken(Base, TimestampMixin):
|
||||
used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
|
||||
class Invite(Base):
|
||||
class Invite(Base, CanonicalEmailMixin):
|
||||
"""Access-request / whitelist row. Source of truth for who may sign in; the env
|
||||
ALLOWED_EMAILS/ADMIN_EMAILS act only as a bootstrap fallback (see auth.is_allowed)."""
|
||||
|
||||
@@ -122,6 +162,9 @@ class Invite(Base):
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
email: Mapped[str] = mapped_column(String(320), unique=True, index=True)
|
||||
# See User.email_canonical. Not unique here either: prod has approved invites for both spellings
|
||||
# of the same mailbox, dating back to 2026-06-16.
|
||||
email_canonical: Mapped[str] = mapped_column(String(320), index=True, server_default="")
|
||||
# pending | approved | denied
|
||||
status: Mapped[str] = mapped_column(String(16), default="pending", server_default="pending")
|
||||
note: Mapped[str | None] = mapped_column(Text)
|
||||
@@ -130,6 +173,17 @@ class Invite(Base):
|
||||
)
|
||||
decided_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
decided_by: Mapped[str | None] = mapped_column(String(320)) # admin email
|
||||
# When the admin last SENT this person an invitation from the whitelist row. Adding an address
|
||||
# stays silent — it opens a door, it does not knock — so telling them is a separate, deliberate
|
||||
# act, and this is what stops it happening twice to a real person.
|
||||
invited_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# The language this person asked us in ("en"/"hu"), captured when the request was made: the page
|
||||
# their form was on, or the locale Google reported for a refused sign-in. NULL for a row the
|
||||
# admin whitelisted by hand — nobody has told us anything about that mailbox, which is exactly
|
||||
# why the invitation send asks (E9.5 S7). Captured rather than derived for the same reason
|
||||
# `preferences.language` and `google_email` are: the signal exists only in that one moment and
|
||||
# cannot be backfilled afterwards.
|
||||
lang: Mapped[str | None] = mapped_column(String(8))
|
||||
|
||||
|
||||
class DemoWhitelist(Base, TimestampMixin):
|
||||
@@ -374,6 +428,7 @@ class VideoState(Base, UpdatedAtMixin):
|
||||
# status: a partially-watched video keeps status "new" but a non-zero position, so
|
||||
# it can render a progress bar and match the "in progress" feed filter. 0 = not
|
||||
# started / finished (trivially-early and near-finished positions are not stored).
|
||||
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
|
||||
position_seconds: Mapped[int] = mapped_column(
|
||||
Integer, default=0, server_default="0", nullable=False
|
||||
)
|
||||
@@ -1127,6 +1182,22 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
|
||||
# Browser-playability class from ffprobe: "direct" | "remux" | "transcode" (None = unprobed).
|
||||
playable: Mapped[str | None] = mapped_column(String(12), index=True)
|
||||
probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# Transcode HW-capability facet (E7 playback-class filter). NULL for non-transcode items; for a
|
||||
# `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
|
||||
# AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
|
||||
# breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
|
||||
# set codec-only at sync (bit depth unknown from the cheap listing), then refined by the bit-depth
|
||||
# backfill (`_probe_media_batch`) which downgrades a 10-bit/HDR HEVC to False. The one filterable
|
||||
# home of the invariant `_want_hw`'s source gate computes at play time — kept in sync via the shared
|
||||
# classifier so the catalog facet and the runtime encoder choice never disagree by codec.
|
||||
transcode_hw: Mapped[bool | None] = mapped_column(Boolean, index=True)
|
||||
# Video bit depth (8/10/12…) from the Plex media Stream, filled by the bit-depth backfill. NULL =
|
||||
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw`.
|
||||
video_bit_depth: Mapped[int | None] = mapped_column(Integer)
|
||||
# Watermark for the bit-depth backfill: when this item's media Stream was last probed for
|
||||
# bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are
|
||||
# probed (the rest are SW-only by codec and need no bit-depth); see `_probe_media_batch`.
|
||||
media_probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# Intro/credit markers: [{"type": "intro"|"credits", "start_s": int, "end_s": int}, ...]
|
||||
markers: Mapped[list | None] = mapped_column(JSON)
|
||||
added_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
|
||||
@@ -1342,3 +1413,52 @@ class PlexPlaylistItem(Base):
|
||||
ForeignKey("plex_items.id", ondelete="CASCADE"), index=True
|
||||
)
|
||||
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0", nullable=False)
|
||||
|
||||
|
||||
class ComfortPin(Base, TimestampMixin):
|
||||
"""One entry on a user's comfort shelf — the anti-paralysis fallback: "just put on X" (E24).
|
||||
|
||||
Deliberately NOT a boolean on `VideoState`/`PlexState`, and NOT a polymorphic `(kind, ref)` string
|
||||
pair. A flag cannot ORDER a shelf and cannot hold a channel or a show — and comfort media is
|
||||
usually "put on X", a series or a channel, not one particular video. A string ref would order and
|
||||
hold anything, but it would also lose the foreign key: a Plex item dropped from the library, or an
|
||||
unsubscribed channel, would leave a dead tile on the one surface whose whole job is to be
|
||||
reliable. So instead: one nullable CASCADE-ing FK per pinnable kind, and a check constraint that
|
||||
exactly one of them is set.
|
||||
|
||||
That constraint is the house rule applied to a five-way union — the invariant lives in the schema,
|
||||
not in a comment every insert site has to remember. `kind`/`ref` below derive FROM the columns, so
|
||||
there is no second vocabulary to keep in sync.
|
||||
|
||||
NULLs are DISTINCT in a Postgres unique index, so the five per-kind unique constraints each stop a
|
||||
double-pin of the same thing without colliding across kinds."""
|
||||
|
||||
__tablename__ = "comfort_pins"
|
||||
__table_args__ = (
|
||||
CheckConstraint(
|
||||
"num_nonnulls(video_id, channel_id, plex_item_id, plex_show_id, playlist_id) = 1",
|
||||
name="ck_comfort_pin_one_ref",
|
||||
),
|
||||
UniqueConstraint("user_id", "video_id", name="uq_comfort_pin_video"),
|
||||
UniqueConstraint("user_id", "channel_id", name="uq_comfort_pin_channel"),
|
||||
UniqueConstraint("user_id", "plex_item_id", name="uq_comfort_pin_plex_item"),
|
||||
UniqueConstraint("user_id", "plex_show_id", name="uq_comfort_pin_plex_show"),
|
||||
UniqueConstraint("user_id", "playlist_id", name="uq_comfort_pin_playlist"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
|
||||
|
||||
video_id: Mapped[str | None] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"))
|
||||
channel_id: Mapped[str | None] = mapped_column(ForeignKey("channels.id", ondelete="CASCADE"))
|
||||
plex_item_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("plex_items.id", ondelete="CASCADE")
|
||||
)
|
||||
plex_show_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("plex_shows.id", ondelete="CASCADE")
|
||||
)
|
||||
playlist_id: Mapped[int | None] = mapped_column(ForeignKey("playlists.id", ondelete="CASCADE"))
|
||||
|
||||
# 0-based shelf order. The shelf is short and hand-curated, so a dense integer the reorder
|
||||
# endpoint rewrites wholesale beats a fractional index nobody would ever exercise here.
|
||||
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0", nullable=False)
|
||||
|
||||
@@ -33,6 +33,7 @@ def movie_card(it: PlexItem, st: PlexState | None) -> dict:
|
||||
"thumb": f"/api/plex/image/{it.rating_key}",
|
||||
"art": _art_url(it.rating_key, it.art_key),
|
||||
"playable": it.playable,
|
||||
"transcode_hw": it.transcode_hw, # HW/SW transcode split → the 4-way corner-dot badge
|
||||
"status": st.status if st else "new",
|
||||
"position_seconds": st.position_seconds if st else 0,
|
||||
}
|
||||
@@ -65,6 +66,7 @@ def episode_card(e: PlexItem, st: PlexState | None) -> dict:
|
||||
"duration_seconds": e.duration_s,
|
||||
"thumb": f"/api/plex/image/{e.rating_key}",
|
||||
"playable": e.playable,
|
||||
"transcode_hw": e.transcode_hw, # HW/SW transcode split → the 4-way corner-dot badge
|
||||
"status": st.status if st else "new",
|
||||
"position_seconds": st.position_seconds if st else 0,
|
||||
}
|
||||
|
||||
@@ -197,7 +197,12 @@ class PlexClient:
|
||||
)
|
||||
|
||||
def unscrobble(self, rating_key: str) -> None:
|
||||
"""Mark the item *unwatched* on the token's Plex account (clears viewCount)."""
|
||||
"""Mark the item *unwatched* on the token's Plex account.
|
||||
|
||||
Clears the view count AND the resume point (`viewOffset`) — measured against the live server
|
||||
on 2026-09-09, from both a watched+offset state and an offset-only one. This one call is the
|
||||
whole un-mark; `/:/timeline?time=0` and `/:/progress?time=0` were both tried there and do
|
||||
NOT clear an offset, so do not reach for them."""
|
||||
self._write(
|
||||
"GET",
|
||||
f"/:/unscrobble?identifier={self._LIBRARY_IDENTIFIER}"
|
||||
|
||||
@@ -11,5 +11,11 @@ _M = TypeVar("_M")
|
||||
def by_rating_key(db: Session, model: type[_M], rating_key: str) -> _M | None:
|
||||
"""The mirrored row for a Plex rating_key, or None. `model` is any of the rating_key-keyed Plex
|
||||
tables (PlexItem / PlexShow / PlexSeason / PlexCollection); the return type is that model, so call
|
||||
sites keep their `Model | None` static type."""
|
||||
sites keep their `Model | None` static type.
|
||||
|
||||
A key with a NUL in it (`%00` in a path segment) names no row — a 404, like any unknown key.
|
||||
Postgres refuses NUL in text, so asking would be a DataError; the app-wide handler turns the
|
||||
places that DO ask (lists and filters compared to rating_key) into a 422 (see main.py)."""
|
||||
if "\x00" in str(rating_key):
|
||||
return None
|
||||
return db.execute(select(model).where(model.rating_key == str(rating_key))).scalars().first()
|
||||
|
||||
@@ -8,9 +8,28 @@ from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import and_, case, func, or_, select, type_coerce
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import PlexItem, PlexLibrary
|
||||
|
||||
TS_CONFIG = "public.unaccent_simple"
|
||||
|
||||
# The ONE Plex sort vocabulary, shared by every surface that offers the Sort control (the browse grid
|
||||
# and the collection info page) and mirrored by the frontend's `plexSort.ts`. It lives here rather than
|
||||
# inside a router because the two surfaces order DIFFERENT things — browse orders a union subquery in
|
||||
# SQL, a collection orders its already-loaded member list in Python — and a key that one of them cannot
|
||||
# honour would put an option in the UI that silently does nothing. Tests pin both implementations to
|
||||
# this tuple, so adding a key here fails the gate until every surface can order by it.
|
||||
PLEX_SORT_KEYS: tuple[str, ...] = (
|
||||
"added",
|
||||
"lastwatched",
|
||||
"release",
|
||||
"year",
|
||||
"rating",
|
||||
"duration",
|
||||
"title",
|
||||
)
|
||||
|
||||
# --- Language-aware column pickers (E5 S6) ----------------------------------
|
||||
#
|
||||
# The HU-overlay twin (E5 S5) stores Hungarian metadata in `*_hu` columns on the SAME EN row. When
|
||||
@@ -206,6 +225,33 @@ def apply_meta_filters(q, model, p: dict, lang: str = "en"):
|
||||
return q.where(*conds) if conds else q
|
||||
|
||||
|
||||
# --- Playback-class filter (E7) — MOVIE-ONLY (a show is an aggregate with no single class) ---------
|
||||
# The 4 classes the feed filter + facet expose, derived from the stored `playable` + `transcode_hw`.
|
||||
# ONE definition (this SQL CASE) so the filter WHERE and the facet counts can't drift: `playable_conds`
|
||||
# filters on it, `facets` groups by it. transcode_hw IS NOT True groups NULL (unprobed) with software.
|
||||
# Public — it is the shared source of truth imported by routes.plex.browse (facet order + zero-fill).
|
||||
PLAYBACK_CLASSES = ("direct", "remux", "transcode_hw", "transcode_sw")
|
||||
|
||||
|
||||
def playback_class_expr():
|
||||
"""SQL CASE mapping a PlexItem (movie) to its playback-class string, or NULL. The HW branch must
|
||||
precede the generic transcode branch (ordered evaluation)."""
|
||||
return case(
|
||||
(PlexItem.playable == "direct", "direct"),
|
||||
(PlexItem.playable == "remux", "remux"),
|
||||
(and_(PlexItem.playable == "transcode", PlexItem.transcode_hw.is_(True)), "transcode_hw"),
|
||||
(PlexItem.playable == "transcode", "transcode_sw"),
|
||||
else_=None,
|
||||
)
|
||||
|
||||
|
||||
def playable_conds(sel: list[str]) -> list:
|
||||
"""A one-element condition list (or empty) selecting the given playback classes — an OR via
|
||||
`class IN (...)` over the shared CASE. Unknown keys are dropped."""
|
||||
keep = [c for c in sel if c in PLAYBACK_CLASSES]
|
||||
return [playback_class_expr().in_(keep)] if keep else []
|
||||
|
||||
|
||||
@dataclass
|
||||
class LibraryFilters:
|
||||
"""The shared sidebar filter query-params for /library + /facets, declared once and injected via
|
||||
@@ -228,6 +274,7 @@ class LibraryFilters:
|
||||
keywords: str | None = None # TMDB keyword chips (E5 S5), CSV; language-neutral
|
||||
collections: str | None = None # CSV of collection rating_keys
|
||||
collection_mode: str = "any" # how multiple collections combine: any (OR, default) | all (AND)
|
||||
playable: str | None = None # CSV of playback classes: direct|remux|transcode_hw|transcode_sw
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {
|
||||
@@ -247,6 +294,7 @@ class LibraryFilters:
|
||||
"keywords": self.keywords,
|
||||
"collections": self.collections,
|
||||
"collection_mode": self.collection_mode,
|
||||
"playable": self.playable,
|
||||
}
|
||||
|
||||
|
||||
@@ -311,3 +359,26 @@ def apply_show_state_filter(query, tot, wat, inp, show: str):
|
||||
if show == "unwatched":
|
||||
return query.where(wat == 0, inp == 0)
|
||||
return query
|
||||
|
||||
|
||||
def content_libs(db: Session) -> tuple[list[int], list[int]]:
|
||||
"""The BROWSABLE content libraries, partitioned into (movie_lib_ids, show_lib_ids) — the single
|
||||
source of the browse scope. Excludes HU overlay libraries (`overlay_of` set): those are never
|
||||
browsed on their own, they only lend their Hungarian metadata to the EN twin's rows (E5 S5).
|
||||
|
||||
Lives here, in the query layer, rather than in `routes/plex/_shared.py` where it started: E24's
|
||||
Home surface needs the same scope, and a domain package importing from `routes/` would invert the
|
||||
one-way dependency CLAUDE.md draws. It never touched request state, so nothing was lost by the
|
||||
move."""
|
||||
libs = (
|
||||
db.execute(
|
||||
select(PlexLibrary).where(
|
||||
PlexLibrary.enabled.is_(True), PlexLibrary.overlay_of.is_(None)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
movie_lib_ids = [lb.id for lb in libs if lb.kind == "movie"]
|
||||
show_lib_ids = [lb.id for lb in libs if lb.kind == "show"]
|
||||
return movie_lib_ids, show_lib_ids
|
||||
|
||||
@@ -3,11 +3,25 @@
|
||||
Drives the show/season grid badges and the Resume / Play-from-start / mark-all buttons.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import and_, case, func, or_, select
|
||||
from sqlalchemy.orm import Session, aliased
|
||||
|
||||
from app.models import PlexItem, PlexState
|
||||
|
||||
# Sort sentinel for a rail row whose timestamp is missing, so `key=lambda r: r[0] or NO_TS` reads the
|
||||
# same at every rail that merges two type-homogeneous lists in Python.
|
||||
#
|
||||
# It is NOT a crash fix, whatever the E24 commit that introduced it says. The `(ts is not None, ts)`
|
||||
# key it replaced cannot raise: tuple comparison finds the first UNEQUAL element via `==`, so two
|
||||
# `(False, None)` keys compare equal and `None < None` is never reached — verified by execution
|
||||
# during the E24 epic-end review, over every permutation of null/non-null rows. What the sentinel
|
||||
# buys is one idiom instead of two, and a key that says "missing sorts last" outright rather than
|
||||
# encoding it in a boolean flag. A BARE `r[0]` would genuinely raise, which is the mutation the
|
||||
# rails' tests do guard against.
|
||||
NO_TS = datetime.min.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def show_status(total: int, watched: int, inprog: int) -> str:
|
||||
"""Roll a show's episodes up into a single watch state: fully watched, partially started, or new."""
|
||||
|
||||
+485
-91
@@ -21,10 +21,12 @@ CPU-only prod host; only P3's full transcode is CPU-heavy.
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Sequence
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy.orm import Session as DbSession
|
||||
@@ -34,6 +36,7 @@ from app.config import settings
|
||||
from app.downloads.edit import encode_thread_cap
|
||||
from app.models import PlexItem
|
||||
from app.plex import paths
|
||||
from app.plex.sync import HW_DECODABLE_CODECS
|
||||
|
||||
log = logging.getLogger("siftlode.plex")
|
||||
|
||||
@@ -54,6 +57,10 @@ _SEG_SECONDS = 6
|
||||
# worse than the old zero-buffer behaviour).
|
||||
_WARMUP_SEGMENTS = 3
|
||||
_WARMUP_SEG_TIMEOUT_S = 8.0
|
||||
# The HLS variant ffmpeg gives the VIDEO rendition of a multi-audio session. `-var_stream_map` names
|
||||
# each audio rendition (`name:a0`, `a1`, …) but leaves the video one unnamed, so ffmpeg falls back to
|
||||
# its index for that one. See `segment_name` for why this is a constant and not a comment.
|
||||
_VIDEO_VARIANT = "0"
|
||||
_SESSION_IDLE_S = 600 # reap a session with no access for this long
|
||||
# A session directory is named `{rating_key}_{int(start_s)}_{tag}_{aoff:+.2f}` (see start_session);
|
||||
# before multi-audio it was just `{rating_key}_{int(start_s)}`. `sweep_orphans` deletes ONLY names
|
||||
@@ -93,15 +100,25 @@ class HlsSession:
|
||||
directory: Path,
|
||||
proc: subprocess.Popen,
|
||||
start_s: float,
|
||||
entry: str,
|
||||
variants: "Sequence[str]",
|
||||
transcode: bool = False,
|
||||
hw: bool = False,
|
||||
tonemap: bool = False,
|
||||
):
|
||||
# Unguessable per-session id, carried in the URLs the client is handed. Sessions are keyed
|
||||
# by rating_key alone, so a seek-restart or a SECOND viewer of the same item replaces the
|
||||
# registered one — and the old viewer's next request for `seg_12.ts`, resolved by rating_key,
|
||||
# was answered 200 with the NEW session's twelfth segment, produced from a different offset
|
||||
# under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it
|
||||
# (deep-review round 5). A request naming a session that is no longer registered is a 404,
|
||||
# which the player recovers from by restarting at the playhead.
|
||||
self.id = secrets.token_urlsafe(9)
|
||||
self.key = key
|
||||
self.dir = directory
|
||||
self.proc = proc
|
||||
self.transcode = transcode # video re-encoded (non-h264 source), not stream-copied
|
||||
self.hw = hw # transcode ran on the VAAPI hardware encoder (else libx264 software)
|
||||
self.tonemap = tonemap # HW HDR/10-bit path: OpenCL tonemap between VAAPI decode and encode
|
||||
self.start_s = start_s # the REQUESTED seek offset
|
||||
# The REAL media start: with `-ss X -c:v copy` ffmpeg can only cut at the nearest earlier
|
||||
# keyframe K<=X, and hls.js zero-bases the timeline to K (so video.currentTime=0 is K, not X).
|
||||
@@ -110,7 +127,13 @@ class HlsSession:
|
||||
# subtitle cue-shift both key off the true content position — no drift. Provisional = X until
|
||||
# the first segment is probed.
|
||||
self.media_start_s = start_s
|
||||
self.entry = entry # the playlist filename hls.js should load (master.m3u8 with subs, else index.m3u8)
|
||||
# The rendition names this session's ffmpeg was told to write — () for a single-audio
|
||||
# session. Held on the session because it is what `artifact_may_arrive` needs to tell a name
|
||||
# this session CAN produce from one belonging to the other argv branch (see it).
|
||||
self.variants = tuple(variants)
|
||||
# The playlist filename hls.js should load: the master of a multi-audio session, else the
|
||||
# single index. Derived, not passed, so it cannot disagree with the argv.
|
||||
self.entry = _MASTER_PL if self.variants else _INDEX_PL
|
||||
self.last_access = time.time()
|
||||
|
||||
|
||||
@@ -143,7 +166,7 @@ _HLS_TAIL = [
|
||||
|
||||
# --- Transcode (playable="transcode") video re-encode -----------------------------------------
|
||||
# The remux path stream-copies the video (`-c:v copy`); a non-h264 source can't be copied and must be
|
||||
# re-encoded to H.264. Two encoders, chosen per-session by `_want_hw`:
|
||||
# re-encoded to H.264. The encoder + tonemap are chosen per-session by `_select_encode`:
|
||||
# * VAAPI hardware (`h264_vaapi`) — the prod default once the iGPU is passed in. A full GPU pipeline
|
||||
# (decode + scale + encode all on-device) offloads the CPU almost entirely, so several concurrent
|
||||
# 1080p transcodes are fine and the API stays responsive.
|
||||
@@ -153,24 +176,149 @@ _HLS_TAIL = [
|
||||
# Everything else — the `-copyts`/`_HLS_TAIL`/segment/seek-restart model and the AAC audio path — is
|
||||
# unchanged, so hls.js, subtitles, multi-audio and seeking keep working exactly as for remux.
|
||||
_VAAPI_DEVICE = "/dev/dri/renderD128"
|
||||
# Where an OpenCL ICD registers its driver. The Intel NEO runtime (intel-opencl-icd) drops an .icd file
|
||||
# here; its presence is the precondition for the HW HDR→SDR tonemap graph (see _opencl_present).
|
||||
_OPENCL_ICD_DIR = Path("/etc/OpenCL/vendors")
|
||||
# Codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for the
|
||||
# full-GPU VAAPI pipeline. A source outside this set (AV1, MPEG-4 ASP/XviD…) falls back to software even
|
||||
# when the render node is present, because the GPU can't decode it to feed its own encoder.
|
||||
_HW_DECODABLE = {"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"}
|
||||
# full-GPU VAAPI pipeline. A source outside it (AV1, MPEG-4 ASP/XviD…) falls back to software even with
|
||||
# the render node present, because the GPU can't decode it to feed its own encoder. THE single source is
|
||||
# plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't disagree by
|
||||
# codec; aliased to the old name to keep the call sites below unchanged.
|
||||
_HW_DECODABLE = HW_DECODABLE_CODECS
|
||||
# Color-transfer values that mark a true-HDR source (PQ / HLG). A HW-decodable HDR source can't take the
|
||||
# plain scale_vaapi=format=nv12 path (Gen 9.5 VPP refuses it) — it needs the OpenCL tonemap graph
|
||||
# (_venc_opts tonemap branch) when the OpenCL runtime is present, else software libx264. A 10-bit source
|
||||
# WITHOUT one of these transfers is SDR, not HDR, and must not be tone-mapped. See _probe_video_kind.
|
||||
_HDR_TRANSFERS = {"smpte2084", "arib-std-b67"}
|
||||
|
||||
|
||||
def _hwaccel_in_opts(hw: bool) -> list[str]:
|
||||
# The three source classes that drive the transcode-pipeline choice (see _probe_video_kind).
|
||||
_VK_HWSAFE = "hwsafe" # 8-bit SDR — the plain scale_vaapi=format=nv12 GPU path
|
||||
_VK_HDR = "hdr" # PQ/HLG HDR — needs the OpenCL tonemap graph (or software if no runtime)
|
||||
_VK_SDR10 = "sdr10" # 10-bit *SDR* — a plain 10→8-bit downconvert, NEVER the HDR tone curve
|
||||
|
||||
|
||||
def _probe_video_kind(src: Path) -> str:
|
||||
"""Classify the source for the transcode pipeline — one of `_VK_HWSAFE` / `_VK_HDR` / `_VK_SDR10`:
|
||||
|
||||
* `_VK_HWSAFE` — 8-bit SDR: safe for the plain VAAPI path (`scale_vaapi=format=nv12` + h264_vaapi).
|
||||
* `_VK_HDR` — PQ/HLG HDR (any depth): the plain scale path breaks on Gen 9.5 VPP, so it needs the
|
||||
OpenCL tonemap graph (`tonemap_opencl=hable`) when the runtime is present, else software libx264.
|
||||
* `_VK_SDR10` — 10-bit but *SDR* (bt709 etc.): NOT HDR. Applying the HDR hable tone curve here would
|
||||
visibly alter an SDR picture (crushed highlights/midtones) where only a 10→8-bit downconvert was
|
||||
needed, so it must NOT take the tonemap graph — it goes to software libx264, which CPU-decodes and
|
||||
downconverts to 8-bit yuv420p exactly. (Kept off the plain scale_vaapi 10-bit path too: that path
|
||||
is unverified on Gen 9.5 VPP, and software is the known-correct downconvert.)
|
||||
|
||||
Ground truth from ffprobe (bit depth via pix_fmt / bits_per_raw_sample, HDR via color_transfer); the
|
||||
Plex mirror doesn't store these. Returns `_VK_HWSAFE` on any probe failure: the runtime HW→SW fallback
|
||||
in start_session backstops a HW session that still produces nothing, so a probe miss only costs the
|
||||
rare wasted first attempt, never a stranded item."""
|
||||
try:
|
||||
out = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"v:0",
|
||||
"-show_entries",
|
||||
"stream=pix_fmt,bits_per_raw_sample,color_transfer",
|
||||
"-of",
|
||||
"default=noprint_wrappers=1",
|
||||
str(src),
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return _VK_HWSAFE
|
||||
pix_fmt = bits = transfer = ""
|
||||
for line in out.stdout.splitlines():
|
||||
k, _, v = line.strip().partition("=")
|
||||
v = v.strip().lower()
|
||||
if k == "pix_fmt":
|
||||
pix_fmt = v
|
||||
elif k == "bits_per_raw_sample":
|
||||
bits = v
|
||||
elif k == "color_transfer":
|
||||
transfer = v
|
||||
if transfer in _HDR_TRANSFERS:
|
||||
return _VK_HDR # PQ/HLG — needs a real HDR→SDR tonemap, not a plain downconvert
|
||||
# >8-bit but not HDR = 10-bit SDR. bits_per_raw_sample OR the pix_fmt bit-depth suffix
|
||||
# (yuv420p10le, p010le, yuv444p12le…) signals >8-bit even when the other is absent.
|
||||
if (bits.isdigit() and int(bits) > 8) or re.search(r"(?:10|12|16)(?:le|be)$", pix_fmt):
|
||||
return _VK_SDR10
|
||||
return _VK_HWSAFE
|
||||
|
||||
|
||||
def _opencl_present() -> bool:
|
||||
"""Whether an OpenCL runtime (Intel NEO / intel-opencl-icd) is installed — the precondition for the
|
||||
HW HDR→SDR tonemap graph. Gen 9.5 VPP can't tonemap natively (tonemap_vaapi fails), so the working
|
||||
HW path for a 10-bit/HDR source is VAAPI-decode → OpenCL tonemap → VAAPI-encode, which needs an ICD
|
||||
registered here. Absent on the dev box and until the image bakes NEO in → those sources take
|
||||
software libx264. Its own function so it's monkeypatchable in tests, mirroring _render_node_present."""
|
||||
try:
|
||||
return any(_OPENCL_ICD_DIR.glob("*.icd"))
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _hwaccel_in_opts(hw: bool, tonemap: bool = False) -> list[str]:
|
||||
"""VAAPI decode options, placed before `-i` so the source is decoded (and kept) on the GPU.
|
||||
`-hwaccel_output_format vaapi` keeps frames as GPU surfaces all the way to `h264_vaapi` — no
|
||||
round-trip to system memory. Empty for the software path."""
|
||||
round-trip to system memory. Empty for the software path.
|
||||
|
||||
The tonemap variant additionally derives an OpenCL device FROM the VAAPI one (`opencl=ocl@va`) so
|
||||
tonemap_opencl can map the decoded surfaces zero-copy (see _venc_opts); it needs named devices, so it
|
||||
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`."""
|
||||
if not hw:
|
||||
return []
|
||||
if tonemap:
|
||||
return [
|
||||
"-init_hw_device",
|
||||
f"vaapi=va:{_VAAPI_DEVICE}",
|
||||
"-init_hw_device",
|
||||
"opencl=ocl@va",
|
||||
"-hwaccel",
|
||||
"vaapi",
|
||||
"-hwaccel_output_format",
|
||||
"vaapi",
|
||||
"-hwaccel_device",
|
||||
"va",
|
||||
]
|
||||
return ["-hwaccel", "vaapi", "-hwaccel_output_format", "vaapi", "-vaapi_device", _VAAPI_DEVICE]
|
||||
|
||||
|
||||
# Audio is ALWAYS re-encoded on the HLS path: the source is typically AC3/DTS 5.1, which no browser
|
||||
# plays, so it is downmixed to stereo AAC. `-ac 2` uses ffmpeg's standard downmix matrix; nothing is
|
||||
# normalised, which is why a film mastered for a cinema arrives with whispered dialogue and deafening
|
||||
# action (bug hunt 2026-09, item 14).
|
||||
_AUDIO_ENC = ["-c:a", "aac", "-ac", "2", "-b:a", "192k"]
|
||||
# Optional loudness normalisation (admin knob `plex_audio_loudnorm`, default OFF). EBU R128 single-pass
|
||||
# `loudnorm` — two-pass is impossible here, since a seek-restart session must start emitting segments
|
||||
# immediately and cannot analyse the file first.
|
||||
#
|
||||
# The downmix is FORCED FIRST (`aformat=channel_layouts=stereo`) rather than left to `-ac 2`: measuring
|
||||
# 5.1 and then folding it down would normalise something other than what the viewer hears. `loudnorm`
|
||||
# runs its own resampler at 192 kHz internally, so the chain resamples back to 48 kHz for the AAC
|
||||
# encoder. Measured on a real 5.1 AC3 film (Westworld S1E6, 60 s from 15:00): I -27.6 → -16.6 LUFS,
|
||||
# LRA 13.3 → 9.8 LU — the quiet/loud gap narrows, and everything gets ~11 dB louder, so expect to turn
|
||||
# the volume down after switching this on. The filter's 3 s lookahead does NOT desync A/V: measured
|
||||
# under this pipeline's own `-copyts`, the audio start PTS is identical with and without it.
|
||||
_LOUDNORM_AF = "aformat=channel_layouts=stereo,loudnorm=I=-16:TP=-1.5:LRA=11,aresample=48000"
|
||||
|
||||
|
||||
def _aenc_opts(loudnorm: bool) -> list[str]:
|
||||
"""Audio encoder args for one output, with the normalisation filter when the admin enabled it."""
|
||||
return [*_AUDIO_ENC, *(["-af", _LOUDNORM_AF] if loudnorm else [])]
|
||||
|
||||
|
||||
def _venc_opts(
|
||||
transcode: bool,
|
||||
hw: bool,
|
||||
tonemap: bool = False,
|
||||
preset: str = "veryfast",
|
||||
crf: int = 20,
|
||||
max_height: int = 0,
|
||||
@@ -183,6 +331,23 @@ def _venc_opts(
|
||||
if not transcode:
|
||||
return ["-c:v", "copy"]
|
||||
force_kf = ["-force_key_frames", f"expr:gte(t,n_forced*{_SEG_SECONDS})"]
|
||||
if hw and tonemap:
|
||||
# HDR/10-bit HW path: Gen 9.5 VPP can't tonemap (scale_vaapi=format=nv12 fails), so the decoded
|
||||
# VAAPI surfaces are mapped to OpenCL (hwmap=derive_device=opencl), tone-mapped HDR→SDR to 8-bit
|
||||
# nv12 there (tonemap_opencl, hable curve, bt709 out), then mapped back to VAAPI (reverse=1) for
|
||||
# h264_vaapi. `-filter_hw_device ocl` tells the filter graph which device to run the OpenCL
|
||||
# filters on. Full GPU pipeline — ~1 CPU core, several×realtime on the UHD 630 (measured).
|
||||
return [
|
||||
"-filter_hw_device",
|
||||
"ocl",
|
||||
"-vf",
|
||||
"hwmap=derive_device=opencl,"
|
||||
"tonemap_opencl=tonemap=hable:transfer=bt709:matrix=bt709:primaries=bt709:format=nv12,"
|
||||
"hwmap=derive_device=vaapi:reverse=1",
|
||||
"-c:v",
|
||||
"h264_vaapi",
|
||||
*force_kf,
|
||||
]
|
||||
if hw:
|
||||
# scale_vaapi with format=nv12 downconverts a 10-bit (p010) HEVC source to 8-bit nv12 so
|
||||
# h264_vaapi (8-bit High) can encode it; for an 8-bit source it's a cheap format pass-through.
|
||||
@@ -220,20 +385,53 @@ def _render_node_present() -> bool:
|
||||
return Path(_VAAPI_DEVICE).exists()
|
||||
|
||||
|
||||
def _want_hw(db: DbSession, item: PlexItem, transcode: bool) -> bool:
|
||||
"""Whether to use the VAAPI hardware encoder for this transcode session, honouring the admin
|
||||
`plex_transcode_hw` policy: "off" → always software; "on" → VAAPI wherever the render node exists
|
||||
(trusting the admin, skipping the source-decodability gate); "auto" (default) → VAAPI only when the
|
||||
render node is present AND the iGPU can hardware-decode the source (else the full-GPU pipeline can't
|
||||
feed itself → software libx264)."""
|
||||
def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> tuple[bool, bool]:
|
||||
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap)`:
|
||||
* `(False, False)` → software libx264 (the fallback: no render node, non-decodable source, or a
|
||||
10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
|
||||
* `(True, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR source.
|
||||
* `(True, True)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG) source, when
|
||||
the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
|
||||
|
||||
A 10-bit *SDR* source is deliberately NOT a `(True, True)` case: it needs a plain 10→8-bit downconvert,
|
||||
not the HDR tone curve (which would visibly darken/flatten an SDR picture), so it drops to software.
|
||||
|
||||
Honours the admin `plex_transcode_hw` policy: "off" → always software; "on" → HW wherever the render
|
||||
node exists (trusting the admin, skipping the decodability gate — the runtime HW→SW fallback still
|
||||
backstops a session that produces nothing — but still routing a true-HDR source through the tonemap
|
||||
graph so it doesn't hit the scale_vaapi wall); "auto" (default) → HW only when the render node is
|
||||
present AND the iGPU can hardware-decode the source, with tonemap chosen only for a true-HDR (PQ/HLG)
|
||||
source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
|
||||
if not transcode:
|
||||
return False
|
||||
return (False, False)
|
||||
mode = (sysconfig.get_str(db, "plex_transcode_hw") or "auto").lower()
|
||||
if mode == "off" or not _render_node_present():
|
||||
return False
|
||||
return (False, False)
|
||||
if mode == "on":
|
||||
return True
|
||||
return (item.codec_video or "").lower() in _HW_DECODABLE
|
||||
# Trust the admin: HW wherever the render node exists, skipping the decodability gate (the runtime
|
||||
# backstop catches a miss). Only a true-HDR (PQ/HLG) source needs the tonemap graph — a 10-bit SDR
|
||||
# source takes the plain scale path here, NOT the HDR tone curve. Probe only when OpenCL exists
|
||||
# (short-circuit), so "on" stays probe-free where tonemap can't apply anyway.
|
||||
tonemap = _opencl_present() and _probe_video_kind(src) == _VK_HDR
|
||||
return (True, tonemap)
|
||||
# auto
|
||||
if (item.codec_video or "").lower() not in _HW_DECODABLE:
|
||||
return (False, False)
|
||||
kind = _probe_video_kind(src)
|
||||
if kind == _VK_HWSAFE:
|
||||
return (True, False) # 8-bit SDR → full-GPU scale path
|
||||
if kind == _VK_HDR:
|
||||
# PQ/HLG: HW OpenCL tonemap when the runtime is present, else software libx264.
|
||||
return (True, True) if _opencl_present() else (False, False)
|
||||
# _VK_SDR10 — 10-bit SDR: a plain 10→8-bit downconvert, done exactly on the CPU. Never the HDR tone
|
||||
# curve (it would alter an SDR picture), and off the unverified scale_vaapi 10-bit path — software.
|
||||
return (False, False)
|
||||
|
||||
|
||||
def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
|
||||
"""The HW gate on its own (the `hw` half of `_select_encode`). A readability shim for the tests that
|
||||
assert only whether a session runs on the GPU (production decides via `_select_encode` directly)."""
|
||||
return _select_encode(db, item, transcode, src)[0]
|
||||
|
||||
|
||||
# CPU-nice increment for the SOFTWARE transcode child. The Plex ffmpeg is a Popen in the long-lived API
|
||||
@@ -288,15 +486,18 @@ def _ffmpeg_cmd(
|
||||
audio_langs: list[str] | None = None,
|
||||
transcode: bool = False,
|
||||
hw: bool = False,
|
||||
tonemap: bool = False,
|
||||
preset: str = "veryfast",
|
||||
crf: int = 20,
|
||||
max_height: int = 0,
|
||||
loudnorm: bool = False,
|
||||
) -> list[str]:
|
||||
# For a transcode session the video is re-encoded (VAAPI or libx264) instead of stream-copied; the
|
||||
# hwaccel decode options go before -i, the encoder args replace `-c:v copy` below. Everything else is
|
||||
# identical to remux.
|
||||
venc = _venc_opts(transcode, hw, preset, crf, max_height)
|
||||
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw)]
|
||||
# identical to remux. `tonemap` is the HDR/10-bit HW variant (OpenCL tonemap between VAAPI decode and
|
||||
# encode) — it needs matching decode-side (named devices) and encode-side (the OpenCL filter graph) opts.
|
||||
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height)
|
||||
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap)]
|
||||
# Keep ORIGINAL timestamps on the output so the first segment carries the true absolute PTS of the
|
||||
# keyframe ffmpeg actually seeked to. hls.js still zero-bases playback to that PTS; we read it back
|
||||
# (see _probe_first_pts) to learn the real content offset. Without -copyts the segment PTS is
|
||||
@@ -313,11 +514,15 @@ def _ffmpeg_cmd(
|
||||
if audio_langs:
|
||||
# MULTI-RENDITION: map EVERY audio track as an alternate HLS rendition in one group, so hls.js
|
||||
# switches audio CLIENT-SIDE on the same timeline (no session restart, no drift) — the premium
|
||||
# audio-switch path. Produces master.m3u8 + stream_%v.m3u8 (v0=video, v1..=audio) + seg_%v_%d.ts.
|
||||
# audio-switch path. Produces master.m3u8 + one stream_%v.m3u8 and seg_%v_*.ts per variant,
|
||||
# where %v is the video rendition's index ("0") and each audio rendition's `name:` (a0, a1,
|
||||
# …). Those names are spelled by `segment_name` / `variant_playlist_name` below, which the
|
||||
# guard and the warm-up read back — see them for the measurement behind the ordering.
|
||||
args += ["-map", "0:v:0"]
|
||||
for i in range(len(audio_langs)):
|
||||
args += ["-map", f"{ai}:a:{i}?"]
|
||||
args += ["-sn", *venc, "-c:a", "aac", "-ac", "2", "-b:a", "192k"]
|
||||
# One `-af` covers EVERY audio output of this file, so each rendition is normalised.
|
||||
args += ["-sn", *venc, *_aenc_opts(loudnorm)]
|
||||
var = ["v:0,agroup:aud"]
|
||||
for i, lang in enumerate(audio_langs):
|
||||
entry = f"a:{i},agroup:aud,name:a{i}"
|
||||
@@ -326,23 +531,25 @@ def _ffmpeg_cmd(
|
||||
if i == 0:
|
||||
entry += ",default:yes"
|
||||
var.append(entry)
|
||||
# The name patterns come from the vocabulary functions, with ffmpeg's own placeholders passed
|
||||
# through them — so the producer and the parser cannot spell the layout differently (round 7).
|
||||
args += _HLS_TAIL + [
|
||||
"-master_pl_name",
|
||||
"master.m3u8",
|
||||
_MASTER_PL,
|
||||
"-var_stream_map",
|
||||
" ".join(var),
|
||||
"-hls_segment_filename",
|
||||
str(out_dir / "seg_%v_%d.ts"),
|
||||
str(out_dir / "stream_%v.m3u8"),
|
||||
str(out_dir / segment_name("%d", "%v")),
|
||||
str(out_dir / variant_playlist_name("%v")),
|
||||
]
|
||||
else:
|
||||
ao = audio_ord if audio_ord is not None else 0
|
||||
args += ["-map", "0:v:0", "-map", f"{ai}:a:{ao}?", "-sn"]
|
||||
args += [*venc, "-c:a", "aac", "-ac", "2", "-b:a", "192k"]
|
||||
args += [*venc, *_aenc_opts(loudnorm)]
|
||||
args += _HLS_TAIL + [
|
||||
"-hls_segment_filename",
|
||||
str(out_dir / "seg_%d.ts"),
|
||||
str(out_dir / "index.m3u8"),
|
||||
str(out_dir / segment_name("%d")),
|
||||
str(out_dir / _INDEX_PL),
|
||||
]
|
||||
return args
|
||||
|
||||
@@ -423,69 +630,117 @@ def start_session(
|
||||
src = paths.local_media_path(db, item.file_path)
|
||||
if src is None:
|
||||
return None
|
||||
# Read once, here: the knob has to be identical for the directory name and the ffmpeg argv, and a
|
||||
# session respawn (the HW->SW fallback) must not pick up a mid-playback config change.
|
||||
loudnorm = sysconfig.get_bool(db, "plex_audio_loudnorm")
|
||||
key = item.rating_key
|
||||
start_s = max(0.0, float(start_s))
|
||||
hw = _want_hw(db, item, transcode)
|
||||
# Software-fallback encode tuning — read only on the SW transcode path (the VAAPI path and remux
|
||||
# ignore preset/crf/max_height, and remux must not touch the DB for them).
|
||||
preset, crf, max_height = "veryfast", 20, 0
|
||||
if transcode and not hw:
|
||||
preset = sysconfig.get_str(db, "plex_transcode_preset") or "veryfast"
|
||||
crf = sysconfig.get_int(db, "plex_transcode_crf")
|
||||
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
|
||||
audio_langs = _probe_audio_langs(src) if multi else []
|
||||
is_multi = len(audio_langs) > 1 # only worth a master playlist when there really are ≥2 tracks
|
||||
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode can't
|
||||
# starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a platform
|
||||
# without fork (the API always runs in a Linux container in prod) it's simply omitted.
|
||||
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
|
||||
with _lock:
|
||||
old = _sessions.pop(key, None)
|
||||
if old is not None:
|
||||
_kill(old)
|
||||
tag = "multi" if is_multi else str(audio_ord)
|
||||
directory = _HLS_ROOT / f"{key}_{int(start_s)}_{tag}_{aoff:+.2f}"
|
||||
shutil.rmtree(directory, ignore_errors=True)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
proc = subprocess.Popen(
|
||||
_ffmpeg_cmd(
|
||||
src,
|
||||
start_s,
|
||||
# The rendition names the argv below will produce — one source for the argv, the warm-up's
|
||||
# segment names, the entry playlist and the artifact guard.
|
||||
variants = session_variants(audio_langs if is_multi else None)
|
||||
# Loudness normalisation changes the AUDIO THE SEGMENTS CONTAIN, so it belongs in the directory
|
||||
# name beside the other parameters that do (offset, audio track, A/V offset): a session left over
|
||||
# from before the admin flipped the knob must not be mistaken for one that honours it. Appended to
|
||||
# the tag with a DASH — `_SESSION_DIR_RE` treats the tag as one underscore-free field, and that
|
||||
# regex is the only thing keeping the orphan sweep off a shared scratch path (see its comment).
|
||||
tag = ("multi" if is_multi else str(audio_ord)) + ("-ln" if loudnorm else "")
|
||||
directory = _HLS_ROOT / f"{key}_{int(start_s)}_{tag}_{aoff:+.2f}"
|
||||
|
||||
def _seg(i: int) -> Path:
|
||||
return directory / segment_name(i, _VIDEO_VARIANT if variants else None)
|
||||
|
||||
def _spawn(
|
||||
hw: bool, tonemap: bool = False, *, replace_only: "HlsSession | None" = None
|
||||
) -> "HlsSession | None":
|
||||
# `replace_only` makes the supersede identity-aware for the runtime fallback: the whole
|
||||
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
|
||||
# seek-restart) installed between the fallback's decision and here is never clobbered. When the
|
||||
# registered session is no longer `replace_only`, we abort and return None (the caller yields to
|
||||
# whoever is registered now). The INITIAL spawn passes replace_only=None and always supersedes —
|
||||
# that is the seek-restart's intended last-writer-wins.
|
||||
# Software-fallback encode tuning — read only on the SW transcode path (the VAAPI path and
|
||||
# remux ignore preset/crf/max_height, and remux must not touch the DB for them).
|
||||
preset, crf, max_height = "veryfast", 20, 0
|
||||
if transcode and not hw:
|
||||
preset = sysconfig.get_str(db, "plex_transcode_preset") or "veryfast"
|
||||
crf = sysconfig.get_int(db, "plex_transcode_crf")
|
||||
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
|
||||
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
|
||||
# can't starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a
|
||||
# platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
|
||||
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
|
||||
with _lock:
|
||||
if replace_only is not None and _sessions.get(key) is not replace_only:
|
||||
return None # superseded/culled while we waited out of lock — don't clobber the newer one
|
||||
old = _sessions.pop(key, None)
|
||||
if old is not None:
|
||||
_kill(old)
|
||||
shutil.rmtree(directory, ignore_errors=True)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
proc = subprocess.Popen(
|
||||
_ffmpeg_cmd(
|
||||
src,
|
||||
start_s,
|
||||
directory,
|
||||
audio_ord,
|
||||
aoff,
|
||||
audio_langs if is_multi else None,
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
tonemap=tonemap,
|
||||
preset=preset,
|
||||
crf=crf,
|
||||
max_height=max_height,
|
||||
loudnorm=loudnorm,
|
||||
),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
preexec_fn=preexec,
|
||||
)
|
||||
s = HlsSession(
|
||||
key,
|
||||
directory,
|
||||
audio_ord,
|
||||
aoff,
|
||||
audio_langs if is_multi else None,
|
||||
proc,
|
||||
start_s,
|
||||
variants,
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
preset=preset,
|
||||
crf=crf,
|
||||
max_height=max_height,
|
||||
),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
preexec_fn=preexec,
|
||||
)
|
||||
s = HlsSession(
|
||||
key,
|
||||
directory,
|
||||
proc,
|
||||
start_s,
|
||||
"master.m3u8" if is_multi else "index.m3u8",
|
||||
transcode=transcode,
|
||||
hw=hw,
|
||||
)
|
||||
_sessions[key] = s
|
||||
# Admin-configurable concurrency cap (Configuration → Plex → Max concurrent transcodes);
|
||||
# at least 1 so playback can't be capped to zero.
|
||||
_enforce_cap(max(1, sysconfig.get_int(db, "plex_max_transcodes")))
|
||||
tonemap=tonemap,
|
||||
)
|
||||
_sessions[key] = s
|
||||
# Admin-configurable concurrency cap (Configuration → Plex → Max concurrent transcodes);
|
||||
# at least 1 so playback can't be capped to zero.
|
||||
_enforce_cap(max(1, sysconfig.get_int(db, "plex_max_transcodes")))
|
||||
return s
|
||||
|
||||
# Wait for the first video segment, then warm up a few more — all outside the lock (this blocks on
|
||||
# ffmpeg producing them, and we must not hold up other sessions).
|
||||
def _seg(i: int) -> Path:
|
||||
# multi-audio: the video rendition is variant v0 → seg_{i}_0.ts; single-audio → seg_{i}.ts
|
||||
return directory / (f"seg_{i}_0.ts" if is_multi else f"seg_{i}.ts")
|
||||
|
||||
if wait_for(_seg(0), timeout=25.0):
|
||||
s = _spawn(*_select_encode(db, item, transcode, src))
|
||||
# Wait for the first video segment outside the lock (this blocks on ffmpeg, and we must not hold up
|
||||
# other sessions). Pass the proc so a HW session that dies at VAAPI init fails fast (not after 25s).
|
||||
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
|
||||
if not ready and s.hw:
|
||||
# We waited on _seg(0) OUTSIDE _lock, so `s.proc` dying is ambiguous: it may have failed on its
|
||||
# own (VAProfile/init/tonemap refusal), OR a concurrent same-key _spawn (a rapid seek-restart) or
|
||||
# _enforce_cap may have killed it to install/keep a newer session. Respawn on libx264 (no HW, no
|
||||
# tonemap) so a genuinely-failed GPU session never strands playback — but pass `replace_only=s` so
|
||||
# the respawn is a no-op if we've been superseded: the identity check and the pop+replace share
|
||||
# ONE lock in _spawn, so we can never tear down the newer session. s.hw is now False, so the route
|
||||
# honestly reports "CPU" and the player's Preparing-stream state stays correct.
|
||||
log.warning(
|
||||
"plex transcode: HW session produced no segments — falling back to software key=%s", key
|
||||
)
|
||||
respawned = _spawn(False, False, replace_only=s)
|
||||
if respawned is None:
|
||||
# Superseded/culled mid-fallback — yield whoever is registered now (a newer seek-restart), or
|
||||
# our own now-dead session if the key was cap-culled. Never None here: the POST /session route
|
||||
# turns a None return into a "Media file not found on disk" 404 — dishonest for a file that
|
||||
# exists. A (dead) session instead degrades to a retryable 404 ("No active playback session" /
|
||||
# "Not available yet") the client re-requests.
|
||||
return current_session(key) or s
|
||||
s = respawned
|
||||
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
|
||||
if ready:
|
||||
# For a seek (start_s>0) learn the REAL start offset K from the first segment's absolute PTS,
|
||||
# so the client's absolute clock + subtitle shift key off the true keyframe, not the requested
|
||||
# offset. On probe failure we keep the provisional requested offset (old behaviour, small drift).
|
||||
@@ -497,17 +752,18 @@ def start_session(
|
||||
# at the first segment that isn't ready in time — ffmpeg produces them faster than realtime once
|
||||
# warm, so a miss means something slow and we proceed rather than block the request further.
|
||||
for i in range(1, _WARMUP_SEGMENTS):
|
||||
if not wait_for(_seg(i), timeout=_WARMUP_SEG_TIMEOUT_S):
|
||||
if not wait_for(_seg(i), timeout=_WARMUP_SEG_TIMEOUT_S, proc=s.proc):
|
||||
break
|
||||
log.info(
|
||||
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s",
|
||||
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s tonemap=%s",
|
||||
key,
|
||||
start_s,
|
||||
s.media_start_s,
|
||||
is_multi,
|
||||
audio_ord,
|
||||
transcode,
|
||||
hw,
|
||||
s.hw,
|
||||
s.tonemap,
|
||||
)
|
||||
return s
|
||||
|
||||
@@ -520,10 +776,109 @@ def current_session(key: str) -> HlsSession | None:
|
||||
return s
|
||||
|
||||
|
||||
def wait_for(path: Path, timeout: float = 20.0) -> bool:
|
||||
# The names ffmpeg writes into a session directory, in ONE place — and "one place" means the argv
|
||||
# BUILDS its patterns from these same functions (`segment_name("%d", "%v")`), not that a comment
|
||||
# says so. Getting this wrong is invisible in both directions: a name we failed to anticipate used
|
||||
# to fall back to "wait and see" (round 6's HIGH), and now that the rule is positive it falls back
|
||||
# to a 404 instead — so the producer and the parser have to be literally the same spelling.
|
||||
#
|
||||
# MEASURED (ffmpeg 7.1.5 in the API container — the binary that actually writes these files — and
|
||||
# re-checked on 8.1.1): `-hls_segment_filename seg_%v_%d.ts` substitutes `%v` with the VARIANT and
|
||||
# `%d` with the SEQUENCE — variant FIRST. A dual-audio session writes seg_0_0.ts… (video, the
|
||||
# unnamed variant) plus seg_a0_0.ts… / seg_a1_0.ts… (the `name:aN` audio renditions), and the
|
||||
# playlists beside them are master.m3u8 + stream_0/a0/a1.m3u8. Single-audio uses `seg_%d.ts` +
|
||||
# index.m3u8, with no variant field at all.
|
||||
#
|
||||
# The two argv branches are DISJOINT vocabularies, which is why the guard below is keyed to the
|
||||
# SESSION and not to HLS in general: a single-audio session can never write `master.m3u8`,
|
||||
# `stream_*.m3u8` or any `seg_<variant>_<n>.ts`, and a multi-audio one can never write `index.m3u8`
|
||||
# or `seg_<n>.ts`. Accepting the union meant a hopeless name of the other shape was waited out for
|
||||
# the full playlist/segment timeout — the threadpool parking this guard exists to end (round 7).
|
||||
_MASTER_PL = "master.m3u8"
|
||||
_INDEX_PL = "index.m3u8"
|
||||
_SEG_NAME_RE = re.compile(r"^seg_(?:([0-9a-z]{1,4})_)?(\d{1,9})\.ts$")
|
||||
|
||||
|
||||
def session_variants(audio_langs: "Sequence[str | None] | None") -> tuple[str, ...]:
|
||||
"""The `-var_stream_map` rendition names a session will write, or () for a single-audio one.
|
||||
|
||||
The video rendition is left unnamed in the map, so ffmpeg falls back to its index
|
||||
(`_VIDEO_VARIANT`); each audio rendition is named `aN` in track order."""
|
||||
if not audio_langs:
|
||||
return ()
|
||||
return (_VIDEO_VARIANT, *(f"a{i}" for i in range(len(audio_langs))))
|
||||
|
||||
|
||||
def segment_name(seq: "int | str", variant: "str | None" = None) -> str:
|
||||
"""The file ffmpeg writes for the `seq`-th segment of `variant`.
|
||||
|
||||
`variant=None` is a single-audio session, whose argv carries no `%v` field at all. `seq` and
|
||||
`variant` are typed loosely because `_ffmpeg_cmd` builds its own `-hls_segment_filename` pattern
|
||||
through this function, passing ffmpeg's `"%d"` / `"%v"` placeholders — that is what keeps the
|
||||
producer and the parser from drifting, rather than a comment asking the next editor not to."""
|
||||
return f"seg_{variant}_{seq}.ts" if variant else f"seg_{seq}.ts"
|
||||
|
||||
|
||||
def variant_playlist_name(variant: str) -> str:
|
||||
"""The sub-playlist ffmpeg writes for one rendition of a multi-audio session."""
|
||||
return f"stream_{variant}.m3u8"
|
||||
|
||||
|
||||
def playlist_names(variants: "Sequence[str]") -> frozenset[str]:
|
||||
"""Every playlist name this session's argv can write — the master plus one per rendition, or
|
||||
just `index.m3u8` for a single-audio session."""
|
||||
if not variants:
|
||||
return frozenset({_INDEX_PL})
|
||||
return frozenset({_MASTER_PL, *(variant_playlist_name(v) for v in variants)})
|
||||
|
||||
|
||||
def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
|
||||
"""Whether waiting for `filename` can possibly end in a file OF THIS SESSION.
|
||||
|
||||
Positive rule, not a blocklist: a session produces segments and playlists and NOTHING else, so
|
||||
any other name is refused at once. ffmpeg also produces segments strictly in order, so a segment
|
||||
whose PREDECESSOR does not exist yet is not the next one — it is beyond the generated region,
|
||||
and no amount of waiting will conjure it. Playlists are the one thing that must still be waited
|
||||
for: they are rewritten continuously and have no ordering to read.
|
||||
|
||||
Keyed to the SESSION's own vocabulary, not to HLS's: the two argv branches are disjoint, so a
|
||||
name belonging to the other shape is as hopeless here as a random string. Reading those as "an
|
||||
HLS name, so wait and see" left the whole class reachable — measured through the real route
|
||||
against a live single-audio session, `master.m3u8` and `stream_q7z.m3u8` were 404s after 30.0 s
|
||||
and `seg_a0_0.ts` / `seg_zzzz_0.ts` after 20.1 s each (deep-review round 7).
|
||||
|
||||
Saying no at once matters because the route that waits is a SYNC one: it holds a thread off the
|
||||
~40-slot pool, and a pooled DB connection with it, for the whole timeout. An ordinary seek past
|
||||
the produced region was enough to park one for 20 s, and forty concurrent requests for a name
|
||||
that will never exist parked the whole API (deep-review round 5). Round 5's guard read only
|
||||
single-audio segment names, so both halves of that were still fully reachable: every multi-audio
|
||||
name fell through, and so did any `.m3u8`/`.vtt` name at all."""
|
||||
if filename in playlist_names(s.variants):
|
||||
return True
|
||||
m = _SEG_NAME_RE.match(filename)
|
||||
if m is None:
|
||||
return False
|
||||
variant, seq = m.group(1), int(m.group(2))
|
||||
# The variant field must be one THIS session writes: present and known for a multi-audio
|
||||
# session, absent for a single-audio one.
|
||||
if variant not in (s.variants or (None,)):
|
||||
return False
|
||||
if seq == 0:
|
||||
return True
|
||||
try:
|
||||
return (s.dir / segment_name(seq - 1, variant)).exists()
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def wait_for(path: Path, timeout: float = 20.0, proc: subprocess.Popen | None = None) -> bool:
|
||||
"""Wait until a session file (playlist / segment) exists and is non-empty. Segments are
|
||||
produced ~faster than realtime, so a segment just ahead of playback appears quickly; a segment
|
||||
far beyond the generated region won't (the frontend restarts the session at a seek instead)."""
|
||||
far beyond the generated region won't (the frontend restarts the session at a seek instead).
|
||||
|
||||
With `proc` given, also stop as soon as that ffmpeg exits: a VAAPI session that fails to init
|
||||
(10-bit/HDR VAProfile refusal) dies within ~1s producing nothing, and without this we'd block the
|
||||
whole `timeout` before the HW→SW fallback could kick in — turning the fix into a stall."""
|
||||
end = time.time() + timeout
|
||||
while time.time() < end:
|
||||
try:
|
||||
@@ -531,6 +886,12 @@ def wait_for(path: Path, timeout: float = 20.0) -> bool:
|
||||
return True
|
||||
except OSError:
|
||||
pass
|
||||
if proc is not None and proc.poll() is not None:
|
||||
# the producer exited — no more segments are coming; one last check then give up.
|
||||
try:
|
||||
return path.exists() and path.stat().st_size > 0
|
||||
except OSError:
|
||||
return False
|
||||
time.sleep(0.15)
|
||||
return path.exists()
|
||||
|
||||
@@ -581,14 +942,47 @@ def sweep_orphans(older_than: float | None = None) -> int:
|
||||
return dropped
|
||||
|
||||
|
||||
def reap_idle() -> int:
|
||||
# How long a session whose ffmpeg has FINISHED may go unfetched before a new session request reaps
|
||||
# it (see reap_idle).
|
||||
_FINISHED_IDLE_S = 30
|
||||
|
||||
|
||||
def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
|
||||
"""Kill sessions nobody has said they still want for `_SESSION_IDLE_S` — or, once their ffmpeg
|
||||
has finished, for `finished_idle_s`.
|
||||
|
||||
A finished session is also what a PAUSED viewer has: a remux runs far faster than realtime, so
|
||||
ffmpeg is done a minute or two in and writes ENDLIST; hls.js then stops asking for the playlist,
|
||||
and with its forward buffer full a paused player fetches nothing at all. With the 30 s rule that
|
||||
killed a one-minute pause, and playback died ~30 s after resuming (deep-review round 4).
|
||||
|
||||
What makes the short rule safe again is that fetching is no longer the only signal: a player
|
||||
parked MID-ITEM checkpoints its position every 10 s, and that route calls `touch_session`, so a
|
||||
viewer who is merely paused says so far more often than this rule asks (round 5). The heartbeat
|
||||
stops once the item is finished or before a position is known (`usePlexProgress.flush`) — both
|
||||
states where reaping is correct, and where the player's 404 → reclaim recovers a viewer who
|
||||
scrubs back in anyway. Do not restate this as "every 10 s whether or not it is playing": round 6
|
||||
found that claim in two comments and not in the code. Round 4's blanket
|
||||
`reap_abandoned` bought the same safety by keeping every finished session for ten minutes — each
|
||||
one a whole segmented copy of the film on a volume shared with the download center."""
|
||||
now = time.time()
|
||||
dropped = 0
|
||||
with _lock:
|
||||
for key, s in list(_sessions.items()):
|
||||
done = s.proc.poll() is not None
|
||||
if now - s.last_access > _SESSION_IDLE_S or (done and now - s.last_access > 30):
|
||||
idle = now - s.last_access
|
||||
if idle > _SESSION_IDLE_S or (done and idle > finished_idle_s):
|
||||
_kill(s)
|
||||
_sessions.pop(key, None)
|
||||
dropped += 1
|
||||
return dropped
|
||||
|
||||
|
||||
def touch_session(key: str) -> None:
|
||||
"""Say a viewer is still there for `key`, without fetching anything. `last_access` used to move
|
||||
only on an HLS file fetch, which a paused player does not do — so the only way to keep a paused
|
||||
viewer's session alive was to keep EVERY abandoned one just as long."""
|
||||
with _lock:
|
||||
s = _sessions.get(key)
|
||||
if s is not None:
|
||||
s.last_access = time.time()
|
||||
|
||||
+730
-14
@@ -1,15 +1,57 @@
|
||||
"""WebVTT conversion, HLS-clock time-shift, and on-disk cache for Plex subtitle streams.
|
||||
"""WebVTT conversion, HLS-clock time-shift, and the on-disk cache + background extractor for Plex
|
||||
subtitle streams.
|
||||
|
||||
Text subtitles (embedded or external) are turned into WebVTT so the browser <track> can render them;
|
||||
bitmap subtitles (Blu-ray/DVD) can't be (would need burn-in). Converted VTTs are cached on the shared
|
||||
Plex on-disk cache root (``cache.CACHE_DIR``).
|
||||
|
||||
WHY THERE IS A BACKGROUND EXTRACTOR HERE — measured on prod, 2026-09-10, against a 17.06 GB 1080p
|
||||
MKV with two embedded SubRip tracks on the 81 TB array:
|
||||
|
||||
=========================================================== ========
|
||||
full pass, ONE track (what this module used to do) 154 s
|
||||
full pass, BOTH text tracks in a SINGLE ffmpeg run 111 s
|
||||
`-c:s copy` instead of converting to WebVTT 143 s
|
||||
from 1:00:00 to the end of the file 66 s
|
||||
a 630 s WINDOW at 1:00:00, both tracks 8 s
|
||||
=========================================================== ========
|
||||
|
||||
The conversion is not the cost (dropping it saved 7%): ffmpeg has to stream every byte of the
|
||||
container between its seek point and its stop point, and the array delivers ~140 MB/s — so the time
|
||||
is (bytes read) / 140 MB/s, and the ONLY lever is reading less. Hence:
|
||||
|
||||
* extract a WINDOW around the playhead first — the viewer sees subtitles in seconds — then complete
|
||||
the whole track in the background so every later seek is instant and the next session is free;
|
||||
* extract EVERY embedded text track in ONE pass, because the read is shared: picking Hungarian used
|
||||
to cost a full read and then switching to English cost another one (the "hektikus" report);
|
||||
* never do any of it inside a request (it froze the <track> with no feedback AND held an API worker
|
||||
thread for the whole read).
|
||||
|
||||
`-copyts` CANNOT be used for a window: it makes ffmpeg ignore `-t`/`-to` (measured — a [0, 600 s]
|
||||
window still took the full 145 s and emitted the whole track), so windows are extracted on a
|
||||
zero-based clock and shifted back to absolute time here with `shift_vtt`.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import PlexItem
|
||||
from app.plex import paths as plex_paths
|
||||
from app.plex.cache import CACHE_DIR
|
||||
from app.plex.client import PlexError
|
||||
from app.plex.client import PlexClient, PlexError
|
||||
from app.plex.lookups import by_rating_key
|
||||
|
||||
log = logging.getLogger("siftlode.plex")
|
||||
|
||||
# Image-based subtitle codecs (Blu-ray/DVD) — can't be turned into text WebVTT (would need burn-in).
|
||||
BITMAP_SUBS = {
|
||||
@@ -31,6 +73,9 @@ def srt_to_vtt(text: str) -> str:
|
||||
|
||||
|
||||
def _vtt_ts_to_s(ts: str) -> float:
|
||||
# Also reads ffmpeg's spelling of a NEGATIVE time (a windowed pass, see `_ffmpeg_extract`): its
|
||||
# webvtt muxer splits the milliseconds with C's truncating division, so every field carries the
|
||||
# sign — -5.005 s is `00:-5.-05`, -65.3 s is `-1:-5.-300` — and summing the signed fields is exact.
|
||||
parts = ts.split(":")
|
||||
h = "0" if len(parts) == 2 else parts[0]
|
||||
m, rest = parts[-2], parts[-1]
|
||||
@@ -39,16 +84,12 @@ def _vtt_ts_to_s(ts: str) -> float:
|
||||
|
||||
|
||||
def _vtt_s_to_ts(x: float) -> str:
|
||||
x = max(0.0, x)
|
||||
h = int(x // 3600)
|
||||
x -= h * 3600
|
||||
m = int(x // 60)
|
||||
x -= m * 60
|
||||
s = int(x)
|
||||
ms = int(round((x - s) * 1000))
|
||||
if ms >= 1000:
|
||||
s += 1
|
||||
ms -= 1000
|
||||
# Rounded to whole milliseconds FIRST, then split: rounding the fraction last carried a rounded-up
|
||||
# millisecond into the seconds and no further, so 59.9996 came out as `00:00:60.000` — a timestamp
|
||||
# the WebVTT parser rejects, silently dropping that cue.
|
||||
h, rest = divmod(int(round(max(0.0, x) * 1000)), 3_600_000)
|
||||
m, rest = divmod(rest, 60_000)
|
||||
s, ms = divmod(rest, 1000)
|
||||
return f"{h:02d}:{m:02d}:{s:02d}.{ms:03d}"
|
||||
|
||||
|
||||
@@ -109,7 +150,7 @@ def ffmpeg_to_vtt(src: str, sub_ord: int | None) -> str:
|
||||
"webvtt",
|
||||
"pipe:1",
|
||||
]
|
||||
r = subprocess.run(cmd, capture_output=True)
|
||||
r = subprocess.run(cmd, capture_output=True, timeout=EXTRACT_TIMEOUT_S)
|
||||
if r.returncode != 0 or not r.stdout:
|
||||
raise PlexError(
|
||||
(r.stderr or b"").decode("utf-8", "replace")[:200] or "ffmpeg subtitle convert failed"
|
||||
@@ -125,11 +166,686 @@ def cache_read(key: str) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def cache_write(key: str, vtt: str) -> None:
|
||||
def cache_write(key: str, vtt: str) -> bool:
|
||||
"""Atomically replace the cached VTT; False when it could not be written (`publish` must then
|
||||
not claim the region it was carrying)."""
|
||||
try:
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
tmp = CACHE_DIR / f"{key}.vtt.tmp"
|
||||
tmp.write_text(vtt, encoding="utf-8")
|
||||
tmp.replace(CACHE_DIR / f"{key}.vtt") # atomic publish
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
# --- cache identity -----------------------------------------------------------------------------
|
||||
|
||||
# The cached VTT holds ABSOLUTE cue times (per rating_key + subtitle ordinal); the per-session shift
|
||||
# is applied on the way out, so a resume/seek to any offset reuses the same file. Beside it, a small
|
||||
# JSON sidecar records WHICH absolute-time ranges the file actually covers — a windowed extraction
|
||||
# publishes a usable but incomplete track, and "incomplete" must never be mistaken for "all there is".
|
||||
_FULL_END_S = 1_000_000_000.0 # stand-in for "to the end of the file" (JSON can't hold infinity)
|
||||
|
||||
|
||||
def cache_key(rating_key: str, sub_ord: int) -> str:
|
||||
return f"sub_{rating_key}_{sub_ord}"
|
||||
|
||||
|
||||
def _vtt_path(key: str) -> Path:
|
||||
return CACHE_DIR / f"{key}.vtt"
|
||||
|
||||
|
||||
def _meta_path(key: str) -> Path:
|
||||
return CACHE_DIR / f"{key}.json"
|
||||
|
||||
|
||||
def read_meta(key: str) -> tuple[list[list[float]], bool]:
|
||||
"""Covered absolute-time ranges + whether the track is fully extracted. A VTT with no sidecar
|
||||
(written by an older build, which only ever cached complete tracks) counts as complete."""
|
||||
try:
|
||||
raw = json.loads(_meta_path(key).read_text(encoding="utf-8"))
|
||||
return [[float(a), float(b)] for a, b in raw.get("ranges", [])], bool(raw.get("complete"))
|
||||
except (OSError, ValueError, TypeError):
|
||||
return ([[0.0, _FULL_END_S]], True) if _exists(_vtt_path(key)) else ([], False)
|
||||
|
||||
|
||||
def _exists(p: Path) -> bool:
|
||||
"""`Path.exists` that answers False for a path the OS refuses outright: pathlib swallows only
|
||||
ENOENT-like errors, so an over-long name (ENAMETOOLONG, from an absurd rating_key or ordinal)
|
||||
escaped as a 500 from the status route."""
|
||||
try:
|
||||
return p.exists()
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _write_meta(key: str, ranges: list[list[float]], complete: bool) -> None:
|
||||
try:
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
tmp = CACHE_DIR / f"{key}.json.tmp"
|
||||
tmp.write_text(json.dumps({"ranges": ranges, "complete": complete}), encoding="utf-8")
|
||||
tmp.replace(_meta_path(key))
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def generation(key: str) -> int:
|
||||
"""Monotonic id of the cached file's current content. It goes into the <track> URL so that when a
|
||||
window is superseded by a wider one the browser refetches instead of keeping the short VTT."""
|
||||
try:
|
||||
return _vtt_path(key).stat().st_mtime_ns
|
||||
except (OSError, ValueError): # ValueError: a NUL in the key — `stat` refuses the path outright
|
||||
return 0
|
||||
|
||||
|
||||
def merge_ranges(ranges: list[list[float]]) -> list[list[float]]:
|
||||
"""Sort + coalesce touching/overlapping covered ranges."""
|
||||
out: list[list[float]] = []
|
||||
for a, b in sorted((r for r in ranges if r[1] > r[0]), key=lambda r: r[0]):
|
||||
if out and a <= out[-1][1]:
|
||||
out[-1][1] = max(out[-1][1], b)
|
||||
else:
|
||||
out.append([a, b])
|
||||
return out
|
||||
|
||||
|
||||
def covers(ranges: list[list[float]], at: float) -> bool:
|
||||
return any(a <= at <= b for a, b in ranges)
|
||||
|
||||
|
||||
def merge_vtt(*sources: str) -> str:
|
||||
"""Union of the cue blocks of several absolute-time VTTs, ordered by start time.
|
||||
|
||||
Two extraction passes over the same file emit identical cues for the region they share, so they
|
||||
are de-duplicated — but NOT on the raw timing text. The two passes do not spell a timestamp the
|
||||
same way: a window comes back through `shift_vtt`, which always writes `HH:MM:SS.mmm`, while a
|
||||
full pass is ffmpeg's own output, and its webvtt muxer drops the hour below one hour
|
||||
(`10:00.123`). Keyed on the text, every cue of a window in the first hour survived twice and the
|
||||
viewer saw each line stacked on itself. So each timing line is parsed and re-emitted in one
|
||||
canonical form, and the key is that canonical timing PLUS the cue text: two different cues that
|
||||
happen to share a time (two speakers, a sign over dialogue) are both kept."""
|
||||
seen: set[str] = set()
|
||||
cues: list[tuple[float, str]] = []
|
||||
for src in sources:
|
||||
for block in re.split(r"\n[ \t]*\n", src or ""):
|
||||
canon = _canonical_cue(block.strip())
|
||||
if canon is None:
|
||||
continue # header / NOTE / STYLE — the merged file gets its own single header
|
||||
start, key, text = canon
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
cues.append((start, text))
|
||||
cues.sort(key=lambda c: c[0])
|
||||
return "WEBVTT\n\n" + "\n\n".join(b for _, b in cues) + "\n"
|
||||
|
||||
|
||||
def _canonical_cue(block: str) -> tuple[float, str, str] | None:
|
||||
"""(start seconds, identity, the block with its timing line canonicalised) — or None for a block
|
||||
that is not a cue. The identity leaves out anything before the timing line (an SRT-style cue
|
||||
number), which is a label, not content."""
|
||||
lines = block.split("\n")
|
||||
for i, ln in enumerate(lines):
|
||||
if "-->" not in ln:
|
||||
continue
|
||||
left, _, right = ln.partition("-->")
|
||||
rparts = right.strip().split(None, 1)
|
||||
settings = (" " + rparts[1]) if len(rparts) > 1 else ""
|
||||
try:
|
||||
st = _vtt_ts_to_s(left.strip())
|
||||
en = _vtt_ts_to_s(rparts[0])
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
lines[i] = f"{_vtt_s_to_ts(st)} --> {_vtt_s_to_ts(en)}{settings}"
|
||||
return st, "\n".join(lines[i:]), "\n".join(lines)
|
||||
return None
|
||||
|
||||
|
||||
def _last_cue_end(vtt: str) -> float:
|
||||
"""Where the last cue of a VTT ends — how far a pass that died early provably got."""
|
||||
end = 0.0
|
||||
for ln in vtt.split("\n"):
|
||||
if "-->" not in ln:
|
||||
continue
|
||||
try:
|
||||
end = max(end, _vtt_ts_to_s(ln.partition("-->")[2].strip().split(None, 1)[0]))
|
||||
except (ValueError, IndexError):
|
||||
continue
|
||||
return end
|
||||
|
||||
|
||||
def publish(key: str, vtt: str, covered: tuple[float, float], complete: bool) -> bool:
|
||||
"""Merge a freshly extracted region into the cached track and widen its covered ranges. False
|
||||
when the region did not land (the cache is unwritable, or the VTT there cannot be read) — the
|
||||
caller must then record a FAILED pass: returning quietly let the next poll schedule the same read
|
||||
again, a whole-container pass every time, while the player said "preparing" for good.
|
||||
|
||||
A read-merge-write, so it runs under `_PUBLISH_LOCK`: two jobs publish into the same keys (a
|
||||
window and the full pass, and every embedded track of the file rides each of them), and without
|
||||
the lock the later writer drops whatever the earlier one merged — permanently, once `complete` is
|
||||
set. It is also the only writer of the cache files, which is what keeps their fixed `.tmp`
|
||||
staging names safe."""
|
||||
with _PUBLISH_LOCK:
|
||||
old = cache_read(key)
|
||||
if old is None and _vtt_path(key).exists():
|
||||
# There, but unreadable (`cache_read` swallows the OSError). Merging into "" would replace
|
||||
# a whole track with this one region while its sidecar still claimed all of it.
|
||||
log.warning("subtitle cache %s unreadable; not publishing into it", key)
|
||||
return False
|
||||
old_ranges, was_complete = read_meta(key)
|
||||
if old is None:
|
||||
# A sidecar with no VTT behind it claims cues that are not there, so it is not merged
|
||||
# into. And a VTT with no sidecar reads as a COMPLETE track (see read_meta — how older
|
||||
# builds cached), while a publish is two renames: so before a key's first VTT lands, its
|
||||
# sidecar is made to claim nothing. Dying between the two then leaves a window that is
|
||||
# extracted again, never one taken for the whole track for good.
|
||||
old_ranges, was_complete = [], False
|
||||
_write_meta(key, [], False)
|
||||
if not cache_write(key, merge_vtt(old or "", vtt)):
|
||||
log.warning("subtitle cache %s could not be written", key)
|
||||
return False # the region never landed, so the sidecar must not claim it
|
||||
_write_meta(key, merge_ranges(old_ranges + [list(covered)]), complete or was_complete)
|
||||
return True
|
||||
|
||||
|
||||
# --- background extraction ----------------------------------------------------------------------
|
||||
|
||||
# How much of the track to extract before the viewer sees anything. WINDOW_PRE_S starts a little
|
||||
# before the playhead so a cue that is already on screen is not missed; WINDOW_LEN_S is the span
|
||||
# published as the first, provisional VTT (~4 s of work on the measured file). It only has to hold
|
||||
# until the full pass lands behind it — 111 s for that file — so it is deliberately much longer than
|
||||
# any realistic full pass.
|
||||
WINDOW_PRE_S = 30.0
|
||||
WINDOW_LEN_S = 300.0
|
||||
# …but it does not always land: a full pass can be backed off (FULL_RETRY_S) or simply slower than
|
||||
# the window lasts (a file past ~40 GB, or several passes sharing the array). A window was only asked
|
||||
# for once the playhead had LEFT coverage, so every edge unmounted the track for a poll plus an
|
||||
# extraction. Within this distance of a covered stretch's end, the next window is read ahead.
|
||||
LOOKAHEAD_S = 60.0
|
||||
# Ceiling on one ffmpeg run so a wedged extraction can't hold a pool thread forever (the old
|
||||
# in-request extraction had no timeout at all).
|
||||
EXTRACT_TIMEOUT_S = 900.0
|
||||
# How long a timed-out ffmpeg gets, after a TERM, to close its outputs before it is killed.
|
||||
_TERM_GRACE_S = 10.0
|
||||
# A failed track (or window) is remembered briefly so a 1 s status poll doesn't respawn ffmpeg every
|
||||
# second, and forgotten again soon after so fixing the file (or Plex coming back) doesn't need a
|
||||
# restart.
|
||||
FAIL_TTL_S = 60.0
|
||||
# …except a FULL pass that failed. That is a read of the whole container (minutes), a failure that
|
||||
# reproduces does so every time, and the viewer is typically inside what it did publish — so the
|
||||
# short memo re-armed a whole-file read every minute for as long as they watched. Windows around the
|
||||
# playhead still run meanwhile; only the full pass waits this out.
|
||||
FULL_RETRY_S = 3600.0
|
||||
# The furthest position the status poll accepts. No film is this long, and an absurd one used to
|
||||
# reach ffmpeg's `-ss` (which rejects anything past its int64 microseconds).
|
||||
MAX_POSITION_S = 100_000.0
|
||||
# Windows and full passes run on SEPARATE pools, so a window never waits behind a full pass. They
|
||||
# shared one FIFO pool of three whose comment promised "room for a window" that nothing enforced:
|
||||
# stepping through three episodes with subtitles on left three full passes (minutes each) holding every
|
||||
# worker, and a seek window on the third queued behind them. Two of each: bounded, so a burst can't
|
||||
# thrash the array.
|
||||
_WIN_POOL = ThreadPoolExecutor(max_workers=2, thread_name_prefix="plex-subs-win")
|
||||
_FULL_POOL = ThreadPoolExecutor(max_workers=2, thread_name_prefix="plex-subs-full")
|
||||
# Bounds on what may wait for those pools — separate too, so held full-pass slots (minutes each)
|
||||
# cannot refuse a window (seconds). The queues are unbounded and every job is a read of the library:
|
||||
# a viewer scrubbing through a film queued a window per stop (the one where they stopped ran LAST),
|
||||
# and a script could queue thousands. Past a bound nothing is queued — the next poll asks again, by
|
||||
# which time something has finished.
|
||||
_MAX_WINDOWS_INFLIGHT = 8
|
||||
_MAX_FULL_INFLIGHT = 6
|
||||
_MAX_TRACK_WINDOWS = 2
|
||||
|
||||
_LOCK = threading.Lock()
|
||||
# Queued or running jobs → the absolute span a window job reads (None for a full pass).
|
||||
_INFLIGHT: dict[tuple, tuple[float, float] | None] = {}
|
||||
# Track-level failures: a track that cannot be served at all (no item, a bitmap track, Plex down…).
|
||||
_FAILURES: dict[str, tuple[str, float]] = {}
|
||||
# rating_key → when its full pass last failed (see FULL_RETRY_S).
|
||||
_FULL_FAILED: dict[str, float] = {}
|
||||
# cache key → the spans windows failed to read, with when. A failed window says nothing about the rest
|
||||
# of the track — the full pass may be about to deliver it — so it is remembered as a SPAN: it must be
|
||||
# neither respawned every poll nor taken as a failure of positions it never tried. (It used to set the
|
||||
# track-level memo, which told every viewer at any uncovered position "failed" — a final answer that
|
||||
# ends the player's poll — including one planted by a request for an absurd position.)
|
||||
_WIN_FAILED: dict[str, list[tuple[float, float, float]]] = {}
|
||||
_PUBLISH_LOCK = threading.Lock()
|
||||
# The ffmpeg children currently running, so a shutdown can kill them (see `shutdown`), and whether it
|
||||
# has: a worker that reaches its Popen after the kill sweep must not start a read.
|
||||
_PROCS: set[subprocess.Popen] = set()
|
||||
_STOPPING = False
|
||||
|
||||
|
||||
class SubtitleUnavailable(Exception):
|
||||
"""A track that cannot be served as text, carrying the stable code the route maps to a status."""
|
||||
|
||||
def __init__(self, code: str):
|
||||
super().__init__(code)
|
||||
self.code = code
|
||||
|
||||
|
||||
# code -> HTTP status. An extraction/upstream failure is 422 with a structured detail (502 is
|
||||
# reserved for the raw byte proxies); the rest are genuine "no such thing" / "not text".
|
||||
UNAVAILABLE_STATUS = {
|
||||
"no_item": 404,
|
||||
"no_track": 404,
|
||||
"no_file": 404,
|
||||
"not_fetchable": 404,
|
||||
"bitmap": 415,
|
||||
"extract_failed": 422,
|
||||
"plex_error": 422,
|
||||
}
|
||||
# The codes that are a FINAL answer about the track: it can never be served as text, so the player
|
||||
# stops asking. The others (`plex_error`, `extract_failed`) are about now or about one stretch — a
|
||||
# memo that lapses, a window that failed where the rest is fine — and answered as `unavailable`: said,
|
||||
# but asked again. Given as `failed`, one Plex hiccup ended the subtitles for the rest of the session,
|
||||
# at cached positions too, and after the full pass had completed behind it.
|
||||
FINAL_CODES = frozenset({"no_item", "no_track", "no_file", "not_fetchable", "bitmap"})
|
||||
|
||||
|
||||
def reset_state() -> None:
|
||||
"""Drop the in-flight/failure registries (tests; nothing in the app needs it)."""
|
||||
global _STOPPING
|
||||
with _LOCK:
|
||||
_INFLIGHT.clear()
|
||||
_FAILURES.clear()
|
||||
_FULL_FAILED.clear()
|
||||
_WIN_FAILED.clear()
|
||||
_STOPPING = False
|
||||
|
||||
|
||||
def shutdown() -> None:
|
||||
"""Stop extracting, for the app's shutdown. The pool's workers are non-daemon threads and Python
|
||||
joins them at interpreter exit — so a worker still inside a full pass (minutes on a big file, up
|
||||
to EXTRACT_TIMEOUT_S) held the process open after uvicorn had stopped, until the container runtime
|
||||
SIGKILLed it. Queued jobs are dropped and running ffmpeg children are killed, which returns their
|
||||
workers at once — and a worker still BEFORE its ffmpeg (the Plex lookup, the DB) finds the flag
|
||||
and never starts one. Nothing that matters is lost: an interrupted publish cannot pass for a whole
|
||||
track (see `publish`), and an interrupted track is simply extracted again on the next poll."""
|
||||
global _STOPPING
|
||||
with _LOCK:
|
||||
_STOPPING = True
|
||||
procs = list(_PROCS)
|
||||
for pool in (_WIN_POOL, _FULL_POOL):
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
for p in procs:
|
||||
p.kill()
|
||||
|
||||
|
||||
def _sweep_memos() -> None:
|
||||
"""(Caller holds _LOCK.) Drop every lapsed failure memo. They were only ever pruned when the SAME
|
||||
key was read again, and the status route takes any rating_key/ordinal — so a script polling fresh
|
||||
keys grew the API process without bound. Swept on every write, they hold at most a TTL's worth."""
|
||||
now = time.monotonic()
|
||||
for k in [k for k, (_, t) in _FAILURES.items() if now - t >= FAIL_TTL_S]:
|
||||
del _FAILURES[k]
|
||||
for k in [k for k, t in _FULL_FAILED.items() if now - t >= FULL_RETRY_S]:
|
||||
del _FULL_FAILED[k]
|
||||
for k in list(_WIN_FAILED):
|
||||
_live_window_failures(k)
|
||||
|
||||
|
||||
def _fail(key: str, code: str) -> None:
|
||||
with _LOCK:
|
||||
_sweep_memos()
|
||||
_FAILURES[key] = (code, time.monotonic())
|
||||
|
||||
|
||||
def _recent_failure(key: str) -> str | None:
|
||||
with _LOCK:
|
||||
hit = _FAILURES.get(key)
|
||||
if hit and time.monotonic() - hit[1] < FAIL_TTL_S:
|
||||
return hit[0]
|
||||
_FAILURES.pop(key, None)
|
||||
return None
|
||||
|
||||
|
||||
def _full_failed(rating_key: str) -> bool:
|
||||
with _LOCK:
|
||||
when = _FULL_FAILED.get(rating_key)
|
||||
if when is not None and time.monotonic() - when < FULL_RETRY_S:
|
||||
return True
|
||||
_FULL_FAILED.pop(rating_key, None)
|
||||
return False
|
||||
|
||||
|
||||
def _live_window_failures(key: str) -> list[tuple[float, float, float]]:
|
||||
"""(Caller holds _LOCK.) The key's failed-window spans that are still remembered."""
|
||||
now = time.monotonic()
|
||||
live = [f for f in _WIN_FAILED.get(key, []) if now - f[2] < FAIL_TTL_S]
|
||||
if live:
|
||||
_WIN_FAILED[key] = live
|
||||
else:
|
||||
_WIN_FAILED.pop(key, None)
|
||||
return live
|
||||
|
||||
|
||||
def _window_fail(key: str, span: tuple[float, float]) -> None:
|
||||
with _LOCK:
|
||||
_sweep_memos()
|
||||
_WIN_FAILED[key] = [*_live_window_failures(key), (span[0], span[1], time.monotonic())]
|
||||
|
||||
|
||||
def _window_failed(key: str, at: float) -> bool:
|
||||
with _LOCK:
|
||||
return any(a <= at <= b for a, b, _ in _live_window_failures(key))
|
||||
|
||||
|
||||
def ensure(rating_key: str, sub_ord: int, at: float = 0.0) -> dict:
|
||||
"""State of one subtitle track, scheduling whatever extraction is still missing. NEVER blocks.
|
||||
|
||||
``ready`` = fully extracted · ``partial`` = usable, covers ``at``, still growing · ``preparing``
|
||||
= nothing to show for ``at`` yet · ``unavailable`` = nothing for ``at`` and nothing being read
|
||||
for it right now (``code`` says why), ask again · ``failed`` = the track can never be shown
|
||||
(``code`` says why) — the only final answer. ``gen`` changes whenever the cached file grows, so
|
||||
the client can refetch a window that has been superseded; 0 = nothing is cached at all."""
|
||||
key = cache_key(rating_key, sub_ord)
|
||||
ranges, complete = read_meta(key)
|
||||
have = cache_read(key) is not None
|
||||
covered = have and covers(ranges, at)
|
||||
gen = generation(key)
|
||||
if complete and have:
|
||||
return {"state": "ready", "code": None, "gen": gen}
|
||||
if not have and _exists(_vtt_path(key)):
|
||||
# On disk but unreadable (permissions, a bad volume): nothing can serve it, and no extraction
|
||||
# can help — `publish` refuses to merge into it, and a job whose sidecar covers the position
|
||||
# skips as already done — so scheduling only kept the player on "preparing" for good. `gen` 0:
|
||||
# there is nothing a <track> could fetch.
|
||||
return {"state": "unavailable", "code": "cache_unreadable", "gen": 0}
|
||||
code = _recent_failure(key)
|
||||
if code and not covered:
|
||||
return {
|
||||
"state": "failed" if code in FINAL_CODES else "unavailable",
|
||||
"code": code,
|
||||
"gen": gen,
|
||||
}
|
||||
# Nothing left that could cover the position for now: the full pass failed, AND so did a window
|
||||
# here. Still not final — the memos lapse, and the rest of the track may be right there.
|
||||
full_failed = _full_failed(rating_key)
|
||||
window_failed = not covered and _window_failed(key, at)
|
||||
if full_failed and window_failed:
|
||||
return {"state": "unavailable", "code": "extract_failed", "gen": gen}
|
||||
if not code:
|
||||
probe, want_window = at, not covered and not window_failed
|
||||
if covered:
|
||||
# Read ahead of an edge instead of waiting to fall off it (see LOOKAHEAD_S). The probe is
|
||||
# just PAST the edge: `covers` is inclusive, so the edge itself reads as already done.
|
||||
edge = max(b for a, b in ranges if a <= at <= b)
|
||||
if edge - at <= LOOKAHEAD_S and edge < _FULL_END_S:
|
||||
probe = edge + 1.0
|
||||
want_window = not _window_failed(key, probe)
|
||||
_schedule(rating_key, sub_ord, probe, want_window=want_window, want_full=not full_failed)
|
||||
return {"state": "partial" if covered else "preparing", "code": None, "gen": gen}
|
||||
|
||||
|
||||
def cached_track(rating_key: str, sub_ord: int) -> tuple[str | None, str | None]:
|
||||
"""What the <track> fetch can serve RIGHT NOW, scheduling nothing: the cached VTT (whatever it
|
||||
covers), else the recent failure that explains why there is none.
|
||||
|
||||
Scheduling belongs to `ensure` — the status poll, which knows where the viewer is. The fetch does
|
||||
not: when it scheduled too, it judged readiness at position 0, so a viewer who resumed an hour in
|
||||
got an unrequested window over the head of the file (plus a remount when it landed), and with a
|
||||
failure memo alive it was refused a window that WAS on disk while the poll called it showable."""
|
||||
key = cache_key(rating_key, sub_ord)
|
||||
hit = cache_read(key)
|
||||
return (hit, None) if hit is not None else (None, _recent_failure(key))
|
||||
|
||||
|
||||
def _schedule(
|
||||
rating_key: str, sub_ord: int, at: float, want_window: bool, want_full: bool = True
|
||||
) -> None:
|
||||
"""Kick off the jobs this track still needs: the window that makes it watchable now, and the one
|
||||
full pass that makes it complete. A 1 s status poll (or a second viewer) joins a job already
|
||||
queued or running instead of starting another ffmpeg.
|
||||
|
||||
A window is joined when one of this track's in-flight windows will COVER the position asked about
|
||||
— decided by what it reads, not by an identity derived from the position. The poll sends the live
|
||||
playhead, so a position-keyed window was a new job every second; bucketing the position instead
|
||||
still queued a second read when the playhead crossed a bucket edge, and joined a backward seek to
|
||||
a window that could not cover it. Nor may it be one window per track: a window cut around the
|
||||
wrong spot (the first poll after a resume can still report 0) made the right one wait for it. The
|
||||
full pass is per ITEM, because one read fans out to every embedded text track of the file."""
|
||||
jobs: list[tuple[tuple, float | None]] = []
|
||||
with _LOCK:
|
||||
if _STOPPING:
|
||||
return
|
||||
n_windows = sum(1 for span in _INFLIGHT.values() if span is not None)
|
||||
n_full = len(_INFLIGHT) - n_windows
|
||||
if want_window and n_windows < _MAX_WINDOWS_INFLIGHT:
|
||||
spans = [
|
||||
span
|
||||
for ident, span in _INFLIGHT.items()
|
||||
if span is not None and ident[1:3] == (rating_key, sub_ord)
|
||||
]
|
||||
if not any(a <= at <= b for a, b in spans) and len(spans) < _MAX_TRACK_WINDOWS:
|
||||
# Rounded to what ffmpeg is told (`-ss %.3f`): the cues come back on ffmpeg's zero,
|
||||
# and shifting them by an unrounded start put them up to half a millisecond off —
|
||||
# enough to spell a timestamp differently from the full pass and double the cue.
|
||||
w_start = float(f"{max(0.0, at - WINDOW_PRE_S):.3f}")
|
||||
ident = ("win", rating_key, sub_ord, w_start)
|
||||
_INFLIGHT[ident] = (w_start, w_start + WINDOW_PRE_S + WINDOW_LEN_S)
|
||||
jobs.append((ident, w_start))
|
||||
if want_full and n_full < _MAX_FULL_INFLIGHT and ("full", rating_key) not in _INFLIGHT:
|
||||
_INFLIGHT[("full", rating_key)] = None
|
||||
jobs.append((("full", rating_key), None))
|
||||
for ident, window_start in jobs:
|
||||
pool = _FULL_POOL if window_start is None else _WIN_POOL
|
||||
pool.submit(_run, ident, rating_key, sub_ord, window_start, at)
|
||||
|
||||
|
||||
def _already_done(key: str, window_start: float | None, at: float) -> bool:
|
||||
"""A queued job looks again before it reads anything. While it waited, another job may have
|
||||
covered it: every embedded track rides each pass over its file, so a window for one language also
|
||||
publishes the others."""
|
||||
if not _exists(_vtt_path(key)):
|
||||
return False
|
||||
ranges, complete = read_meta(key)
|
||||
return complete or (window_start is not None and covers(ranges, at))
|
||||
|
||||
|
||||
def _run(
|
||||
ident: tuple, rating_key: str, sub_ord: int, window_start: float | None, at: float
|
||||
) -> None:
|
||||
key = cache_key(rating_key, sub_ord)
|
||||
try:
|
||||
if _STOPPING or _already_done(key, window_start, at):
|
||||
return
|
||||
_extract_and_publish(rating_key, sub_ord, window_start)
|
||||
except SubtitleUnavailable as e:
|
||||
if e.code == "extract_failed":
|
||||
_pass_failed(rating_key, [key], window_start)
|
||||
else:
|
||||
_fail(key, e.code)
|
||||
except PlexError as e:
|
||||
log.warning("subtitle extraction failed for %s/%s: %s", rating_key, sub_ord, e)
|
||||
_fail(key, "plex_error")
|
||||
except (
|
||||
Exception
|
||||
): # a worker thread has nobody to raise to — record it, let the next poll see it
|
||||
log.exception("subtitle extraction crashed for %s/%s", rating_key, sub_ord)
|
||||
_pass_failed(rating_key, [key], window_start)
|
||||
finally:
|
||||
with _LOCK:
|
||||
_INFLIGHT.pop(ident, None)
|
||||
|
||||
|
||||
def _pass_failed(
|
||||
rating_key: str,
|
||||
keys: list[str],
|
||||
window_start: float | None,
|
||||
span: tuple[float, float] | None = None,
|
||||
) -> None:
|
||||
"""Record an ffmpeg pass that failed: a full pass backs off the whole item (FULL_RETRY_S); a
|
||||
window marks only the stretch it failed to read, for the tracks it was reading."""
|
||||
if window_start is None:
|
||||
with _LOCK:
|
||||
_sweep_memos()
|
||||
_FULL_FAILED[rating_key] = time.monotonic()
|
||||
return
|
||||
for key in keys:
|
||||
_window_fail(key, span or (window_start, window_start + WINDOW_PRE_S + WINDOW_LEN_S))
|
||||
|
||||
|
||||
def _text_streams(db: Session, rating_key: str) -> tuple[PlexItem, list[dict]]:
|
||||
"""The item plus its subtitle streams, in the ordinal order the API and the UI both use."""
|
||||
it = by_rating_key(db, PlexItem, rating_key)
|
||||
if it is None:
|
||||
raise SubtitleUnavailable("no_item")
|
||||
with PlexClient(db) as plex:
|
||||
part = ((plex.metadata(it.rating_key) or {}).get("Media") or [{}])[0].get("Part") or [{}]
|
||||
return it, [s for s in (part[0].get("Stream") or []) if s.get("streamType") == 3]
|
||||
|
||||
|
||||
def _is_bitmap(stream: dict) -> bool:
|
||||
return (stream.get("codec") or stream.get("format") or "").lower() in BITMAP_SUBS
|
||||
|
||||
|
||||
def _extract_and_publish(rating_key: str, sub_ord: int, window_start: float | None) -> None:
|
||||
"""One extraction job. A sidecar track is fetched whole from Plex (cheap, no ffmpeg); an embedded
|
||||
one is extracted together with EVERY other embedded text track in the file, because they all ride
|
||||
on the same single read — the whole point of the measurement at the top of this module."""
|
||||
with SessionLocal() as db:
|
||||
it, subs = _text_streams(db, rating_key)
|
||||
if sub_ord < 0 or sub_ord >= len(subs):
|
||||
raise SubtitleUnavailable("no_track")
|
||||
want = subs[sub_ord]
|
||||
if _is_bitmap(want):
|
||||
raise SubtitleUnavailable("bitmap")
|
||||
if want.get("key"):
|
||||
# An external sidecar is one small fetch from Plex, never a read of the media file, so a
|
||||
# window job fetches the whole thing too. It must not be left to the full job: that
|
||||
# identity is per ITEM (the embedded fan-out shares it), so a sidecar picked while an
|
||||
# embedded track's full pass was running sat behind that pass for minutes.
|
||||
vtt = _sidecar_vtt(db, want)
|
||||
if not publish(cache_key(rating_key, sub_ord), vtt, (0.0, _FULL_END_S), True):
|
||||
raise SubtitleUnavailable("extract_failed")
|
||||
return
|
||||
if want.get("index") is None:
|
||||
raise SubtitleUnavailable("not_fetchable")
|
||||
src = plex_paths.local_media_path(db, it.file_path)
|
||||
if src is None:
|
||||
raise SubtitleUnavailable("no_file")
|
||||
# Ordinal among the file's EMBEDDED subtitle streams — what `-map 0:s:N` counts. Sidecars sit
|
||||
# in the same list but not in the file, so they must not shift the mapping.
|
||||
targets = {
|
||||
sum(1 for x in subs[:i] if x.get("index") is not None): i
|
||||
for i, s in enumerate(subs)
|
||||
if s.get("index") is not None and not _is_bitmap(s)
|
||||
}
|
||||
start = window_start or 0.0
|
||||
length = (WINDOW_LEN_S + WINDOW_PRE_S) if window_start is not None else None
|
||||
end = _FULL_END_S if length is None else start + length
|
||||
texts, clean = _ffmpeg_extract(str(src), sorted(targets), window_start, length)
|
||||
for emb_ord, vtt in texts.items():
|
||||
key = cache_key(rating_key, targets[emb_ord])
|
||||
# A pass that exited non-zero holds only what ffmpeg reached before it died. Claim no more
|
||||
# than its last cue, never call it complete, and remember the failure: published as complete,
|
||||
# the track read "ready" for good while its subtitles stopped wherever ffmpeg did (the cache
|
||||
# has no eviction, so nothing would ever have extracted it again).
|
||||
covered_end = end if clean else start + _last_cue_end(vtt)
|
||||
landed = publish(
|
||||
key,
|
||||
shift_vtt(vtt, -start), # zero-based window clock → absolute cue times
|
||||
(start, covered_end),
|
||||
clean and length is None,
|
||||
)
|
||||
if not landed:
|
||||
# A read that could not be kept is a failed read (see publish) — backed off like one.
|
||||
_pass_failed(rating_key, [key], window_start, (start, end))
|
||||
elif not clean:
|
||||
_pass_failed(rating_key, [key], window_start, (covered_end, end))
|
||||
|
||||
|
||||
def _sidecar_vtt(db: Session, stream: dict) -> str:
|
||||
"""An external subtitle file, fetched through Plex and converted to WebVTT."""
|
||||
with PlexClient(db) as plex:
|
||||
raw = plex.raw_get(stream["key"])
|
||||
codec = (stream.get("codec") or stream.get("format") or "").lower()
|
||||
if codec in ("srt", "subrip", ""):
|
||||
return srt_to_vtt(raw.decode("utf-8", "replace"))
|
||||
with tempfile.NamedTemporaryFile(suffix="." + (codec or "sub"), delete=False) as fh:
|
||||
fh.write(raw)
|
||||
tmp = fh.name
|
||||
try:
|
||||
return ffmpeg_to_vtt(tmp, None)
|
||||
finally:
|
||||
Path(tmp).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def _ffmpeg_extract(
|
||||
src: str, emb_ords: list[int], start: float | None, length: float | None
|
||||
) -> tuple[dict[int, str], bool]:
|
||||
"""Extract several embedded subtitle streams to WebVTT in ONE ffmpeg run (= one read of the file).
|
||||
Returns the texts and whether ffmpeg exited cleanly — a non-zero exit can still leave partial
|
||||
output, which is worth publishing but must never be taken for the whole track.
|
||||
|
||||
`start`/`length` bound the read to a window. Without `-copyts` the cues come out on a zero-based
|
||||
clock (the caller shifts them back) — which is exactly why `-copyts` is absent: with it ffmpeg
|
||||
ignores `-t` and reads the whole container anyway (measured)."""
|
||||
if not emb_ords:
|
||||
return {}, True
|
||||
with tempfile.TemporaryDirectory(prefix="plexsub-") as tmpdir:
|
||||
cmd = ["ffmpeg", "-nostdin", "-loglevel", "error"]
|
||||
if start:
|
||||
cmd += ["-ss", f"{start:.3f}"]
|
||||
cmd += ["-i", src]
|
||||
outs = {o: Path(tmpdir) / f"{o}.vtt" for o in emb_ords}
|
||||
for emb_ord, out in outs.items():
|
||||
cmd += ["-map", f"0:s:{emb_ord}"]
|
||||
if length is not None:
|
||||
cmd += ["-t", f"{length:.3f}"]
|
||||
# The input seek lands on the keyframe at or BEFORE `start`, so a cue already running
|
||||
# at `start` comes out at a negative time — and by default the muxer then shifts the
|
||||
# WHOLE output so that cue sits at 0: every cue of the window late by the distance to
|
||||
# that keyframe (1–5 s measured on a real library file, a whole GOP on a synthetic
|
||||
# one), and doubled for good once the full pass merged in beside it. Kept negative, the
|
||||
# times are exact on the zero-based clock (`_vtt_ts_to_s` reads the sign).
|
||||
cmd += ["-avoid_negative_ts", "disabled"]
|
||||
cmd += ["-f", "webvtt", "-y", str(out)]
|
||||
# Popen rather than run(): the child is registered so `shutdown` can kill it — and one that
|
||||
# would start after the shutdown's kill sweep never does, or is killed right here.
|
||||
with _LOCK:
|
||||
if _STOPPING:
|
||||
raise SubtitleUnavailable("extract_failed")
|
||||
proc = subprocess.Popen(
|
||||
cmd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE
|
||||
)
|
||||
with _LOCK:
|
||||
_PROCS.add(proc)
|
||||
if _STOPPING:
|
||||
proc.kill()
|
||||
timed_out = False
|
||||
try:
|
||||
try:
|
||||
_, err = proc.communicate(timeout=EXTRACT_TIMEOUT_S)
|
||||
except subprocess.TimeoutExpired:
|
||||
# A pass that ran out of time is a pass that died partway: what it reached is kept and
|
||||
# published as partial (the caller's unclean path), like any other non-zero exit. It
|
||||
# used to be killed and its outputs deleted unread — and with the full-pass backoff, the
|
||||
# same doomed read came back every hour. TERM first: ffmpeg closes its outputs on it,
|
||||
# whereas a KILL leaves only what was already flushed in whole blocks (measured: none).
|
||||
timed_out = True
|
||||
proc.terminate()
|
||||
try:
|
||||
_, err = proc.communicate(timeout=_TERM_GRACE_S)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
_, err = proc.communicate()
|
||||
finally:
|
||||
with _LOCK:
|
||||
_PROCS.discard(proc)
|
||||
got = {
|
||||
o: p.read_text(encoding="utf-8", errors="replace")
|
||||
for o, p in outs.items()
|
||||
if p.exists()
|
||||
}
|
||||
if not got:
|
||||
raise SubtitleUnavailable("extract_failed")
|
||||
clean = proc.returncode == 0 and not timed_out
|
||||
if not clean:
|
||||
log.warning(
|
||||
"ffmpeg subtitle pass %s: %s",
|
||||
"timed out" if timed_out else f"exited {proc.returncode}",
|
||||
(err or b"")[:200],
|
||||
)
|
||||
return got, clean
|
||||
|
||||
+130
-3
@@ -18,7 +18,7 @@ item-detail/player path (P2), not here.
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import sysconfig
|
||||
@@ -58,6 +58,57 @@ def classify_playable(vcodec: str | None, acodec: str | None, container: str | N
|
||||
return "direct"
|
||||
|
||||
|
||||
# Video codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for
|
||||
# the full-GPU VAAPI transcode pipeline. THE single source of this set: plex.stream imports it here so
|
||||
# the runtime encoder choice (`_want_hw`) and this catalog facet can never disagree by codec. A source
|
||||
# outside it (AV1, MPEG-4 ASP/XviD…) is software-only even on the iGPU (it can't decode it to feed its
|
||||
# own encoder).
|
||||
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"})
|
||||
# Color-transfer values marking an HDR source (PQ / HLG). A 10-bit/HDR source breaks the VAAPI
|
||||
# scale=nv12 path on Gen 9.5 (see plex.stream) → software transcode, so it counts as NOT HW-capable.
|
||||
_HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
|
||||
|
||||
|
||||
def transcode_hw_capable(
|
||||
playable: str | None,
|
||||
vcodec: str | None,
|
||||
bit_depth: int | None = None,
|
||||
hdr: bool = False,
|
||||
) -> bool | None:
|
||||
"""The stored `PlexItem.transcode_hw` facet: for a `playable="transcode"` item, whether the prod
|
||||
iGPU can HARDWARE-transcode it (else software libx264 only); None for a non-transcode item. HW needs
|
||||
a HW-decodable codec AND an 8-bit SDR source — a 10-bit/HDR source (bit_depth>8 or an HDR transfer)
|
||||
breaks the VAAPI nv12 pipeline and falls back to software. `bit_depth`/`hdr` are unknown at sync
|
||||
time (the cheap listing carries no Stream), so they default to 8-bit SDR (codec-only) and the
|
||||
bit-depth backfill refines a 10-bit/HDR HEVC to False later."""
|
||||
if playable != "transcode":
|
||||
return None
|
||||
if (vcodec or "").lower() not in HW_DECODABLE_CODECS:
|
||||
return False
|
||||
if bit_depth is not None and bit_depth > 8:
|
||||
return False
|
||||
return not hdr
|
||||
|
||||
|
||||
def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
|
||||
"""(bit_depth, is_hdr) from a FULL Plex metadata dict's video Stream (Media→Part→Stream, streamType
|
||||
1). Only the rich metadata call carries Stream; the cheap listing does not — so this is read by the
|
||||
bit-depth backfill, not the listing sync. (None, False) when absent/unparseable."""
|
||||
for m in meta.get("Media") or []:
|
||||
for part in m.get("Part") or []:
|
||||
for s in part.get("Stream") or []:
|
||||
if s.get("streamType") != 1: # 1 = video
|
||||
continue
|
||||
bd = s.get("bitDepth")
|
||||
try:
|
||||
depth = int(bd) if bd else None
|
||||
except (TypeError, ValueError):
|
||||
depth = None
|
||||
hdr = str(s.get("colorTrc") or "").lower() in _HDR_TRANSFERS
|
||||
return depth, hdr
|
||||
return None, False
|
||||
|
||||
|
||||
def _epoch(v) -> datetime | None:
|
||||
try:
|
||||
return datetime.fromtimestamp(int(v), tz=timezone.utc) if v else None
|
||||
@@ -217,6 +268,13 @@ def sync(db: Session) -> dict:
|
||||
stats["people_enriched"] = _enrich_people_batch(db, plex)
|
||||
except Exception as e: # best-effort — never fail the sync over the backfill
|
||||
log.warning("Plex people enrich batch failed: %s", e)
|
||||
# Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
|
||||
# playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
|
||||
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
|
||||
try:
|
||||
stats["media_probed"] = _probe_media_batch(db, plex)
|
||||
except Exception as e: # best-effort — never fail the sync over the backfill
|
||||
log.warning("Plex media probe batch failed: %s", e)
|
||||
except PlexNotConfigured as e:
|
||||
return {"skipped": str(e)}
|
||||
except PlexError as e:
|
||||
@@ -236,6 +294,49 @@ def sync(db: Session) -> dict:
|
||||
# How many titles to rich-fetch per sync run for the cast backfill — bounds the extra Plex load; the
|
||||
# whole library is covered over successive runs (each fetched item is stamped people_enriched_at once).
|
||||
_ENRICH_BATCH = 60
|
||||
# Same bounded-per-run pattern for the bit-depth probe. Only HW-decodable-codec transcode items are
|
||||
# probed (typically a small minority — the bulk of transcodes are old codecs that are SW by codec), so
|
||||
# a modest batch covers the work-list in a run or two; a stuck item retries next run (watermark unset).
|
||||
_PROBE_BATCH = 60
|
||||
|
||||
|
||||
def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH) -> int:
|
||||
"""Rich-fetch up to `limit` not-yet-probed HW-decodable transcode items, read the video Stream's
|
||||
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit/HDR HEVC →
|
||||
False/software). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
|
||||
so an item is probed once and not re-fetched every sync; a transient fetch FAILURE leaves the
|
||||
watermark unset to retry. Oldest-id first. Returns how many items were probed this run."""
|
||||
rows = (
|
||||
db.execute(
|
||||
select(PlexItem)
|
||||
.where(
|
||||
PlexItem.playable == "transcode",
|
||||
func.lower(PlexItem.codec_video).in_(HW_DECODABLE_CODECS),
|
||||
PlexItem.media_probed_at.is_(None),
|
||||
)
|
||||
.order_by(PlexItem.id)
|
||||
.limit(limit)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
now = datetime.now(timezone.utc)
|
||||
n = 0
|
||||
for row in rows:
|
||||
try:
|
||||
meta = plex.metadata(row.rating_key) or {}
|
||||
except Exception as e: # transient — leave media_probed_at NULL so it retries next run
|
||||
log.debug("Plex media probe fetch failed for %s: %s", row.rating_key, e)
|
||||
continue
|
||||
bit_depth, hdr = _video_stream_props(meta)
|
||||
row.video_bit_depth = bit_depth
|
||||
row.transcode_hw = transcode_hw_capable("transcode", row.codec_video, bit_depth, hdr)
|
||||
row.media_probed_at = (
|
||||
now # probed (even if Plex carried no Stream — don't re-fetch forever)
|
||||
)
|
||||
n += 1
|
||||
db.commit()
|
||||
return n
|
||||
|
||||
|
||||
def _enrich_people_batch(db: Session, plex: PlexClient, limit: int = _ENRICH_BATCH) -> int:
|
||||
@@ -369,15 +470,41 @@ def _apply_item(row: PlexItem, lib_id: int, kind: str, meta: dict) -> None:
|
||||
row.added_at = _epoch(meta.get("addedAt"))
|
||||
_apply_facet_fields(row, meta) # filterable/orderable extras (shared with shows)
|
||||
mf = _media_facts(meta)
|
||||
new_vcodec = mf.get("codec_video")
|
||||
new_part_key = mf.get("part_key")
|
||||
# The physical file was replaced when its video codec changes OR — for a SINGLE-version item — when its
|
||||
# Plex Part key changes (a re-encode / quality upgrade gets a new part id). Either way any probed
|
||||
# bit-depth is stale, so reset the probe watermark and re-derive transcode_hw from codec alone until the
|
||||
# backfill re-probes. Codec alone is NOT enough: a same-codec bit-depth re-encode (8-bit→10-bit HEVC)
|
||||
# keeps the codec `hevc` but flips the HW class, and only the part-key move catches it.
|
||||
#
|
||||
# The part-key check is gated on single-version because `_media_facts` reads media[0]/part[0], and for a
|
||||
# MULTI-version item (an original + a Plex "Optimized" copy) that order isn't guaranteed stable across
|
||||
# listings — an unguarded part-key compare would then reset+re-probe every sync and oscillate the
|
||||
# transcode_hw bucket between the two versions. A multi-version same-codec re-encode not self-healing is
|
||||
# an acceptable rare edge; a per-sync re-probe loop is not.
|
||||
single_version = len(meta.get("Media") or []) <= 1
|
||||
codec_changed = (row.codec_video or "").lower() != (new_vcodec or "").lower()
|
||||
if codec_changed or (single_version and row.part_key != new_part_key):
|
||||
row.video_bit_depth = None
|
||||
row.media_probed_at = None
|
||||
row.file_path = mf.get("file_path")
|
||||
row.part_key = mf.get("part_key")
|
||||
row.part_key = new_part_key
|
||||
row.container = mf.get("container")
|
||||
row.codec_video = mf.get("codec_video")
|
||||
row.codec_video = new_vcodec
|
||||
row.codec_audio = mf.get("codec_audio")
|
||||
row.size_bytes = mf.get("size_bytes")
|
||||
row.playable = classify_playable(
|
||||
mf.get("codec_video"), mf.get("codec_audio"), mf.get("container")
|
||||
)
|
||||
# transcode_hw: keep the bit-depth backfill's authoritative value (it folds in bit-depth + HDR) once
|
||||
# probed; otherwise (re)derive the provisional codec-only class. A non-transcode item is always None,
|
||||
# even if it had been probed as a transcode before (an audio/container flip to remux without a codec
|
||||
# change wouldn't reset the watermark, so pin it here).
|
||||
if row.playable != "transcode":
|
||||
row.transcode_hw = None
|
||||
elif row.media_probed_at is None:
|
||||
row.transcode_hw = transcode_hw_capable("transcode", row.codec_video, row.video_bit_depth)
|
||||
|
||||
|
||||
def _prune_stale(
|
||||
|
||||
@@ -18,6 +18,13 @@ class RateLimiter:
|
||||
self._hits: dict[str, list[float]] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Forget every recorded attempt. Each limiter is a module-level singleton, so a test suite
|
||||
sharing one process would otherwise pool all of its attempts into a single window and
|
||||
eventually trip an unrelated test with a 429."""
|
||||
with self._lock:
|
||||
self._hits.clear()
|
||||
|
||||
def allow(self, key: str) -> bool:
|
||||
"""Record an attempt for ``key`` and return whether it is within the limit.
|
||||
|
||||
|
||||
@@ -17,12 +17,20 @@ from app.models import User
|
||||
M = TypeVar("M", bound=Base)
|
||||
|
||||
|
||||
def get_or_404(db: Session, model: type[M], id_: Any, *, detail: str = "Not found") -> M:
|
||||
def not_found(what: str = "generic") -> HTTPException:
|
||||
"""The 404 for a row that is gone — structured, so the bilingual UI says it in the reader's
|
||||
language (`errors.notFound.<what>`, generic fallback). This is the 404 a user actually meets: the
|
||||
thing was deleted in another tab or on another device. It used to be a bare English string
|
||||
("Unknown view", "Not found"), which the Hungarian UI toasted verbatim."""
|
||||
return HTTPException(status_code=404, detail={"code": "not_found", "reason": what})
|
||||
|
||||
|
||||
def get_or_404(db: Session, model: type[M], id_: Any, *, what: str = "generic") -> M:
|
||||
"""Fetch a row by primary key or raise 404. Replaces the `db.get(Model, id); if None: 404`
|
||||
idiom repeated across routes."""
|
||||
obj = db.get(model, id_)
|
||||
if obj is None:
|
||||
raise HTTPException(status_code=404, detail=detail)
|
||||
raise not_found(what)
|
||||
return obj
|
||||
|
||||
|
||||
@@ -33,13 +41,13 @@ def owned_or_404(
|
||||
user: User,
|
||||
*,
|
||||
owner_attr: str = "user_id",
|
||||
detail: str = "Not found",
|
||||
what: str = "generic",
|
||||
) -> M:
|
||||
"""Fetch a row by primary key that `user` owns, or raise 404. A row owned by someone
|
||||
else 404s (NOT 403) on purpose — a 403 would confirm the id exists to a stranger.
|
||||
Generalizes the near-identical per-route ownership helpers (e.g. plex's
|
||||
`_own_playlist_or_404`, the saved-views owner check)."""
|
||||
obj = get_or_404(db, model, id_, detail=detail)
|
||||
obj = get_or_404(db, model, id_, what=what)
|
||||
if getattr(obj, owner_attr) != user.id:
|
||||
raise HTTPException(status_code=404, detail=detail)
|
||||
raise not_found(what)
|
||||
return obj
|
||||
|
||||
+466
-31
@@ -1,20 +1,26 @@
|
||||
"""Admin-only onboarding: review access requests (Invites), approve/deny, manual add.
|
||||
Also the demo-account admin surface: the demo email whitelist + a manual state reset."""
|
||||
|
||||
from collections.abc import Iterable
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import audit
|
||||
from app import email as email_mod
|
||||
from app import mailtext
|
||||
from app.audit import AuditAction
|
||||
from app.auth import (
|
||||
admin_user,
|
||||
can_activate,
|
||||
count_admins,
|
||||
env_allowed_canonicals,
|
||||
find_user_by_email,
|
||||
get_or_create_demo_user,
|
||||
is_allowed,
|
||||
operator_contact,
|
||||
purge_user,
|
||||
)
|
||||
@@ -25,13 +31,47 @@ from app.models import (
|
||||
Invite,
|
||||
User,
|
||||
)
|
||||
from app.routes.auth import revoke_preset_password_if_unverified
|
||||
from app.utils import iso, valid_email
|
||||
from app.routes._common import get_or_404
|
||||
from app.routes.auth import (
|
||||
APPROVAL_VERIFY_TTL,
|
||||
issue_set_password_link,
|
||||
issue_verification_link,
|
||||
send_verification_link,
|
||||
)
|
||||
from app.utils import canonical_email, iso, valid_email
|
||||
|
||||
router = APIRouter(prefix="/api/admin", tags=["admin"])
|
||||
|
||||
|
||||
def _serialize(inv: Invite) -> dict:
|
||||
def _accounts_for(db: Session, emails: Iterable[str]) -> set[str]:
|
||||
"""Which of these addresses already have an account, as a set of CANONICAL forms. Batched, so
|
||||
listing the invites costs one extra query. An invite whose address has an account must not be
|
||||
deletable: `is_allowed` reads this row, so removing it would silently revoke a live user's access.
|
||||
|
||||
Canonical on both sides, because `is_allowed` matches that way: an invite for `x.y@gmail.com`
|
||||
whose account registered as `xy@gmail.com` would otherwise look account-less, offer the admin a
|
||||
delete button, and take away a live user's access — precisely the accident this guard exists to
|
||||
prevent."""
|
||||
wanted = {canonical_email(e) for e in emails if e}
|
||||
if not wanted:
|
||||
return set()
|
||||
rows = db.execute(
|
||||
select(User.email_canonical).where(User.email_canonical.in_(wanted))
|
||||
).scalars()
|
||||
return {e for e in rows if e}
|
||||
|
||||
|
||||
def _env_allowed(email: str) -> bool:
|
||||
"""Whether the env bootstrap lists still let this address in. `is_allowed` consults them only
|
||||
when no Invite row exists, so this is exactly what deleting the row would fall back to — and the
|
||||
admin has to know that BEFORE they confirm, not from a toast afterwards.
|
||||
|
||||
Canonical, because `is_allowed` is: a warning that disagrees with the gate it is warning about is
|
||||
worse than no warning."""
|
||||
return canonical_email(email or "") in env_allowed_canonicals()
|
||||
|
||||
|
||||
def _serialize(inv: Invite, *, has_account: bool = False) -> dict:
|
||||
return {
|
||||
"id": inv.id,
|
||||
"email": inv.email,
|
||||
@@ -40,6 +80,19 @@ def _serialize(inv: Invite) -> dict:
|
||||
"requested_at": iso(inv.requested_at),
|
||||
"decided_at": iso(inv.decided_at),
|
||||
"decided_by": inv.decided_by,
|
||||
# When (if ever) the admin sent this person an invitation. Adding an address is silent, so
|
||||
# without this the admin has no way to answer "did I already write to them?" — and the only
|
||||
# failure mode is writing to a real person twice.
|
||||
"invited_at": iso(inv.invited_at),
|
||||
# Drives the delete affordance: an entry backed by an account is not removable here.
|
||||
"has_account": has_account,
|
||||
# …and whether removing it would actually take access away, or leave the env allow-list
|
||||
# granting it anyway (for an ADMIN_EMAILS address, as an admin).
|
||||
"still_allowed_by_env": _env_allowed(inv.email),
|
||||
# The language this person asked us in, or null when nobody ever told us (a hand-added
|
||||
# whitelist row). The invitation dialog preselects it, and falls back to the admin's own UI
|
||||
# language — a guess, but a visible one the admin can correct before sending.
|
||||
"lang": inv.lang,
|
||||
}
|
||||
|
||||
|
||||
@@ -56,26 +109,71 @@ def list_invites(
|
||||
rows = db.execute(stmt.order_by(Invite.requested_at.desc())).scalars().all()
|
||||
order = {"pending": 0, "approved": 1, "denied": 2}
|
||||
rows.sort(key=lambda i: order.get(i.status, 9))
|
||||
return [_serialize(i) for i in rows]
|
||||
accounts = _accounts_for(db, (i.email for i in rows))
|
||||
return [_serialize(i, has_account=canonical_email(i.email) in accounts) for i in rows]
|
||||
|
||||
|
||||
def _welcome_mail_parts(db: Session, email: str) -> tuple[str | None, bool, int]:
|
||||
"""What a welcome or invitation mail should tell this address to do next: a confirmation link (if
|
||||
there is an unconfirmed account behind it), whether an account exists at all, and how long the
|
||||
link lasts.
|
||||
|
||||
This is the C option's whole mechanic. In the common ordering — the address is approved and the
|
||||
person signs up either side of that — they would otherwise have to find and open TWO messages:
|
||||
one saying they are in, another asking them to confirm. Carrying the link inside the mail that is
|
||||
already going out makes it one mail and one click.
|
||||
|
||||
Resolved through `find_user_by_email`, so an account registered under another spelling of the
|
||||
same Gmail mailbox is still recognised — and, deliberately, so several candidates resolve to
|
||||
"no single answer" rather than a guess: an invitation that carried the wrong account's link would
|
||||
be worse than one that simply pointed at the sign-up page.
|
||||
"""
|
||||
hours = int(APPROVAL_VERIFY_TTL.total_seconds() // 3600)
|
||||
user = find_user_by_email(db, email)
|
||||
if user is None or user.is_demo:
|
||||
return None, False, hours
|
||||
if user.email_verified or not email_mod.email_enabled(db):
|
||||
return None, True, hours
|
||||
return issue_verification_link(db, user, APPROVAL_VERIFY_TTL), True, hours
|
||||
|
||||
|
||||
def _activate_user_by_email(db: Session, email: str) -> None:
|
||||
"""Approving access activates a matching (e.g. pending password-registration) account so
|
||||
it can sign in. No-op if no such user exists yet (a Google user is created at first login)."""
|
||||
user = db.execute(select(User).where(User.email == email.lower())).scalar_one_or_none()
|
||||
it can sign in. No-op if no such user exists yet (a Google user is created at first login).
|
||||
|
||||
E9.5 S1: this path deliberately does NOT revoke a preset password. Revoking here destroyed the
|
||||
password a legitimate registrant had just chosen and made "your password was cleared for
|
||||
security" the DEFAULT experience, because approve-before-verify is the normal ordering.
|
||||
|
||||
S1 justified that by saying `password_login` refuses an unverified account BEFORE it looks at
|
||||
`is_active`, so a pre-emptor could not sign in "approved or not". A review of the finished epic
|
||||
showed that argument only holds while the address stays unverified — and S6's approval mail
|
||||
hands the real owner a confirmation link precisely so they will end that state. Nothing replaced
|
||||
the revoke, and that is deliberate: round 2 of the same review tried "the newest registration
|
||||
owns an unverified account" and reverted it, because the rule is symmetric — an attacker can
|
||||
always register last — and it handed anonymous callers a write onto a real account's credential.
|
||||
The owner's recourse is password-reset, which proves the mailbox, rewrites the hash and bumps
|
||||
the session epoch. The residual is logged as SEC-16. The revoke also stays on the Google-adoption path
|
||||
(auth.revoke_preset_password_if_unverified), where ownership is actually attested."""
|
||||
# Canonical, with the literal spelling preferred when both exist: approving
|
||||
# `npeter83.temu@gmail.com` must activate the account that registered as `npeter83temu@gmail.com`,
|
||||
# since Gmail delivers both to one mailbox and the admin decided about the mailbox.
|
||||
email = email.lower()
|
||||
candidates = (
|
||||
db.execute(
|
||||
select(User).where(User.email_canonical == canonical_email(email)).order_by(User.id)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
user = next((u for u in candidates if u.email == email), None) or next(iter(candidates), None)
|
||||
if user is not None and not user.is_active:
|
||||
# SEC-10: same guard as the Google-adoption path — approving an account whose email was never
|
||||
# verified must not hand a working login to a password set on it by whoever registered the
|
||||
# address (a pre-emption). Shared single-source helper so the two paths can't drift.
|
||||
revoke_preset_password_if_unverified(user)
|
||||
user.is_active = True
|
||||
db.commit()
|
||||
|
||||
|
||||
def _decide(db: Session, invite_id: int, admin: User, approved: bool) -> Invite:
|
||||
inv = db.get(Invite, invite_id)
|
||||
if inv is None:
|
||||
raise HTTPException(status_code=404, detail="No such invite")
|
||||
inv = get_or_404(db, Invite, invite_id, what="invite")
|
||||
# Idempotency guard: only a still-pending invite can be decided. Without this a repeat/stale
|
||||
# approve or deny re-runs its side effects — re-activating a user, or (now that deny emails)
|
||||
# sending a duplicate decision mail, even flipping an already-approved invite to "denied" and
|
||||
@@ -106,8 +204,13 @@ def approve_invite(
|
||||
) -> dict:
|
||||
inv = _decide(db, invite_id, admin, approved=True)
|
||||
_activate_user_by_email(db, inv.email)
|
||||
background.add_task(email_mod.send_access_approved, inv.email)
|
||||
return _serialize(inv)
|
||||
# Computed here, not in the background task: issuing the token needs this request's session, and
|
||||
# the task must not outlive it. Approval answers a request the person MADE, so unlike the
|
||||
# invitation below it is sent automatically — a reply is owed, not a nudge.
|
||||
link, has_account, hours = _welcome_mail_parts(db, inv.email)
|
||||
lang = mailtext.lang_for_email(db, inv.email)
|
||||
background.add_task(email_mod.send_access_approved, inv.email, lang, link, has_account, hours)
|
||||
return _serialize(inv, has_account=bool(_accounts_for(db, [inv.email])))
|
||||
|
||||
|
||||
@router.post("/invites/{invite_id}/deny")
|
||||
@@ -120,8 +223,10 @@ def deny_invite(
|
||||
# Symmetry with approve: the requester hears back either way, instead of a silent limbo. No
|
||||
# account is activated, so this is an email only (the requester has no in-app inbox yet).
|
||||
inv = _decide(db, invite_id, admin, approved=False)
|
||||
background.add_task(email_mod.send_access_denied, inv.email)
|
||||
return _serialize(inv)
|
||||
background.add_task(
|
||||
email_mod.send_access_denied, inv.email, mailtext.lang_for_email(db, inv.email)
|
||||
)
|
||||
return _serialize(inv, has_account=bool(_accounts_for(db, [inv.email])))
|
||||
|
||||
|
||||
class AddInviteIn(BaseModel):
|
||||
@@ -138,7 +243,16 @@ def add_invite(
|
||||
email = body.email.strip().lower()
|
||||
if not valid_email(email):
|
||||
raise HTTPException(status_code=400, detail="Enter a valid email address.")
|
||||
inv = db.execute(select(Invite).where(Invite.email == email)).scalar_one_or_none()
|
||||
# Reuse a canonically-equal row instead of adding a second one for the same mailbox. Prod carries
|
||||
# exactly that pair for one person from 2026-06-16 (hartmann.peter1986@ / hartmannpeter1986@),
|
||||
# which is how an admin ends up approving the same human twice and still seeing them bounce.
|
||||
canon = canonical_email(email)
|
||||
rows = (
|
||||
db.execute(select(Invite).where(Invite.email_canonical == canon).order_by(Invite.id))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
inv = next((r for r in rows if r.email == email), None) or next(iter(rows), None)
|
||||
if inv is None:
|
||||
inv = Invite(email=email)
|
||||
db.add(inv)
|
||||
@@ -150,7 +264,124 @@ def add_invite(
|
||||
)
|
||||
db.commit()
|
||||
_activate_user_by_email(db, email)
|
||||
return _serialize(inv)
|
||||
return _serialize(inv, has_account=bool(_accounts_for(db, [email])))
|
||||
|
||||
|
||||
class InviteSendIn(BaseModel):
|
||||
# Which language to write this person in. Optional so an existing caller (and a test) can post
|
||||
# an empty body, but the admin UI always sends it: this is the one recipient the app has no
|
||||
# signal about, because the admin typed the address in themselves.
|
||||
lang: str | None = None
|
||||
|
||||
|
||||
@router.post("/invites/{invite_id}/invite")
|
||||
def send_invitation(
|
||||
invite_id: int,
|
||||
body: InviteSendIn | None = None,
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Tell someone their address is whitelisted — only when the admin asks for it.
|
||||
|
||||
Whitelisting stays SILENT on purpose. The operator opens a door as a possibility, usually having
|
||||
already agreed it with the person, and an address can sit approved for weeks with nobody
|
||||
registering; a mail fired automatically would be nudging someone who never asked to be nudged.
|
||||
So the send is its own act, on its own button, and `invited_at` records it so the admin can see
|
||||
they have already written rather than writing twice.
|
||||
|
||||
Only an APPROVED row can be invited: there is nothing to invite someone to while their request is
|
||||
undecided, and inviting a denied address would contradict the decision."""
|
||||
inv = get_or_404(db, Invite, invite_id, what="invite")
|
||||
if inv.status != "approved":
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={"code": "invite_send", "reason": "not_approved", "email": inv.email},
|
||||
)
|
||||
if not email_mod.email_enabled(db):
|
||||
# Same reasoning as the admin-side verification resend: mail is fail-soft everywhere else,
|
||||
# but a button the admin pressed must not answer with a success toast for a mail that was
|
||||
# never sent.
|
||||
raise HTTPException(status_code=409, detail={"code": "invite_send", "reason": "email_off"})
|
||||
link, has_account, hours = _welcome_mail_parts(db, inv.email)
|
||||
# An explicit pick from the dialog is recorded, so a later mail to this address matches the one
|
||||
# the admin already sent. Only the explicit pick — writing the fallback back would turn a column
|
||||
# that means "what this person told us" into "what we last guessed".
|
||||
picked = body.lang if body else None
|
||||
if picked in mailtext.SUPPORTED:
|
||||
lang = inv.lang = picked
|
||||
else:
|
||||
lang = mailtext.lang_for_email(db, inv.email)
|
||||
# Sent INLINE, so `invited_at` records the send rather than the intent. Every other mail here is
|
||||
# fail-soft in the background, which is right when nobody is waiting on it — but this column
|
||||
# exists to answer "did I already write to them?", and the UI dims the button and warns about a
|
||||
# second mail on the strength of it. Stamping it before a send that then failed makes the one
|
||||
# question it answers a lie, and talks the admin out of the retry that would fix it.
|
||||
#
|
||||
# One thing a failed send does leave behind: the confirmation token `_welcome_mail_parts` minted
|
||||
# to render the link, which commits inside `_issue_token`. Accepted rather than chased — the
|
||||
# token is single-use, expires on its own, and was mailed nowhere; deferring it would mean
|
||||
# rewriting a helper three callers share so the handler owns its commit.
|
||||
if not email_mod.send_access_invitation(inv.email, lang, link, has_account, hours):
|
||||
raise HTTPException(
|
||||
status_code=422, detail={"code": "invite_send", "reason": "send_failed"}
|
||||
)
|
||||
# Only after the mail is actually out. Same as the other two paths that decide for an address
|
||||
# (`approve_invite`, `add_invite`): an approved row means this mailbox may come in, so finish it,
|
||||
# or the mail can promise "your account is ready — just sign in" to someone `password_login` will
|
||||
# refuse for `is_active`. Doing it BEFORE the send would mean a request the UI reports as failed
|
||||
# had quietly activated an account behind the admin's back.
|
||||
_activate_user_by_email(db, inv.email)
|
||||
inv.invited_at = datetime.now(timezone.utc)
|
||||
audit.record(
|
||||
db,
|
||||
admin.id,
|
||||
AuditAction.INVITE_SEND,
|
||||
target_type="invite",
|
||||
target_id=inv.id,
|
||||
summary=inv.email,
|
||||
)
|
||||
db.commit()
|
||||
return _serialize(inv, has_account=has_account)
|
||||
|
||||
|
||||
@router.delete("/invites/{invite_id}")
|
||||
def delete_invite(
|
||||
invite_id: int,
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Remove a whitelist/request entry outright — the counterpart to add_invite, for an address that
|
||||
never became an account (an approval added by mistake, or one that is no longer wanted). Deny
|
||||
records a decision and keeps the row for the audit trail; delete forgets the address. Neither
|
||||
stops the person asking again: `upsert_pending_invite` reopens a denied row, and a deleted one
|
||||
simply comes back as a fresh pending request.
|
||||
|
||||
Refused once an account exists for the address, whatever its state: this row is the account's
|
||||
approval gate, and orphaning the account from it would leave a person who can neither sign in nor
|
||||
be approved. Delete the USER instead — purge_user removes the invite along with it.
|
||||
|
||||
Reports `still_allowed_by_env`, because deleting the row is NOT always a revoke: `is_allowed`
|
||||
falls back to the env ALLOWED_EMAILS / ADMIN_EMAILS bootstrap lists once no invite exists, so for
|
||||
an address on those the deletion leaves access intact (and an ADMIN_EMAILS address is still
|
||||
promoted to admin on its next Google sign-in). The admin has to hear that, not guess it."""
|
||||
inv = get_or_404(db, Invite, invite_id, what="invite")
|
||||
if _accounts_for(db, [inv.email]):
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail="That address already has an account — delete the account instead.",
|
||||
)
|
||||
still_allowed_by_env = _env_allowed(inv.email)
|
||||
audit.record(
|
||||
db,
|
||||
admin.id,
|
||||
AuditAction.INVITE_DELETE,
|
||||
target_type="invite",
|
||||
target_id=inv.id,
|
||||
summary=inv.email,
|
||||
)
|
||||
db.delete(inv)
|
||||
db.commit()
|
||||
return {"deleted": invite_id, "still_allowed_by_env": still_allowed_by_env}
|
||||
|
||||
|
||||
# --- Users & roles --------------------------------------------------------------------
|
||||
@@ -167,7 +398,15 @@ def _serialize_user(u: User) -> dict:
|
||||
"email_verified": u.email_verified,
|
||||
"is_demo": u.is_demo,
|
||||
"has_password": u.password_hash is not None,
|
||||
# Distinguishes "this account never had a password" (Google-only) from "SEC-10 cleared it" —
|
||||
# the same distinction _send_reset_if_eligible needs, and the one an admin has to see before
|
||||
# deciding what kind of help someone needs.
|
||||
"password_revoked_at": iso(u.password_revoked_at),
|
||||
"has_google": u.google_sub is not None,
|
||||
# Only when it DIFFERS from the account address (E9.5 S3): "connected" is not the same
|
||||
# statement as "connected as someone else", and the second one is the one that explains a
|
||||
# confusing sign-in.
|
||||
"google_email": u.google_email if u.google_email != u.email else None,
|
||||
"created_at": iso(u.created_at),
|
||||
}
|
||||
|
||||
@@ -195,9 +434,7 @@ def set_user_role(
|
||||
role = body.role
|
||||
if role not in ("user", "admin"):
|
||||
raise HTTPException(status_code=400, detail="role must be 'user' or 'admin'")
|
||||
target = db.get(User, user_id)
|
||||
if target is None:
|
||||
raise HTTPException(status_code=404, detail="No such user")
|
||||
target = get_or_404(db, User, user_id, what="user")
|
||||
if target.is_demo:
|
||||
raise HTTPException(status_code=400, detail="The demo account's role can't be changed.")
|
||||
if target.id == admin.id:
|
||||
@@ -224,7 +461,13 @@ def set_user_role(
|
||||
target.role = role
|
||||
db.commit()
|
||||
if changed:
|
||||
background.add_task(email_mod.send_role_changed, target.email, role, operator_contact())
|
||||
background.add_task(
|
||||
email_mod.send_role_changed,
|
||||
target.email,
|
||||
mailtext.lang_for_user(target),
|
||||
role,
|
||||
operator_contact(),
|
||||
)
|
||||
return _serialize_user(target)
|
||||
|
||||
|
||||
@@ -246,9 +489,7 @@ def set_user_suspended(
|
||||
un-suspend the user is emailed they can return (suspend is announced reactively, on a blocked
|
||||
sign-in)."""
|
||||
suspended = body.suspended
|
||||
target = db.get(User, user_id)
|
||||
if target is None:
|
||||
raise HTTPException(status_code=404, detail="No such user")
|
||||
target = get_or_404(db, User, user_id, what="user")
|
||||
if target.is_demo:
|
||||
raise HTTPException(status_code=400, detail="The demo account can't be suspended.")
|
||||
if target.id == admin.id:
|
||||
@@ -273,7 +514,12 @@ def set_user_suspended(
|
||||
target.is_suspended = suspended
|
||||
db.commit()
|
||||
if was_suspended and not suspended:
|
||||
background.add_task(email_mod.send_account_unsuspended, target.email, operator_contact())
|
||||
background.add_task(
|
||||
email_mod.send_account_unsuspended,
|
||||
target.email,
|
||||
mailtext.lang_for_user(target),
|
||||
operator_contact(),
|
||||
)
|
||||
return _serialize_user(target)
|
||||
|
||||
|
||||
@@ -287,9 +533,7 @@ def admin_delete_user(
|
||||
"""Admin-triggered account deletion — the same full erasure a user can perform on themselves
|
||||
(cascade delete of all personal data + access-request cleanup + Google-grant revoke). Guards:
|
||||
not the demo account, not yourself (use Settings → Account), and not the last admin."""
|
||||
target = db.get(User, user_id)
|
||||
if target is None:
|
||||
raise HTTPException(status_code=404, detail="No such user")
|
||||
target = get_or_404(db, User, user_id, what="user")
|
||||
if target.is_demo:
|
||||
raise HTTPException(status_code=400, detail="The demo account can't be deleted.")
|
||||
if target.id == admin.id:
|
||||
@@ -318,6 +562,197 @@ def admin_delete_user(
|
||||
# --- Demo account: email whitelist + state reset -------------------------------------
|
||||
|
||||
|
||||
# --- Stuck sign-ups: the admin lifeline (E9.5 S5, roadmap UX-25) ---------------------------------
|
||||
|
||||
|
||||
def _pending_access_rows(db: Session) -> list[User]:
|
||||
"""Accounts that have signed up but still cannot sign in, newest first.
|
||||
|
||||
Two different dead ends share this list, because from the admin's chair they look the same — a
|
||||
person who registered and is still outside:
|
||||
· `is_active=False` — approved BEFORE they signed up, so nothing ever landed in the
|
||||
Access-requests queue and its Approve button doesn't apply (`_decide` only takes a pending
|
||||
row). This is roadmap UX-25, and it is why the list exists.
|
||||
· `email_verified=False` — the confirmation mail hasn't been acted on. Not the admin's to
|
||||
decide, but theirs to help with: on 2026-09-03 the first mail to a fresh Gmail landed in
|
||||
Spam, so "send it again" is a real move.
|
||||
A suspended account is deliberately excluded: it cannot sign in either, but that is a decision
|
||||
already made, not a person stuck waiting on one.
|
||||
"""
|
||||
return list(
|
||||
db.execute(
|
||||
select(User)
|
||||
.where(
|
||||
User.is_demo.is_(False),
|
||||
User.is_suspended.is_(False),
|
||||
or_(User.is_active.is_(False), User.email_verified.is_(False)),
|
||||
)
|
||||
.order_by(User.created_at.desc())
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
@router.get("/pending-access")
|
||||
def list_pending_access(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> list[dict]:
|
||||
"""The sign-ups that are still outside, with which gate is shut for each. `can_activate` cannot
|
||||
be derived on the client — it needs the invite/allow-list lookup — so it is answered here.
|
||||
|
||||
One `is_allowed` query per row rather than a batched re-implementation of it: the row count is
|
||||
the admin's own backlog of stuck sign-ups (single digits on any real instance), and a second,
|
||||
subtly different copy of the "may this address sign in" rule is exactly the drift this codebase
|
||||
keeps getting bitten by."""
|
||||
out = []
|
||||
for u in _pending_access_rows(db):
|
||||
# The whole rule in one line: this address MAY sign in, and this account still can't. An
|
||||
# undecided request fails it (nothing is allowed yet) and so does a denied one — both belong
|
||||
# to the queue above, which owns that decision. Without this, every fresh registration
|
||||
# appeared in both lists at once, which UAT caught on 2026-09-05.
|
||||
if not is_allowed(db, u.email):
|
||||
continue
|
||||
out.append(_serialize_user(u) | {"can_activate": can_activate(db, u)})
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/users/{user_id}/access")
|
||||
def user_access(
|
||||
user_id: int,
|
||||
_: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Everything the account panel shows about why this person can or cannot get in (E12 S5).
|
||||
|
||||
Its own endpoint rather than extra columns on /users: `is_allowed` and `can_activate` each run a
|
||||
query, and the panel opens for ONE user at a time — folding them into the list would pay that cost
|
||||
per row on every load to answer a question nobody asked yet."""
|
||||
target = get_or_404(db, User, user_id, what="user")
|
||||
return _serialize_user(target) | {
|
||||
"is_allowed": is_allowed(db, target.email),
|
||||
"can_activate": can_activate(db, target),
|
||||
"email_configured": email_mod.email_enabled(db),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/users/{user_id}/set-password-link")
|
||||
def create_set_password_link(
|
||||
user_id: int,
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Mint a one-time link that lets this account set a password, with no email involved (E12 S5).
|
||||
|
||||
The answer to SEC-15, SEC-16 and "no SMTP and someone forgot their password" at once. The admin
|
||||
hands the link over by whatever channel they trust; opening it sets a password and signs out
|
||||
whoever held the old one.
|
||||
|
||||
Deliberately NOT gated on the account's state. It is useful for a pre-empted address, for a
|
||||
revoked password, for a Google-only account that wants one, and for a plain forgotten password on
|
||||
an instance with no mail — narrowing it to one of those would only make the admin's real case the
|
||||
one it refuses. The demo account is the single exception: it is shared, so handing anyone a way to
|
||||
change its password is handing them everyone else's session."""
|
||||
target = get_or_404(db, User, user_id, what="user")
|
||||
if target.is_demo:
|
||||
raise HTTPException(
|
||||
status_code=400, detail={"code": "setpw", "reason": "demo", "email": target.email}
|
||||
)
|
||||
url, expires_at = issue_set_password_link(db, target)
|
||||
# The URL itself is NOT recorded: the audit log is read by humans and kept long, and a live
|
||||
# credential-setting link is the last thing that belongs in it. That the link was issued, by whom,
|
||||
# for whom, is the auditable fact.
|
||||
audit.record(
|
||||
db,
|
||||
admin.id,
|
||||
AuditAction.USER_SETPW_LINK,
|
||||
target_type="user",
|
||||
target_id=target.id,
|
||||
summary=target.email,
|
||||
)
|
||||
db.commit()
|
||||
return {"ok": True, "url": url, "expires_at": iso(expires_at), "email": target.email}
|
||||
|
||||
|
||||
@router.post("/users/{user_id}/activate")
|
||||
def activate_user(
|
||||
user_id: int,
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Let in a registrant whose address was approved before they signed up (roadmap UX-25).
|
||||
|
||||
Admin-side ONLY, and that is the whole design. Two earlier attempts at this were reverted after
|
||||
review, and both moved authority rather than visibility: auto-activating on registration turns a
|
||||
standing whitelist entry into a bearer grant for whoever registers the address first, and
|
||||
reopening the approved invite to `pending` lets an anonymous sign-up REVOKE a standing approval.
|
||||
This endpoint changes nothing about what an unauthenticated caller can cause — the admin could
|
||||
already do exactly this by re-entering the address in the whitelist box (`add_invite` →
|
||||
`_activate_user_by_email`). What was missing was any way to know they had to."""
|
||||
target = get_or_404(db, User, user_id, what="user")
|
||||
if target.is_demo:
|
||||
raise HTTPException(status_code=400, detail="The demo account can't be activated.")
|
||||
if target.is_active:
|
||||
return _serialize_user(target) # idempotent: a double click is not an error
|
||||
if not is_allowed(db, target.email):
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={"code": "activate", "reason": "not_allowed", "email": target.email},
|
||||
)
|
||||
audit.record(
|
||||
db,
|
||||
admin.id,
|
||||
AuditAction.USER_ACTIVATE,
|
||||
target_type="user",
|
||||
target_id=target.id,
|
||||
summary=target.email,
|
||||
before={"is_active": False},
|
||||
after={"is_active": True},
|
||||
)
|
||||
target.is_active = True
|
||||
db.commit()
|
||||
return _serialize_user(target)
|
||||
|
||||
|
||||
@router.post("/users/{user_id}/resend-verification")
|
||||
def resend_user_verification(
|
||||
user_id: int,
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Re-send an account's confirmation link from the admin side.
|
||||
|
||||
The public /auth/verify/resend answers uniformly and reveals nothing, which is right for an
|
||||
anonymous caller but leaves the admin unable to tell a delivered mail from a dropped one. Here
|
||||
the caller is already authenticated as an admin, so this one says plainly what happened.
|
||||
|
||||
Not rate-limited: an admin is not a threat model to enumerate against, and the deliberate silence
|
||||
of the public endpoint is exactly what this exists to escape."""
|
||||
target = get_or_404(db, User, user_id, what="user")
|
||||
if target.is_demo or target.email_verified:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={"code": "resend", "reason": "not_applicable", "email": target.email},
|
||||
)
|
||||
if not email_mod.email_enabled(db):
|
||||
# Mail is fail-soft everywhere else, which is what left the 0.78.5 sign-ups with nothing to
|
||||
# fall back on. An admin pressing a button deserves the failure, not a success toast.
|
||||
raise HTTPException(status_code=409, detail={"code": "resend", "reason": "email_off"})
|
||||
if not send_verification_link(db, target):
|
||||
# Same rule as the invitation above: a button an admin pressed must not answer with a
|
||||
# success for a mail that was dropped. Mail is fail-soft everywhere else because nobody is
|
||||
# waiting on the answer; here somebody is.
|
||||
raise HTTPException(status_code=422, detail={"code": "resend", "reason": "send_failed"})
|
||||
audit.record(
|
||||
db,
|
||||
admin.id,
|
||||
AuditAction.USER_VERIFY_RESEND,
|
||||
target_type="user",
|
||||
target_id=target.id,
|
||||
summary=target.email,
|
||||
)
|
||||
db.commit()
|
||||
return {"ok": True, "email": target.email}
|
||||
|
||||
|
||||
def _serialize_demo(row: DemoWhitelist) -> dict:
|
||||
return {
|
||||
"id": row.id,
|
||||
|
||||
+513
-111
@@ -6,6 +6,8 @@ helpers) stays in app/auth.py, imported below."""
|
||||
import ipaddress
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import NamedTuple
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from authlib.integrations.starlette_client import OAuth, OAuthError
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
|
||||
@@ -16,6 +18,7 @@ from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import email as email_mod
|
||||
from app import mailtext
|
||||
from app import sysconfig
|
||||
from app.auth import (
|
||||
_YOUTUBE_SCOPES,
|
||||
@@ -25,6 +28,8 @@ from app.auth import (
|
||||
READ_SCOPES,
|
||||
WRITE_SCOPES,
|
||||
current_user,
|
||||
env_admin_canonicals,
|
||||
find_user_by_email,
|
||||
get_or_create_demo_user,
|
||||
google_enabled,
|
||||
is_allowed,
|
||||
@@ -37,27 +42,53 @@ from app.db import SessionLocal, get_db
|
||||
from app.models import AuthToken, DemoWhitelist, Invite, OAuthToken, User
|
||||
from app.ratelimit import RateLimiter
|
||||
from app.security import encrypt, hash_password, hash_token, verify_password
|
||||
from app.utils import valid_email
|
||||
from app.utils import canonical_email, valid_email
|
||||
|
||||
VERIFY_TTL = timedelta(hours=24)
|
||||
# A confirmation link that rides an APPROVAL or an invitation lives longer than one the person is
|
||||
# waiting for. Nobody sits refreshing their inbox for an approval — it can easily be read on Monday
|
||||
# after a weekend, and if it has expired by then the one-mail-one-click promise is spent and they are
|
||||
# back to two messages. The risk is small: this token only ever confirms an address. It signs nobody
|
||||
# in and sets no password.
|
||||
APPROVAL_VERIFY_TTL = timedelta(days=7)
|
||||
RESET_TTL = timedelta(hours=1)
|
||||
# An admin-issued set-password link (E12 S5) travels out of band — the admin has to actually reach the
|
||||
# person and hand it over — so the one-hour reset window is far too short. A day matches VERIFY_TTL and
|
||||
# the affordance the admin was shown.
|
||||
ADMIN_SETPW_TTL = timedelta(hours=24)
|
||||
# Both kinds set a password from a link, and the SIGNED-OUT card consumes them through the same
|
||||
# `#reset=` entry, so the user-facing flow is one flow. They stay separate kinds for one reason: a
|
||||
# `reset` token proves control of the MAILBOX (it was mailed there) and therefore confirms the address;
|
||||
# an admin-issued one proves nothing about the mailbox — the admin could hand it to anyone — so it must
|
||||
# not silently satisfy the email gate. See password_reset_confirm.
|
||||
_PASSWORD_TOKEN_KINDS = ("reset", "setpw")
|
||||
|
||||
_DECOY_PASSWORD_HASH = hash_password(secrets.token_urlsafe(16))
|
||||
|
||||
_register_limiter = RateLimiter(max_events=5, window_seconds=300)
|
||||
_login_limiter = RateLimiter(max_events=10, window_seconds=300)
|
||||
_reset_limiter = RateLimiter(max_events=5, window_seconds=300)
|
||||
# Separate from _reset_limiter on purpose: opening a link must not eat the budget for REQUESTING one.
|
||||
_reset_check_limiter = RateLimiter(max_events=10, window_seconds=300)
|
||||
_verify_resend_limiter = RateLimiter(max_events=5, window_seconds=300)
|
||||
_request_access_limiter = RateLimiter(max_events=5, window_seconds=300)
|
||||
_suspend_email_limiter = RateLimiter(max_events=1, window_seconds=3600)
|
||||
|
||||
|
||||
def _notify_suspended(background: BackgroundTasks, email: str) -> None:
|
||||
def _notify_suspended(background: BackgroundTasks, user: User) -> None:
|
||||
"""Schedule the one-per-hour 'account suspended' email to a user whose (otherwise valid)
|
||||
sign-in was just blocked. Gated by verified credentials at the call site, so this can't be
|
||||
triggered by someone who doesn't control the account."""
|
||||
if _suspend_email_limiter.allow(email):
|
||||
background.add_task(email_mod.send_account_suspended, email, operator_contact())
|
||||
triggered by someone who doesn't control the account.
|
||||
|
||||
Takes the user rather than the address because the mail needs their language, and the send runs
|
||||
as a background task — after this request's session, and the row, are out of reach."""
|
||||
if _suspend_email_limiter.allow(user.email):
|
||||
background.add_task(
|
||||
email_mod.send_account_suspended,
|
||||
user.email,
|
||||
mailtext.lang_for_user(user),
|
||||
operator_contact(),
|
||||
)
|
||||
|
||||
|
||||
_demo_limiter = RateLimiter(max_events=5, window_seconds=60)
|
||||
@@ -90,18 +121,43 @@ def google_oauth(db: Session):
|
||||
return _oauth.google
|
||||
|
||||
|
||||
def admin_notify_emails(db: Session) -> list[str]:
|
||||
"""Who to email about admin events (e.g. a new access request). The ACTUAL admins — every active,
|
||||
non-suspended `role=admin` user — UNIONED with the env ADMIN_EMAILS bootstrap list. Using the DB
|
||||
roles means an admin promoted through the UI is notified too (env alone would miss them, yet they
|
||||
CAN approve requests — the approve UI is role-gated, so the notify set must match); the env union
|
||||
keeps a freshly-seeded instance working before any DB admin exists."""
|
||||
db_admins = db.execute(
|
||||
select(User.email).where(
|
||||
def admin_notify_groups(db: Session) -> dict[str, list[str]]:
|
||||
"""Who to email about admin events (e.g. a new access request), grouped by the language each of
|
||||
them reads. The ACTUAL admins — every active, non-suspended `role=admin` user — UNIONED with the
|
||||
env ADMIN_EMAILS bootstrap list. Using the DB roles means an admin promoted through the UI is
|
||||
notified too (env alone would miss them, yet they CAN approve requests — the approve UI is
|
||||
role-gated, so the notify set must match); the env union keeps a freshly-seeded instance working
|
||||
before any DB admin exists.
|
||||
|
||||
Grouped rather than one blast to everyone, because these recipients are people with their own UI
|
||||
language: a single message would have to pick one of them and be wrong for the rest. An env
|
||||
address with no admin account still gets looked up, so an operator listed in ADMIN_EMAILS whose
|
||||
account is (say) suspended is still written to in the language they chose."""
|
||||
rows = db.execute(
|
||||
select(User.email, User.preferences).where(
|
||||
User.role == "admin", User.is_active.is_(True), User.is_suspended.is_(False)
|
||||
)
|
||||
).scalars()
|
||||
return sorted({e.lower() for e in db_admins if e} | settings.admin_email_set)
|
||||
).all()
|
||||
by_email = {e.lower(): mailtext.normalize((p or {}).get("language")) for e, p in rows if e}
|
||||
for email in settings.admin_email_set:
|
||||
if email not in by_email:
|
||||
by_email[email] = mailtext.lang_for_email(db, email)
|
||||
groups: dict[str, list[str]] = {}
|
||||
for email in sorted(by_email):
|
||||
groups.setdefault(by_email[email], []).append(email)
|
||||
return groups
|
||||
|
||||
|
||||
def notify_admins_of_request(
|
||||
db: Session, background: BackgroundTasks | None, requester: str
|
||||
) -> None:
|
||||
"""Tell every admin that someone is waiting — one mail per language group. Sends inline when
|
||||
there is no BackgroundTasks to hand (the registration path already runs outside a response)."""
|
||||
for lang, recipients in admin_notify_groups(db).items():
|
||||
if background is None:
|
||||
email_mod.send_admin_new_request(recipients, lang, requester)
|
||||
else:
|
||||
background.add_task(email_mod.send_admin_new_request, recipients, lang, requester)
|
||||
|
||||
|
||||
def is_demo_allowed(db: Session, email: str) -> bool:
|
||||
@@ -147,14 +203,64 @@ def _client_ip(request: Request) -> str:
|
||||
return peer
|
||||
|
||||
|
||||
def upsert_pending_invite(db: Session, email: str) -> Invite | None:
|
||||
"""Idempotently record an access request. Returns the Invite if a *new* pending row
|
||||
was created (so the caller can notify admins), else None for already-known emails."""
|
||||
class InviteRequest(NamedTuple):
|
||||
"""What an access request settled on.
|
||||
|
||||
`invite` is the row the request now lives under — which is NOT always the address that was just
|
||||
typed: one Gmail mailbox spelled two ways settles on whichever row already exists, and that row's
|
||||
address is where mail will go. Callers that tell the user what happened need it (E9.5 S4).
|
||||
`is_new` is the old boolean return in a name: true only when a row was created or a denied one
|
||||
reopened, i.e. exactly when admins should hear about it. A repeat request must not re-spam them.
|
||||
"""
|
||||
|
||||
invite: Invite | None
|
||||
is_new: bool
|
||||
|
||||
|
||||
def upsert_pending_invite(db: Session, email: str, lang: str | None = None) -> InviteRequest:
|
||||
"""Idempotently record an access request, and the language it was made in.
|
||||
|
||||
"Already known" is decided on the CANONICAL form: `npeter83.temu@` and `npeter83temu@` are one
|
||||
Gmail mailbox, and treating them as two is what put a second, silent access request in the admin
|
||||
queue on 2026-09-03 for an already-approved user. An approved sibling wins — re-requesting must
|
||||
not reopen a decision that has already been made for that mailbox. The row is still STORED under
|
||||
the address as typed, because that is where mail has to go.
|
||||
|
||||
`lang` is the language of the surface the request came from — the sign-in page, or the locale
|
||||
Google reported for a refused sign-in. The latest one wins: someone who asks again from a
|
||||
Hungarian page is telling us something newer than last time. Anything unsupported is ignored
|
||||
rather than stored, so a forged body cannot put junk in the column.
|
||||
|
||||
The commit contract is unchanged: this still commits ONLY when it writes something. Callers rely
|
||||
on that — `_register_account` persists its own flushed user precisely because a no-op here
|
||||
commits nothing (E9.5 S6).
|
||||
"""
|
||||
email = (email or "").lower()
|
||||
if not valid_email(email):
|
||||
return None
|
||||
inv = db.execute(select(Invite).where(Invite.email == email)).scalar_one_or_none()
|
||||
return InviteRequest(None, False)
|
||||
lang = lang if lang in mailtext.SUPPORTED else None
|
||||
canon = canonical_email(email)
|
||||
rows = (
|
||||
db.execute(select(Invite).where(Invite.email_canonical == canon).order_by(Invite.id))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
# An approved row wins, so a denied duplicate can't mask an approval. Then a row that is not
|
||||
# DENIED, preferring the exact spelling — a denied sibling must never be picked while a pending
|
||||
# row exists, or the reopen branch below would fire and put one person in the admin's queue
|
||||
# twice. Only when every row is denied does the spelling decide which one reopens.
|
||||
undecided = [r for r in rows if r.status != "denied"]
|
||||
inv = (
|
||||
next((r for r in rows if r.status == "approved"), None)
|
||||
or next((r for r in undecided if r.email == email), None)
|
||||
or next(iter(undecided), None)
|
||||
or next((r for r in rows if r.email == email), None)
|
||||
or next(iter(rows), None)
|
||||
)
|
||||
if inv is not None:
|
||||
relearned = lang is not None and inv.lang != lang
|
||||
if relearned:
|
||||
inv.lang = lang
|
||||
# Let a previously-denied person ask again; don't disturb approved/pending rows.
|
||||
if inv.status == "denied":
|
||||
inv.status = "pending"
|
||||
@@ -162,12 +268,39 @@ def upsert_pending_invite(db: Session, email: str) -> Invite | None:
|
||||
inv.decided_at = None
|
||||
inv.decided_by = None
|
||||
db.commit()
|
||||
return inv
|
||||
return None
|
||||
inv = Invite(email=email, status="pending")
|
||||
return InviteRequest(inv, True)
|
||||
if relearned:
|
||||
db.commit()
|
||||
return InviteRequest(inv, False)
|
||||
inv = Invite(email=email, status="pending", lang=lang)
|
||||
db.add(inv)
|
||||
db.commit()
|
||||
return inv
|
||||
return InviteRequest(inv, True)
|
||||
|
||||
|
||||
def _access_notice_url(kind: str, email: str = "", inv: Invite | None = None) -> str:
|
||||
"""The landing URL for a Google sign-in that did not reach an account: `requested` (we recorded
|
||||
the request) or `denied` (nothing to record).
|
||||
|
||||
It rides the FRAGMENT, not the query string, for the same reason the verify/reset tokens do
|
||||
(SB3): a fragment never reaches the server, a proxy access log or a Referer header — and this
|
||||
notice now carries the user's address. Naming it is the point of E9.5 S4: on 2026-09-03 a
|
||||
registration as `npeter83.temu@` met an OAuth claim of `npeter83temu@`, and a screen that said
|
||||
"request recorded" without saying FOR WHAT hid the mismatch for a day.
|
||||
|
||||
`linked` names the address the request actually lives under, and ONLY when that differs from the
|
||||
one Google just returned — the honest version of "we didn't open a second request". Same
|
||||
spelling twice is just a repeat, and saying anything there would be noise.
|
||||
|
||||
urlencode (not an f-string): a `+tag` address must arrive as %2B, or the SPA's URLSearchParams
|
||||
would read it back as a space and mangle exactly the address this epic exists to protect.
|
||||
"""
|
||||
params = {"access": kind}
|
||||
if email:
|
||||
params["email"] = email
|
||||
if inv is not None and inv.email != email:
|
||||
params["linked"] = inv.email
|
||||
return "/#" + urlencode(params)
|
||||
|
||||
|
||||
_DEV_OAUTH_HOSTS = frozenset(
|
||||
@@ -201,6 +334,12 @@ class EmailIn(BaseModel):
|
||||
class EmailPasswordIn(BaseModel):
|
||||
email: str | None = None
|
||||
password: str | None = None
|
||||
# The language the sign-up page was in. Captured at registration because it is the ONLY moment
|
||||
# the choice exists anywhere the server can see: it lives in the browser until an account does,
|
||||
# and a preference nobody recorded cannot be backfilled later (the lesson `google_email` taught
|
||||
# in E9.5 S3 — those rows still read NULL until each user signs in again). Nothing reads it yet;
|
||||
# the bilingual mail templates are E9.5 S7.
|
||||
lang: str | None = None
|
||||
|
||||
|
||||
class TokenIn(BaseModel):
|
||||
@@ -283,26 +422,37 @@ async def callback(
|
||||
# A denied Google login doubles as an access request: record it for the admin and
|
||||
# land the user on a friendly "request received" screen instead of a raw 403.
|
||||
if email:
|
||||
inv = upsert_pending_invite(db, email)
|
||||
admins = admin_notify_emails(db)
|
||||
if inv is not None and admins:
|
||||
background.add_task(email_mod.send_admin_new_request, admins, email)
|
||||
return RedirectResponse(url="/?access=requested")
|
||||
return RedirectResponse(url="/?access=denied")
|
||||
# Google just told us this person's locale — the only language signal that will ever
|
||||
# exist for an address with no account, and the decision mail is written from it.
|
||||
req = upsert_pending_invite(db, email, (userinfo.get("locale") or "").split("-")[0])
|
||||
if req.is_new:
|
||||
notify_admins_of_request(db, background, email)
|
||||
return RedirectResponse(url=_access_notice_url("requested", email, req.invite))
|
||||
return RedirectResponse(url=_access_notice_url("denied"))
|
||||
|
||||
user = db.query(User).filter(User.google_sub == sub).one_or_none()
|
||||
if user is None:
|
||||
# No account carries this Google identity yet. If one already exists for this email —
|
||||
# No account carries this Google identity yet. If one already exists for this MAILBOX —
|
||||
# e.g. an email+password registration — adopt the Google identity onto it instead of
|
||||
# inserting a duplicate (which would collide on the unique email and 500). Google has
|
||||
# verified the address, so this safely unifies password + Google sign-in on one account.
|
||||
existing = db.query(User).filter(User.email == email).one_or_none()
|
||||
# inserting a duplicate. Google has verified the address, so this safely unifies password +
|
||||
# Google sign-in on one account.
|
||||
#
|
||||
# Matched canonically, and that is not cosmetic: on 2026-09-03 a registration as
|
||||
# `npeter83.temu@gmail.com` met an OAuth claim of `npeter83temu@gmail.com`, the literal
|
||||
# lookup found nothing, and prod ended up with TWO active user rows for one Gmail mailbox
|
||||
# (ids 13 and 14). Exact spelling still wins when both exist, so an account that already
|
||||
# matches literally is never passed over for a canonical sibling.
|
||||
canon = canonical_email(email)
|
||||
candidates = db.query(User).filter(User.email_canonical == canon).order_by(User.id).all()
|
||||
existing = next((u for u in candidates if u.email == email), None) or next(
|
||||
iter(candidates), None
|
||||
)
|
||||
if existing is not None:
|
||||
if existing.google_sub is not None and existing.google_sub != sub:
|
||||
# A different Google identity is already bound to this email — refuse rather than
|
||||
# hijack it. (Near-impossible with real Google accounts: email↔sub is stable.)
|
||||
log.warning("Google login email collision (different sub): %s", email)
|
||||
return RedirectResponse(url="/?access=denied")
|
||||
return RedirectResponse(url=_access_notice_url("denied"))
|
||||
if not userinfo.get("email_verified", False):
|
||||
# Adopting (and activating) a pre-existing password account requires Google to
|
||||
# actually attest the address — otherwise an unverified Google email that merely
|
||||
@@ -311,7 +461,7 @@ async def callback(
|
||||
"Google login: unverified email, refusing to adopt account: %s",
|
||||
email,
|
||||
)
|
||||
return RedirectResponse(url="/?access=denied")
|
||||
return RedirectResponse(url=_access_notice_url("denied"))
|
||||
user = existing
|
||||
user.google_sub = sub
|
||||
# SEC-10: drop any password pre-set on this account while its email was still unverified,
|
||||
@@ -328,22 +478,23 @@ async def callback(
|
||||
# tell the (verified) owner why — a successful Google auth proves they control this account.
|
||||
if user.is_suspended:
|
||||
log.info("Suspended account blocked at Google login: %s", email)
|
||||
_notify_suspended(background, user.email)
|
||||
_notify_suspended(background, user)
|
||||
return RedirectResponse(url="/?login=suspended")
|
||||
# Keep the email in sync with Google, but never overwrite it with one another account already
|
||||
# owns — that would hit the unique constraint and 500, wedging this user's sign-in (rare: a
|
||||
# Google-side email change to a value that collides with a different local account).
|
||||
if user.email != email:
|
||||
clash = db.query(User).filter(User.email == email, User.id != user.id).one_or_none()
|
||||
if clash is None:
|
||||
user.email = email
|
||||
# Record what Google actually answers with, beside the address the user registered with. The two
|
||||
# are the same string for most people; where they differ (a deliberate `+tag`, a different dot
|
||||
# pattern) the UI shows both, so nobody has to guess which one signs them in.
|
||||
if email:
|
||||
user.google_email = email
|
||||
_sync_google_email(db, user, email)
|
||||
user.display_name = userinfo.get("name")
|
||||
user.avatar_url = userinfo.get("picture")
|
||||
# ADMIN_EMAILS (env) is the bootstrap admin list and always wins, so the configured admin can't
|
||||
# be locked out. Everyone else's role is managed in the admin UI and stored in the DB, so a
|
||||
# routine login must NOT reset it (that would clobber a UI promotion/demotion). Only seed a
|
||||
# default for a brand-new account (its role attribute is still unset before flush).
|
||||
if email in settings.admin_email_set:
|
||||
# Canonical, like every other read of the env lists: a dotted or `+tag`ged ADMIN_EMAILS entry
|
||||
# would otherwise never bootstrap its admin, because Google answers with the plain primary.
|
||||
if canonical_email(email) in env_admin_canonicals():
|
||||
user.role = "admin"
|
||||
elif not user.role:
|
||||
user.role = "user"
|
||||
@@ -355,8 +506,7 @@ async def callback(
|
||||
# picked one yet); unsupported locales fall back to English.
|
||||
prefs = dict(user.preferences or {})
|
||||
if not prefs.get("language"):
|
||||
loc = (userinfo.get("locale") or "").split("-")[0].lower()
|
||||
prefs["language"] = loc if loc in {"en", "hu"} else "en"
|
||||
prefs["language"] = mailtext.normalize((userinfo.get("locale") or "").split("-")[0].lower())
|
||||
user.preferences = prefs
|
||||
db.flush()
|
||||
|
||||
@@ -375,6 +525,30 @@ async def callback(
|
||||
return RedirectResponse(url="/")
|
||||
|
||||
|
||||
def _sync_google_email(db: Session, user: User, email: str) -> None:
|
||||
"""Follow a genuine Google-side email change — and ONLY that.
|
||||
|
||||
A different SPELLING of the same mailbox is the user's own choice and must survive. People
|
||||
register as `me+siftlode@gmail.com` precisely so a later leak from this site is identifiable, and
|
||||
silently rewriting that to `me@gmail.com` destroys the thing it was for. It also locked them out
|
||||
of the spelling they chose, because sign-in matches the stored address first. Google returns the
|
||||
account's PRIMARY address, which need not be what they registered with — and can never contain a
|
||||
`+` at all — so without this guard every plus-tagged registration lost its tag on the first Google
|
||||
sign-in. (E9.5 S3 made that reachable by matching adoption across spellings; the rewrite itself is
|
||||
older.)
|
||||
|
||||
Never overwrite with an address another account already owns either: that would hit the unique
|
||||
constraint and 500, wedging this user's sign-in.
|
||||
"""
|
||||
if not email or user.email == email:
|
||||
return
|
||||
if canonical_email(user.email) == canonical_email(email):
|
||||
return # same mailbox, their spelling — leave it alone
|
||||
clash = db.query(User).filter(User.email == email, User.id != user.id).one_or_none()
|
||||
if clash is None:
|
||||
user.email = email
|
||||
|
||||
|
||||
def _store_token(db: Session, user: User, token: dict) -> None:
|
||||
"""Persist the OAuth grant on the user's single token row, creating it if absent. Google
|
||||
only returns a refresh_token on (re)consent, so the previous one is kept otherwise.
|
||||
@@ -435,6 +609,9 @@ def _complete_link(
|
||||
if user.google_sub is not None and user.google_sub != sub:
|
||||
return RedirectResponse(url="/?link=mismatch")
|
||||
user.google_sub = sub
|
||||
linked_email = (userinfo.get("email") or "").strip().lower()
|
||||
if linked_email:
|
||||
user.google_email = linked_email # see the sign-in callback: the other half of the identity
|
||||
if not user.display_name:
|
||||
user.display_name = userinfo.get("name")
|
||||
if not user.avatar_url:
|
||||
@@ -468,15 +645,23 @@ async def logout(request: Request):
|
||||
return JSONResponse({"ok": True, "switched": False})
|
||||
|
||||
|
||||
class AccessRequestIn(EmailIn):
|
||||
# The language of the page this request was made from. Same reasoning as registration's `lang`
|
||||
# (and the same shape): it is the ONLY moment this preference exists anywhere the server can
|
||||
# see, since the requester has no account to carry it, and it decides what language the approval
|
||||
# or denial mail is written in.
|
||||
lang: str | None = None
|
||||
|
||||
|
||||
@router.post("/request-access")
|
||||
async def request_access(
|
||||
body: EmailIn,
|
||||
body: AccessRequestIn,
|
||||
request: Request,
|
||||
background: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Public: ask for access. Idempotent — repeat requests for the same email don't
|
||||
duplicate or re-spam admins (upsert returns None for an already-pending row).
|
||||
duplicate or re-spam admins (upsert answers `is_new=False` for an already-pending row).
|
||||
|
||||
Answers a uniform ``"pending"`` for EVERY outcome — already-allowed, newly-pending,
|
||||
already-pending, and rate-limited alike. The old ``"approved"`` reply for an allow-listed
|
||||
@@ -491,10 +676,9 @@ async def request_access(
|
||||
return {"status": "pending"}
|
||||
if is_allowed(db, email):
|
||||
return {"status": "pending"} # already allowed — don't reveal it
|
||||
inv = upsert_pending_invite(db, email)
|
||||
admins = admin_notify_emails(db)
|
||||
if inv is not None and admins:
|
||||
background.add_task(email_mod.send_admin_new_request, admins, email)
|
||||
req = upsert_pending_invite(db, email, body.lang)
|
||||
if req.is_new:
|
||||
notify_admins_of_request(db, background, email)
|
||||
return {"status": "pending"}
|
||||
|
||||
|
||||
@@ -551,6 +735,29 @@ def _issue_token(db: Session, user: User, kind: str, ttl: timedelta) -> str:
|
||||
return raw
|
||||
|
||||
|
||||
def _peek_token(db: Session, raw: str | None, kind: str) -> User | None:
|
||||
"""Is this token still usable — WITHOUT spending it. Same four conditions as _consume_token, and
|
||||
the same "does its user still exist" answer, so the two can never disagree about a given token.
|
||||
|
||||
Deliberately NOT the read half of a read-check-write: _consume_token stays one atomic UPDATE, and
|
||||
this never feeds it. A peek is only ever an early warning — the confirm endpoint remains the
|
||||
authority, because a token can expire, be spent, or lose its account between the two calls.
|
||||
"""
|
||||
if not raw:
|
||||
return None
|
||||
row = db.execute(
|
||||
select(AuthToken).where(
|
||||
AuthToken.token_hash == hash_token(raw),
|
||||
AuthToken.kind == kind,
|
||||
AuthToken.used_at.is_(None),
|
||||
AuthToken.expires_at >= datetime.now(timezone.utc),
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
# db.get, exactly as _consume_token does: a deleted account cascades its tokens away, so this is
|
||||
# belt-and-braces — but the two must agree, and agreeing by construction beats agreeing by luck.
|
||||
return db.get(User, row.user_id) if row is not None else None
|
||||
|
||||
|
||||
def _consume_token(db: Session, raw: str | None, kind: str) -> User | None:
|
||||
"""Validate + burn a token atomically. Returns its user, or None if missing/expired/used/wrong
|
||||
kind. The burn is a single conditional UPDATE (…WHERE used_at IS NULL…RETURNING) rather than a
|
||||
@@ -576,6 +783,35 @@ def _consume_token(db: Session, raw: str | None, kind: str) -> User | None:
|
||||
return db.get(User, row.user_id)
|
||||
|
||||
|
||||
def _consume_password_token(db: Session, raw: str | None) -> tuple[User, str] | None:
|
||||
"""Spend a password-setting token of either kind; returns (user, kind). The kind is returned, not
|
||||
swallowed, because the caller has to treat them differently — only a mailed reset attests the
|
||||
address."""
|
||||
for kind in _PASSWORD_TOKEN_KINDS:
|
||||
user = _consume_token(db, raw, kind)
|
||||
if user is not None:
|
||||
return user, kind
|
||||
return None
|
||||
|
||||
|
||||
def issue_set_password_link(db: Session, user: User) -> tuple[str, datetime]:
|
||||
"""An admin-issued, single-use link that lets `user` set a password without any email (E12 S5).
|
||||
|
||||
One mechanism answers three things: SEC-15 (on a no-SMTP instance every registration is
|
||||
auto-verified, so a pre-empted address cannot be recovered by mail), SEC-16 (a pre-empted address
|
||||
stays takeable until its owner resets, and password-reset needs working mail), and the everyday
|
||||
"no SMTP configured and someone forgot their password".
|
||||
|
||||
Issuing does NOT revoke the current password: an admin misclick must not lock anyone out, and the
|
||||
account may be perfectly healthy. USING the link replaces the hash and bumps the session epoch,
|
||||
which is what evicts a squatter — see password_reset_confirm."""
|
||||
expires_at = datetime.now(timezone.utc) + ADMIN_SETPW_TTL
|
||||
raw = _issue_token(db, user, "setpw", ADMIN_SETPW_TTL)
|
||||
# The same `#reset=` fragment the mailed link uses: the signed-out card already knows how to
|
||||
# consume it, and a second entry shape would be a second thing to keep correct.
|
||||
return f"{_app_base()}/#reset={raw}", expires_at
|
||||
|
||||
|
||||
@router.post("/register")
|
||||
def register(
|
||||
body: EmailPasswordIn,
|
||||
@@ -606,17 +842,59 @@ def register(
|
||||
# session). The create path hashes the password + writes several rows + schedules emails; doing it
|
||||
# inline would make a NEW email respond measurably slower than an already-registered one (which
|
||||
# skips all that) — a timing oracle for enumeration. Off-path, any valid email responds the same.
|
||||
background.add_task(_register_account, email, password)
|
||||
background.add_task(_register_account, email, password, body.lang)
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
def _register_account(email: str, password: str) -> None:
|
||||
def _register_account(email: str, password: str, lang: str | None = None) -> None:
|
||||
"""Background worker for /register: create the pending account (+ verification email + admin
|
||||
notice) for a genuinely new email; no-op for an already-registered one. Runs after the response
|
||||
with its own DB session, so the request's timing never reveals whether the email already exists."""
|
||||
with SessionLocal() as db:
|
||||
if db.execute(select(User).where(User.email == email)).scalar_one_or_none() is not None:
|
||||
return # already registered — nothing to do (the uniform "ok" was already returned)
|
||||
# Canonical, so a second spelling of one Gmail mailbox cannot open a SECOND account. Prod
|
||||
# already carries such a pair (user ids 13 and 14, both for npeter83temu@gmail.com); this is
|
||||
# the end that stops new ones appearing, while the Google callback copes with the existing.
|
||||
candidates = (
|
||||
db.execute(
|
||||
select(User).where(User.email_canonical == canonical_email(email)).order_by(User.id)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
# Exact spelling wins when a mailbox carries two rows, same as every other canonical lookup
|
||||
# in this epic (`find_user_by_email`, `_activate_user_by_email`, the Google callback). Without
|
||||
# it `.first()` handed back whichever row Postgres felt like, so a re-registration of the
|
||||
# dotted spelling could resolve to the undotted, already-confirmed twin and send nothing.
|
||||
existing = next((u for u in candidates if u.email == email), None) or next(
|
||||
iter(candidates), None
|
||||
)
|
||||
if existing is not None:
|
||||
# Already registered. This used to return in silence, and that silence is the most
|
||||
# likely explanation for the 2026-09-03 report that a confirmation mail "never arrived":
|
||||
# nothing was ever SENT. The person re-registered, got the same cheerful "check your
|
||||
# inbox", and waited for a message that did not exist.
|
||||
#
|
||||
# Re-sending the link instead leaks nothing new. The response and its timing are
|
||||
# unchanged (uniform "ok", decided off the response path), the mail goes to the address
|
||||
# ON THE ACCOUNT, and this is exactly what the public /auth/verify/resend already does
|
||||
# for the same input — so it is not a new capability, just the end of a dead end. An
|
||||
# account that is already confirmed still gets nothing: there is nothing to confirm.
|
||||
#
|
||||
# And a re-send is ALL it does: the submitted password is discarded. E9.5's closing
|
||||
# review tried "while the address is unverified, the newest registration owns it"
|
||||
# (replace the hash, kill the outstanding verify tokens) as a fix for address
|
||||
# pre-emption, and round 2 showed it was worse than the problem — DO NOT REPEAT IT.
|
||||
# The rule is symmetric, and an attacker can always be last: it hands an unauthenticated
|
||||
# caller the power to overwrite a real registrant's chosen password and delete the
|
||||
# confirmation link already in their inbox, and on a SEC-10-stripped row (password_hash
|
||||
# NULL, password_revoked_at set) it would install a password where the whole point was
|
||||
# that there is none. The owner's real recourse already exists and needs no new
|
||||
# primitive: password-reset accepts this account, proves the mailbox, rewrites the hash
|
||||
# and bumps the session epoch. The residual is logged as SEC-16.
|
||||
user = existing
|
||||
if not user.email_verified and not user.is_demo and email_mod.email_enabled():
|
||||
send_verification_link(db, user)
|
||||
return
|
||||
# Without working SMTP we can't deliver a verification link, so email ownership can't gate
|
||||
# sign-in. Admin approval stays the real gate (is_active=False); mark the account verified so
|
||||
# the flow still completes on a no-SMTP self-host. See email.email_enabled().
|
||||
@@ -626,6 +904,8 @@ def _register_account(email: str, password: str) -> None:
|
||||
password_hash=hash_password(password),
|
||||
is_active=False,
|
||||
email_verified=not email_ok,
|
||||
# Only a supported code is stored, so a forged body can't put junk in preferences.
|
||||
preferences={"language": lang} if lang in mailtext.SUPPORTED else {},
|
||||
)
|
||||
db.add(user)
|
||||
try:
|
||||
@@ -636,28 +916,81 @@ def _register_account(email: str, password: str) -> None:
|
||||
# no-op as the pre-check hit — roll back and stop.
|
||||
db.rollback()
|
||||
return
|
||||
upsert_pending_invite(db, email) # admin-approval gate (commits)
|
||||
upsert_pending_invite(db, email) # admin-approval gate
|
||||
# Persist the account ourselves. upsert_pending_invite commits only when it actually writes an
|
||||
# invite row — not when this address already has a pending or approved one — and _issue_token
|
||||
# below only runs with SMTP on. So for an address the admin had already approved (or one whose
|
||||
# request was still pending), on an instance with no working SMTP, NOTHING along this path
|
||||
# committed and the flushed user was silently rolled back when the session closed: the
|
||||
# would-be user got the uniform "ok" and no account, and re-registering did the same again.
|
||||
# Commit unconditionally; it is a no-op when upsert already committed the same transaction.
|
||||
db.commit()
|
||||
if email_ok:
|
||||
raw = _issue_token(db, user, "verify", VERIFY_TTL)
|
||||
# Fragment (#), not a query token: keeps the token out of proxy/access logs + Referer (SB3).
|
||||
email_mod.send_verify_email(email, f"{_app_base()}/#verify={raw}")
|
||||
admins = admin_notify_emails(db)
|
||||
if admins:
|
||||
email_mod.send_admin_new_request(admins, email)
|
||||
send_verification_link(db, user)
|
||||
# Fired unconditionally, unlike the other two callers which gate on a fresh invite row: for
|
||||
# an address the admin already approved, nothing lands in the Access-requests queue, so this
|
||||
# is the only place the account's existence is announced at all. Delivery is a separate
|
||||
# matter — with no working SMTP `_send` drops it, which is why the pre-approved registrant
|
||||
# still needs the admin to notice them (roadmap UX-25).
|
||||
notify_admins_of_request(db, None, email)
|
||||
|
||||
|
||||
def issue_verification_link(db: Session, user: User, ttl: timedelta | None = None) -> str:
|
||||
"""A fresh confirmation link for `user`, returned rather than sent — for a mail that carries the
|
||||
link inside a larger message (the branching approval/invitation mails, E9.5 S6).
|
||||
|
||||
Same token machinery and the same fragment (SB3) as the standalone confirmation mail, so the two
|
||||
kinds of link cannot drift apart in shape or in how they are consumed."""
|
||||
raw = _issue_token(db, user, "verify", ttl or VERIFY_TTL)
|
||||
return f"{_app_base()}/#verify={raw}"
|
||||
|
||||
|
||||
def send_verification_link(db: Session, user: User) -> bool:
|
||||
"""Issue a fresh verification token for `user` and mail them the link.
|
||||
|
||||
Always to the address ON THE ACCOUNT, never a spelling typed into some form: the account's
|
||||
address is the one its owner chose to be reachable at, and a `+tag` there is usually deliberate.
|
||||
The token rides the URL FRAGMENT (#) so it can never reach a proxy/access log or a Referer (SB3).
|
||||
|
||||
One definition, three callers — registration, the self-service resend, and the admin-side resend
|
||||
(E9.5 S5). The link's shape and TTL must not drift between them; a verification mail that differs
|
||||
by who triggered it is a bug waiting to be discovered by exactly one user.
|
||||
|
||||
Returns whether the mail actually went out, so a caller with a human waiting on the answer (the
|
||||
admin-side resend) can say so instead of reporting success for a message that was dropped.
|
||||
"""
|
||||
return email_mod.send_verify_email(
|
||||
user.email,
|
||||
mailtext.lang_for_user(user),
|
||||
issue_verification_link(db, user),
|
||||
hours=int(VERIFY_TTL.total_seconds() // 3600),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/verify")
|
||||
def verify_email_confirm(body: TokenIn, db: Session = Depends(get_db)) -> dict:
|
||||
"""Confirm an email-verification token the SPA read from the URL fragment (SB3). Uniform 400
|
||||
on any invalid/expired/used token."""
|
||||
on any invalid/expired/used token.
|
||||
|
||||
`password_reset_required` tells the SPA to send the user straight to the reset flow: SEC-10 had
|
||||
dropped the password this account registered with, so sign-in would answer with the deliberately
|
||||
uniform "Invalid email or password" — true, but unreadable to the person who just proved they own
|
||||
the mailbox. Saying so leaks nothing: holding a valid single-use verify token for this account
|
||||
already proves that much. Since E9.5 S1 the common source of that state is gone (admin approval no
|
||||
longer revokes), leaving accounts stripped before that change and the Google-adoption path, so this
|
||||
flag is rare now — but it must stay, because those users have no other way back in."""
|
||||
user = _consume_token(db, body.token, "verify")
|
||||
if user is None:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="This verification link is invalid or has expired."
|
||||
)
|
||||
raise HTTPException(status_code=400, detail={"code": "verify", "reason": "invalid"})
|
||||
user.email_verified = True
|
||||
needs_password = user.password_hash is None and user.password_revoked_at is not None
|
||||
email = user.email
|
||||
db.commit()
|
||||
return {"ok": True}
|
||||
# Naming the address lets the SPA say WHICH mailbox it just confirmed — the same reasoning as
|
||||
# password_reset_required above: holding a valid single-use verify token for this account already
|
||||
# proves that much, so echoing it back leaks nothing. It matters in a browser that holds a session
|
||||
# for a different account, where an unqualified "email confirmed" would be actively misleading.
|
||||
return {"ok": True, "password_reset_required": needs_password, "email": email}
|
||||
|
||||
|
||||
@router.post("/verify/resend")
|
||||
@@ -687,22 +1020,36 @@ def _resend_verify_if_eligible(email: str) -> None:
|
||||
if not email_mod.email_enabled():
|
||||
return
|
||||
with SessionLocal() as db:
|
||||
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
|
||||
user = find_user_by_email(db, email)
|
||||
if user is not None and not user.email_verified and not user.is_demo:
|
||||
raw = _issue_token(db, user, "verify", VERIFY_TTL)
|
||||
email_mod.send_verify_email(email, f"{_app_base()}/#verify={raw}")
|
||||
send_verification_link(db, user)
|
||||
|
||||
|
||||
@router.get("/verify")
|
||||
def verify_email(token: str, db: Session = Depends(get_db)):
|
||||
"""Legacy query-token verification link (pre-SB3 emails still in flight). New emails use the
|
||||
fragment + POST flow above; this bounces to a friendly status for older links until they expire."""
|
||||
user = _consume_token(db, token, "verify")
|
||||
if user is None:
|
||||
return RedirectResponse(url="/?verify=invalid")
|
||||
user.email_verified = True
|
||||
db.commit()
|
||||
return RedirectResponse(url="/?verify=ok")
|
||||
def _login_refused(status: int, reason: str, user: User | None = None, **extra) -> HTTPException:
|
||||
"""Every refusal password_login can answer with, in the codebase's structured `{code, reason}`
|
||||
shape so the SPA can say it in the user's own language (the old bare English strings were
|
||||
unreadable under a bilingual UI, and unusable as anything but a one-line error).
|
||||
|
||||
The account-shaped reasons carry the address ON THE ACCOUNT plus both gate flags. That is what
|
||||
turns a flat 403 into an honest screen — it names the mailbox and says which of the two gates is
|
||||
still shut. It is not an enumeration leak: this branch is only reachable once the password has
|
||||
been proven, so the caller already knows the account exists. The two reasons that answer BEFORE
|
||||
that proof (`invalid`, `rate_limited`) carry nothing at all, so they stay byte-identical for a
|
||||
real account and an unknown one.
|
||||
|
||||
Since E9.5 S1 the two gates are order-independent, so `approved` really can be true while
|
||||
`email_verified` is false, and vice versa — the SPA renders whichever is actually outstanding
|
||||
rather than assuming a sequence.
|
||||
"""
|
||||
detail: dict = {"code": "login", "reason": reason}
|
||||
if user is not None:
|
||||
detail |= {
|
||||
"email": user.email,
|
||||
"email_verified": user.email_verified,
|
||||
"approved": user.is_active,
|
||||
}
|
||||
detail |= {k: v for k, v in extra.items() if v is not None}
|
||||
return HTTPException(status_code=status, detail=detail)
|
||||
|
||||
|
||||
@router.post("/password-login")
|
||||
@@ -716,29 +1063,23 @@ def password_login(
|
||||
Once the password is proven correct, the owner gets a specific reason if their account is
|
||||
still pending or suspended — that's not an enumeration leak (they already hold the password)."""
|
||||
if not _login_limiter.allow(_client_ip(request)):
|
||||
raise HTTPException(status_code=429, detail="Too many attempts. Try again shortly.")
|
||||
raise _login_refused(429, "rate_limited")
|
||||
email = (body.email or "").strip().lower()
|
||||
password = body.password or ""
|
||||
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
|
||||
user = find_user_by_email(db, email)
|
||||
# Always run the KDF (against a decoy hash for an unknown / passwordless / demo account) so the
|
||||
# response time is the same whether or not the email exists — no timing-based enumeration.
|
||||
candidate_hash = user.password_hash if (user and not user.is_demo) else None
|
||||
password_ok = verify_password(password, candidate_hash or _DECOY_PASSWORD_HASH)
|
||||
if user is None or user.is_demo or not password_ok:
|
||||
raise HTTPException(status_code=401, detail="Invalid email or password.")
|
||||
raise _login_refused(401, "invalid")
|
||||
if user.is_suspended:
|
||||
_notify_suspended(background, user.email)
|
||||
op = operator_contact()
|
||||
detail = "Your account has been suspended."
|
||||
if op:
|
||||
detail += f" If you have questions, contact the operator at {op}."
|
||||
raise HTTPException(status_code=403, detail=detail)
|
||||
_notify_suspended(background, user)
|
||||
raise _login_refused(403, "suspended", user, operator=operator_contact() or None)
|
||||
if not user.email_verified:
|
||||
raise HTTPException(
|
||||
status_code=403, detail="Please verify your email first (check your inbox)."
|
||||
)
|
||||
raise _login_refused(403, "email_unverified", user)
|
||||
if not user.is_active:
|
||||
raise HTTPException(status_code=403, detail="Your account is awaiting admin approval.")
|
||||
raise _login_refused(403, "pending_approval", user)
|
||||
_establish_session(request, user)
|
||||
log.info("Password login: %s (id=%s, role=%s)", email, user.id, user.role)
|
||||
return {"ok": True}
|
||||
@@ -767,12 +1108,43 @@ def _send_reset_if_eligible(email: str) -> None:
|
||||
"""Background worker for password_reset_request: issue a reset token + email the link, but ONLY
|
||||
for a real (non-demo) password account. Runs after the response with its own DB session, so the
|
||||
request's timing never reveals whether the account exists. Token rides the URL FRAGMENT (#) so it
|
||||
can't leak into proxy/access logs or a Referer (SB3)."""
|
||||
can't leak into proxy/access logs or a Referer (SB3).
|
||||
|
||||
"Password account" is `password_hash OR password_revoked_at` — an account SEC-10 stripped had a
|
||||
password and must be able to get a new one, or it is locked out permanently (no password to sign in
|
||||
with, and formerly no reset link either). A Google-only account that never had a password stays
|
||||
ineligible on purpose: its owner adds one from Settings (/auth/set-password) while signed in, which
|
||||
proves identity through the session rather than by mail."""
|
||||
with SessionLocal() as db:
|
||||
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
|
||||
if user is not None and user.password_hash and not user.is_demo:
|
||||
user = find_user_by_email(db, email)
|
||||
if user is None or user.is_demo:
|
||||
return
|
||||
if user.password_hash or user.password_revoked_at:
|
||||
raw = _issue_token(db, user, "reset", RESET_TTL)
|
||||
email_mod.send_password_reset(email, f"{_app_base()}/#reset={raw}")
|
||||
# The account's own address, as above — never the spelling that was typed into the form.
|
||||
email_mod.send_password_reset(
|
||||
user.email, mailtext.lang_for_user(user), f"{_app_base()}/#reset={raw}"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/password-reset/check")
|
||||
def password_reset_check(body: TokenIn, request: Request, db: Session = Depends(get_db)) -> dict:
|
||||
"""Is the reset link the SPA just opened still good? Answers without spending the token.
|
||||
|
||||
Why this exists: the card used to render "set a new password" purely because a `#reset=` fragment
|
||||
was present, never asking whether it could still succeed. Reported on 2026-09-05 after deleting an
|
||||
account and then opening its reset link — the user types a new password into a form that answers
|
||||
400. Opening a dead reset link is not an edge case: the token lives one hour, mail is slow or lands
|
||||
in spam, and whoever got two mails usually clicks the older one.
|
||||
|
||||
Not an enumeration oracle: probing requires already holding a 43-character random, and the answer
|
||||
is exactly what submitting it would reveal. Throttled per IP anyway — and a throttled caller is
|
||||
told `true`, never `false`: this check may only ever save a user a wasted form, so when it cannot
|
||||
answer it must degrade to the old behaviour instead of declaring a live link dead.
|
||||
"""
|
||||
if not _reset_check_limiter.allow(_client_ip(request)):
|
||||
return {"ok": True}
|
||||
return {"ok": any(_peek_token(db, body.token, k) is not None for k in _PASSWORD_TOKEN_KINDS)}
|
||||
|
||||
|
||||
@router.post("/password-reset/confirm")
|
||||
@@ -783,17 +1155,25 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
|
||||
if len(new_password) < PASSWORD_MIN_LEN:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Password must be at least {PASSWORD_MIN_LEN} characters.",
|
||||
detail={"code": "reset", "reason": "weak_password", "min": PASSWORD_MIN_LEN},
|
||||
)
|
||||
user = _consume_token(db, token, "reset")
|
||||
if user is None:
|
||||
raise HTTPException(status_code=400, detail="This reset link is invalid or has expired.")
|
||||
spent = _consume_password_token(db, token)
|
||||
if spent is None:
|
||||
raise HTTPException(status_code=400, detail={"code": "reset", "reason": "invalid"})
|
||||
user, kind = spent
|
||||
user.password_hash = hash_password(new_password)
|
||||
user.password_revoked_at = None # the account has a password again — SEC-10's trace is spent
|
||||
# A valid reset link proves control of the mailbox, so completing a reset also verifies the
|
||||
# email — this rescues an account that registered but never clicked the verification link (it
|
||||
# can then sign in). The separate admin-approval gate (is_active) is deliberately NOT touched
|
||||
# here: a reset proves email ownership, not that an admin approved the account.
|
||||
user.email_verified = True
|
||||
#
|
||||
# An admin-issued `setpw` link proves NOTHING about the mailbox — the admin hands it over by
|
||||
# whatever means they choose — so it must not satisfy the email gate. Claiming otherwise would let
|
||||
# an admin confirm an address nobody ever proved, which is the same false attestation SEC-15 is
|
||||
# about. The person still confirms their address the normal way.
|
||||
if kind == "reset":
|
||||
user.email_verified = True
|
||||
bump_session_epoch(
|
||||
user
|
||||
) # SA4: a reset (often a compromise response) kills all existing sessions.
|
||||
@@ -801,7 +1181,7 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
|
||||
for row in db.execute(
|
||||
select(AuthToken).where(
|
||||
AuthToken.user_id == user.id,
|
||||
AuthToken.kind == "reset",
|
||||
AuthToken.kind.in_(_PASSWORD_TOKEN_KINDS),
|
||||
AuthToken.used_at.is_(None),
|
||||
)
|
||||
).scalars():
|
||||
@@ -828,12 +1208,33 @@ def bump_session_epoch(user: User) -> None:
|
||||
def revoke_preset_password_if_unverified(user: User) -> None:
|
||||
"""SEC-10 (account pre-emption): a password set on an account whose email was NEVER verified is
|
||||
untrusted — whoever registered the address (to pre-empt the real owner) could have chosen it. When
|
||||
ownership is finally attested (Google adoption, or admin approval), drop that password and bump the
|
||||
epoch to revoke any session it minted; the owner re-sets one via password-reset. No-op for an
|
||||
already-verified account, so a legit no-SMTP registration is unaffected. The single source for this
|
||||
rule — both the Google-callback and admin-activation paths call it so they can't drift."""
|
||||
the real owner attests ownership through Google, drop that password and bump the epoch to revoke any
|
||||
session it minted; the owner re-sets one via password-reset. No-op for an already-verified account,
|
||||
so a legit no-SMTP registration is unaffected.
|
||||
|
||||
The Google-adoption callback is the ONLY caller (E9.5 S1). Admin approval used to call this too, and
|
||||
dropping it was right for the reason S1 gave — the revoke destroyed the password a legitimate
|
||||
registrant had just chosen, and since the admin normally decides before the user reads their mail,
|
||||
that made the "your password was cleared" detour the DEFAULT path rather than an edge case.
|
||||
|
||||
S1 also claimed the revoke was REDUNDANT there, because `password_login` refuses an unverified
|
||||
account before it reads `is_active`. That part was wrong, and the epic's own review caught it: the
|
||||
gate holds only while the address is unverified, and S6's approval mail exists to get the real
|
||||
owner to end that state — at which point a pre-emptor's password works. Nothing replaced the
|
||||
revoke, and that is deliberate. The review tried "the newest registration owns an unverified
|
||||
account" and reverted it: the rule is symmetric, so an attacker can simply register last, and it
|
||||
would have handed anonymous callers a write onto a real account's credential. What the owner
|
||||
actually has is password-reset, which proves the mailbox, rewrites the hash and bumps the epoch.
|
||||
The residual is logged as SEC-16; re-adding a caller here would put "your password was cleared"
|
||||
back in front of every legitimate registrant, which is the whole reason S1 removed it.
|
||||
|
||||
`password_revoked_at` is what makes "the owner re-sets one via password-reset" actually true. The
|
||||
revoke leaves `password_hash IS NULL`, which is also what a Google-only account looks like — and
|
||||
_send_reset_if_eligible keyed off exactly that, so a revoked account silently got no reset link and
|
||||
was locked out for good (it could not sign in either). The stamp separates the two cases."""
|
||||
if not user.email_verified:
|
||||
user.password_hash = None
|
||||
user.password_revoked_at = datetime.now(timezone.utc)
|
||||
bump_session_epoch(user)
|
||||
|
||||
|
||||
@@ -845,7 +1246,7 @@ def logout_others(
|
||||
are rejected, then re-stamp THIS cookie so the caller stays signed in. For "I left myself logged
|
||||
in somewhere" / a suspected session compromise. The demo account (shared) is excluded."""
|
||||
if user.is_demo:
|
||||
raise HTTPException(status_code=403, detail="Not available in the demo account.")
|
||||
raise HTTPException(status_code=403, detail={"code": "demo"})
|
||||
# Commit the bump BEFORE re-stamping this cookie, so a failed commit can't strand the current
|
||||
# session at a newer epoch than the DB (see set_password).
|
||||
bump_session_epoch(user)
|
||||
@@ -892,7 +1293,7 @@ def set_password(
|
||||
account) needs no current password — the session already proves identity. Changing an
|
||||
existing one requires the current password, so a hijacked session can't silently rotate it."""
|
||||
if user.is_demo:
|
||||
raise HTTPException(status_code=403, detail="Not available in the demo account.")
|
||||
raise HTTPException(status_code=403, detail={"code": "demo"})
|
||||
new_password = body.password or ""
|
||||
if len(new_password) < PASSWORD_MIN_LEN:
|
||||
raise HTTPException(
|
||||
@@ -903,6 +1304,7 @@ def set_password(
|
||||
if not verify_password(body.current_password or "", user.password_hash):
|
||||
raise HTTPException(status_code=403, detail="Current password is incorrect.")
|
||||
user.password_hash = hash_password(new_password)
|
||||
user.password_revoked_at = None # same as the reset path: a password exists again
|
||||
# SA4: a password change logs out every OTHER session; re-stamp this cookie at the new epoch so
|
||||
# the person making the change stays signed in here. Commit BEFORE re-stamping so a failed commit
|
||||
# can't leave the cookie at a newer epoch than the DB (which would wrongly 401 THIS session).
|
||||
|
||||
@@ -26,7 +26,7 @@ from app.models import (
|
||||
User,
|
||||
Video,
|
||||
)
|
||||
from app.routes._common import owned_or_404
|
||||
from app.routes._common import get_or_404, not_found, owned_or_404
|
||||
from app.sync.explore import explore_ingest_page
|
||||
from app.sync.runner import backfill_channel_now, run_recent_backfill
|
||||
from app.sync.subscriptions import apply_channel_details
|
||||
@@ -287,9 +287,7 @@ def channel_detail(
|
||||
needed — a stub, or a channel last enriched before those fields existed (published_at is a
|
||||
reliable sentinel: every real channel has a creation date). The enrich uses the API key
|
||||
(1 unit, cached after), so it's skipped for the quota-less demo account."""
|
||||
channel = db.get(Channel, channel_id)
|
||||
if channel is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown channel")
|
||||
channel = get_or_404(db, Channel, channel_id, what="channel")
|
||||
if channel.published_at is None and not user.is_demo:
|
||||
try:
|
||||
with (
|
||||
@@ -366,9 +364,7 @@ def explore_channel(
|
||||
it, then ingests one page of its uploads (newest-first) marked via_explore + enriched.
|
||||
Returns the next page token so the page can "load more". Guarded against the shared
|
||||
backfill quota reserve; blocked for the demo account (it spends shared quota)."""
|
||||
channel = db.get(Channel, channel_id)
|
||||
if channel is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown channel")
|
||||
channel = get_or_404(db, Channel, channel_id, what="channel")
|
||||
if _is_blocked(db, user, channel_id):
|
||||
raise HTTPException(status_code=403, detail="You've blocked this channel.")
|
||||
if quota.backfill_reserve_reached():
|
||||
@@ -426,7 +422,7 @@ def _user_subscription(db: Session, user: User, channel_id: str) -> Subscription
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if sub is None:
|
||||
raise HTTPException(status_code=404, detail="Not subscribed to this channel")
|
||||
raise not_found("subscription")
|
||||
return sub
|
||||
|
||||
|
||||
@@ -512,9 +508,7 @@ def reset_backfill(
|
||||
state (a "reset" trigger). Clears the channel's backfill markers, opts it back into deep
|
||||
backfill, and re-runs its recent pull immediately; the deep scheduler re-pages the full
|
||||
back-catalog on its next run. Idempotent — videos upsert by id, so nothing duplicates."""
|
||||
channel = db.get(Channel, channel_id)
|
||||
if channel is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown channel")
|
||||
channel = get_or_404(db, Channel, channel_id, what="channel")
|
||||
channel.backfill_done = False
|
||||
channel.backfill_cursor = None
|
||||
channel.recent_synced_at = None
|
||||
@@ -544,9 +538,7 @@ def subscribe(
|
||||
local subscription with YouTube's returned resource id and enriches the channel's metadata
|
||||
if it's still a stub, so the new row renders properly in the manager. It does NOT pull the
|
||||
channel's videos — the scheduler picks those up on its next run, like any subscribed channel."""
|
||||
channel = db.get(Channel, channel_id)
|
||||
if channel is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown channel")
|
||||
channel = get_or_404(db, Channel, channel_id, what="channel")
|
||||
existing = db.execute(
|
||||
select(Subscription).where(
|
||||
Subscription.user_id == user.id, Subscription.channel_id == channel_id
|
||||
@@ -663,8 +655,7 @@ def block_channel(
|
||||
"""Block a channel for this user: its videos won't appear in their live search (and won't be
|
||||
ingested for them) and are hidden from their feed / explore. Idempotent. The now-hidden
|
||||
un-kept search/explore videos are reclaimed by the discovery-cleanup job in due course."""
|
||||
if db.get(Channel, channel_id) is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown channel")
|
||||
get_or_404(db, Channel, channel_id, what="channel")
|
||||
if not _is_blocked(db, user, channel_id):
|
||||
db.add(BlockedChannel(user_id=user.id, channel_id=channel_id))
|
||||
# Stop any active exploration of it so it can be cleaned up.
|
||||
@@ -706,7 +697,7 @@ def attach_tag(
|
||||
_user_subscription(db, user, channel_id)
|
||||
# Only the user's own tags may be attached as personal links (owned_or_404 → 404 if the tag
|
||||
# is missing OR belongs to someone else, without leaking which).
|
||||
owned_or_404(db, Tag, body.tag_id, user, detail="Unknown tag")
|
||||
owned_or_404(db, Tag, body.tag_id, user, what="tag")
|
||||
exists = db.execute(
|
||||
select(ChannelTag).where(
|
||||
ChannelTag.channel_id == channel_id,
|
||||
|
||||
@@ -10,6 +10,7 @@ from sqlalchemy.orm import Session
|
||||
|
||||
from app import audit
|
||||
from app import email as email_mod
|
||||
from app import mailtext
|
||||
from app import sysconfig
|
||||
from app.audit import AuditAction
|
||||
from app.auth import admin_user
|
||||
@@ -115,7 +116,7 @@ def send_test_email(
|
||||
lets the admin verify the settings actually work."""
|
||||
if not email_mod.email_enabled():
|
||||
raise HTTPException(status_code=400, detail="SMTP is not configured.")
|
||||
sent = email_mod.send_test(user.email)
|
||||
sent = email_mod.send_test(user.email, mailtext.lang_for_user(user))
|
||||
if not sent:
|
||||
raise HTTPException(status_code=422, detail="SMTP send failed — check the settings.")
|
||||
return {"sent": True, "to": user.email}
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
"""E24 · the Home surface's HTTP shell.
|
||||
|
||||
Thin by construction: every decision — what counts as in progress, what fits a time budget, which
|
||||
tier the hero comes from — lives in `app/discover/`, so the scheduler, a future CLI or a test can ask
|
||||
the same questions without a request. This file validates, gates and serializes.
|
||||
|
||||
The consolidated `GET /api/discover` mirrors the Plex Discover route's one-payload shape (E5 S4): a
|
||||
landing page that fires four requests to answer one question would spend its whole budget on the
|
||||
thing it exists to remove.
|
||||
"""
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import select, update
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.auth import current_user, require_human
|
||||
from app.db import get_db
|
||||
from app.discover import hero as hero_mod
|
||||
from app.discover import pins, rails
|
||||
from app.discover.budget import Budget, DEFAULT_BUDGET
|
||||
from app.models import ComfortPin, User
|
||||
from app.routes._common import owned_or_404
|
||||
|
||||
router = APIRouter(prefix="/api/discover", tags=["discover"])
|
||||
|
||||
|
||||
class PinIn(BaseModel):
|
||||
# From `pins.PIN_KINDS`, not a second hand-written list: the vocabulary, the schema columns and
|
||||
# this request contract have to move together, and re-typing them here is how they would not.
|
||||
kind: Literal[*pins.PIN_KINDS]
|
||||
# The id the rest of the API speaks for that kind: a video id, a channel id, a Plex rating_key,
|
||||
# or a playlist id. `pins.resolve_ref` does the translation to the row's internal key.
|
||||
ref: str
|
||||
|
||||
|
||||
class ReorderIn(BaseModel):
|
||||
ids: list[int]
|
||||
|
||||
|
||||
@router.get("")
|
||||
def get_discover(
|
||||
budget: Budget = DEFAULT_BUDGET,
|
||||
seed: int = 0,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Everything the Home page renders, in one pass.
|
||||
|
||||
`resume` and `shelf` are built once and handed to the hero, so the two rails the hero draws from
|
||||
are not queried twice for a single page load."""
|
||||
resume = rails.continue_watching(db, user)
|
||||
shelf = pins.shelf(db, user)
|
||||
return {
|
||||
"budget": budget,
|
||||
"hero": hero_mod.pick(db, user, budget, seed, resume=resume, shelf=shelf),
|
||||
"continue_watching": resume,
|
||||
"comfort": shelf,
|
||||
}
|
||||
|
||||
|
||||
# How many already-shown cards a reroll may carry, matched to the client's own rolling window
|
||||
# (`SEEN_WINDOW` in `Home.tsx`). A cheap sanity clamp, and deliberately no more than that: `exclude`
|
||||
# never reaches SQL — `hero.pick` uses it only as set membership over an already-bounded candidate
|
||||
# pool (the resume rail's `RAIL_LIMIT` rows, the shelf, `SURPRISE_DRAW` surprise draws) — so there is
|
||||
# no scan for a crafted URL to unbound, and the set is built in full before this slice anyway. The
|
||||
# comment that claimed otherwise was corrected in round 2; the request line's own length is what
|
||||
# actually bounds a hostile `exclude`.
|
||||
MAX_EXCLUDE = 32
|
||||
|
||||
|
||||
@router.get("/pick")
|
||||
def get_pick(
|
||||
budget: Budget = DEFAULT_BUDGET,
|
||||
seed: int = 0,
|
||||
exclude: str = "",
|
||||
start: int = 0,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""The "Mást kérek" reroll: a hero on its own, for a new seed and/or a new budget.
|
||||
|
||||
One seed drives BOTH sources (see `discover/queries.py`) — otherwise a reroll would re-draw the
|
||||
Plex half while the YouTube half stood still, and the button would look broken half the time.
|
||||
|
||||
`exclude` is a comma-separated list of `<source>:<id>` keys the user has already been shown. It
|
||||
is what makes the button honest: a seed alone only moves the surprise tier, so a user with
|
||||
something half-finished got the same card back every press.
|
||||
|
||||
`start` is the press count: it rotates which tier answers first, so consecutive presses offer a
|
||||
different KIND of answer instead of walking down the resume rail. Taken modulo the tier count in
|
||||
`hero.pick`, so any integer is valid and no value can 422 a button."""
|
||||
skip = {p for p in exclude.split(",") if p}
|
||||
return {
|
||||
"budget": budget,
|
||||
"hero": hero_mod.pick(
|
||||
db, user, budget, seed, exclude=set(list(skip)[:MAX_EXCLUDE]), start=start
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/pins")
|
||||
def list_pins(
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""The shelf on its own — for the shelf editor, which needs it without the rest of the page."""
|
||||
return {"items": pins.shelf(db, user)}
|
||||
|
||||
|
||||
@router.post("/pins")
|
||||
def add_pin(
|
||||
body: PinIn,
|
||||
user: User = Depends(require_human),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Pin something to the comfort shelf.
|
||||
|
||||
`require_human`, like every other personal-state write: the shared demo account must not be able
|
||||
to curate a communal shelf. Re-pinning something already on the shelf is a no-op that returns the
|
||||
existing pin rather than a 409 — the gesture is "make sure this is on my shelf", and the unique
|
||||
constraint is the mechanism, not the message."""
|
||||
resolved = pins.resolve_ref(db, user, body.kind, body.ref)
|
||||
if resolved is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown item")
|
||||
column, value = resolved
|
||||
|
||||
existing = db.execute(
|
||||
select(ComfortPin).where(
|
||||
ComfortPin.user_id == user.id,
|
||||
getattr(ComfortPin, column) == value,
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if existing is not None:
|
||||
return {"ok": True, "id": existing.id, "created": False}
|
||||
|
||||
pin = ComfortPin(user_id=user.id, position=pins.next_position(db, user), **{column: value})
|
||||
db.add(pin)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
# A concurrent pin of the same thing from a second tab. The unique constraint did its job;
|
||||
# the user's intent ("be on my shelf") is satisfied either way, so report the winner rather
|
||||
# than a 500 the user can do nothing with.
|
||||
db.rollback()
|
||||
winner = db.execute(
|
||||
select(ComfortPin).where(
|
||||
ComfortPin.user_id == user.id,
|
||||
getattr(ComfortPin, column) == value,
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if winner is None:
|
||||
# Not the unique constraint, then: the only other way in is the target row being pruned
|
||||
# between `resolve_ref` confirming it and this INSERT (a Plex sync dropping a removed
|
||||
# item), which trips the FK. Semantically that is the same "no such thing" the pre-check
|
||||
# returns above — re-raising handed the user a 500 they could do nothing with.
|
||||
raise HTTPException(status_code=404, detail="Unknown item")
|
||||
return {"ok": True, "id": winner.id, "created": False}
|
||||
return {"ok": True, "id": pin.id, "created": True}
|
||||
|
||||
|
||||
@router.delete("/pins/{pin_id}")
|
||||
def remove_pin(
|
||||
pin_id: int,
|
||||
user: User = Depends(require_human),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
pin = owned_or_404(db, ComfortPin, pin_id, user, what="pin")
|
||||
db.delete(pin)
|
||||
db.commit()
|
||||
return {"deleted": pin_id}
|
||||
|
||||
|
||||
@router.post("/pins/reorder")
|
||||
def reorder_pins(
|
||||
body: ReorderIn,
|
||||
user: User = Depends(require_human),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Rewrite the shelf order from a full list of pin ids.
|
||||
|
||||
Whole-list rather than a move-one delta: the shelf is short, and a full rewrite cannot leave two
|
||||
tiles claiming the same slot the way an interrupted sequence of single moves can. Ids that are not
|
||||
this user's are ignored rather than 404'd — a stale tab replaying an order that includes a pin
|
||||
deleted meanwhile should reorder what still exists, not fail wholesale."""
|
||||
owned = {p.id for p in pins.list_pins(db, user)}
|
||||
for position, pin_id in enumerate(i for i in body.ids if i in owned):
|
||||
db.execute(
|
||||
update(ComfortPin)
|
||||
.where(ComfortPin.id == pin_id, ComfortPin.user_id == user.id)
|
||||
.values(position=position)
|
||||
)
|
||||
db.commit()
|
||||
return {"ok": True}
|
||||
@@ -28,6 +28,7 @@ from app.downloads import edit as editmod
|
||||
from app.downloads import links as linksmod
|
||||
from app.downloads import quota, service, storage
|
||||
from app.downloads import upload as uploadmod
|
||||
from app.downloads import urlguard
|
||||
from app.models import (
|
||||
DownloadJob,
|
||||
DownloadLink,
|
||||
@@ -37,6 +38,7 @@ from app.models import (
|
||||
MediaAsset,
|
||||
User,
|
||||
)
|
||||
from app.routes._common import get_or_404, not_found
|
||||
from app.security import hash_password
|
||||
from app.userscope import is_messageable_user, messageable_clauses
|
||||
from app.utils import iso
|
||||
@@ -65,7 +67,18 @@ def resolve_source(raw: str) -> tuple[str, str]:
|
||||
if _VIDEO_ID.match(raw):
|
||||
return "youtube", raw
|
||||
if raw.startswith(("http://", "https://")):
|
||||
u = urlparse(raw)
|
||||
# `urlparse` raises ValueError on a malformed authority ("http://[::1/x"), and this runs
|
||||
# BEFORE the urlguard check that would otherwise turn it into a structured refusal — so
|
||||
# unguarded, that typo escapes as a 500. Answered in urlguard's own shape, because it is the
|
||||
# same typo class: ":99999" is caught there and localized, and one of the two arriving as a
|
||||
# bare English string would be exactly the split the canonical-patterns table forbids.
|
||||
try:
|
||||
u = urlparse(raw)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail={"code": "url_blocked", "reason": urlguard.REASON_MALFORMED},
|
||||
) from exc
|
||||
host = u.netloc.lower()
|
||||
if "youtube.com" in host:
|
||||
if u.path.startswith(("/shorts/", "/embed/", "/live/")):
|
||||
@@ -352,6 +365,17 @@ def enqueue_download(
|
||||
body: EnqueueDownloadIn, user: User = Depends(require_human), db: Session = Depends(get_db)
|
||||
) -> dict:
|
||||
source_kind, source_ref = resolve_source(body.source)
|
||||
# SEC-12: a non-YouTube source is a raw URL the worker will fetch server-side, so refuse the
|
||||
# ones pointing back into our own network BEFORE a job exists. The worker re-checks (DNS can
|
||||
# move between here and the download); see downloads/urlguard.py for what that does and does
|
||||
# not cover.
|
||||
if source_kind == "url":
|
||||
try:
|
||||
urlguard.check_url(source_ref)
|
||||
except urlguard.BlockedUrl as e:
|
||||
raise HTTPException(
|
||||
status_code=422, detail={"code": "url_blocked", "reason": e.reason}
|
||||
) from e
|
||||
spec, profile_id = _resolve_spec(db, user, body.profile_id, body.spec)
|
||||
display_name = (body.display_name or "").strip() or None
|
||||
try:
|
||||
@@ -532,7 +556,12 @@ def _clean_url(raw) -> str | None:
|
||||
return None
|
||||
if not url.startswith(("http://", "https://")):
|
||||
return None
|
||||
parsed = urlparse(url)
|
||||
# The third site in this module handing user input to `urlparse`, which raises on a malformed
|
||||
# authority. This one already has a "not a plausible URL" answer, so the typo belongs in it.
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
except ValueError:
|
||||
return None
|
||||
if not parsed.netloc:
|
||||
return None
|
||||
return url[:2048]
|
||||
@@ -773,7 +802,7 @@ def share_download(
|
||||
# Uniform "no user" for a missing OR non-messageable (demo/suspended/deactivated) target, so
|
||||
# sharing can't be used to probe which addresses exist or to reach a suspended account.
|
||||
if not is_messageable_user(recipient):
|
||||
raise HTTPException(status_code=404, detail="No user with that email.")
|
||||
raise not_found("recipient")
|
||||
if recipient.id == user.id:
|
||||
raise HTTPException(status_code=400, detail="That's already your download.")
|
||||
exists = db.execute(
|
||||
@@ -1063,8 +1092,7 @@ def admin_set_quota(
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
if db.get(User, user_id) is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown user")
|
||||
get_or_404(db, User, user_id, what="user")
|
||||
cur = quota.resolve(db, user_id)
|
||||
row = db.get(DownloadQuota, user_id)
|
||||
if row is None:
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from datetime import date, datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy import and_, func, or_, select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -22,12 +22,14 @@ from app.feedquery import (
|
||||
sort_keys,
|
||||
)
|
||||
from app.models import (
|
||||
POSITION_MAX_S,
|
||||
ChannelTag,
|
||||
Tag,
|
||||
User,
|
||||
Video,
|
||||
VideoState,
|
||||
)
|
||||
from app.routes._common import get_or_404
|
||||
from app.sync.videos import parse_iso8601_duration
|
||||
from app.utils import iso
|
||||
from app.youtube.client import YouTubeClient, YouTubeError
|
||||
@@ -234,8 +236,7 @@ def set_video_state(
|
||||
status = body.status
|
||||
if status not in VALID_STATES:
|
||||
raise HTTPException(status_code=400, detail=f"status must be one of {VALID_STATES}")
|
||||
if db.get(Video, video_id) is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown video")
|
||||
get_or_404(db, Video, video_id, what="video")
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
@@ -303,11 +304,16 @@ def clear_video_state(
|
||||
|
||||
|
||||
class VideoProgressIn(BaseModel):
|
||||
# Finite only: JSON parsing here accepts `Infinity`/`NaN`, and the handler's `int()` turned
|
||||
# one into a 500.
|
||||
model_config = ConfigDict(allow_inf_nan=False)
|
||||
|
||||
# Accept a float and truncate (below) — the player may post a raw currentTime (a float), which
|
||||
# the old `int(payload.get(...))` silently truncated. A plain `int` field would 422 that on the
|
||||
# 10s progress hot path, losing the resume position; `float` keeps the old tolerance.
|
||||
position_seconds: float = 0
|
||||
duration_seconds: float = 0
|
||||
# Bounded above by the int4 column (see POSITION_MAX_S): past it the flush was a 500.
|
||||
position_seconds: float = Field(0, le=POSITION_MAX_S)
|
||||
duration_seconds: float = Field(0, le=POSITION_MAX_S)
|
||||
|
||||
|
||||
@router.post("/videos/{video_id}/progress")
|
||||
@@ -326,8 +332,7 @@ def set_video_progress(
|
||||
if position < 0:
|
||||
position = 0
|
||||
|
||||
if db.get(Video, video_id) is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown video")
|
||||
get_or_404(db, Video, video_id, what="video")
|
||||
|
||||
# Decide whether this position is worth keeping (else clear it).
|
||||
near_end = duration > 0 and position > duration - FINISH_MARGIN_SECONDS
|
||||
|
||||
@@ -5,6 +5,7 @@ from sqlalchemy.orm import Session
|
||||
|
||||
from app.auth import (
|
||||
count_admins,
|
||||
count_pending_activations,
|
||||
current_user,
|
||||
google_enabled,
|
||||
has_read_scope,
|
||||
@@ -13,6 +14,7 @@ from app.auth import (
|
||||
optional_current_user,
|
||||
purge_user,
|
||||
)
|
||||
from app import email as email_mod
|
||||
from app import sysconfig
|
||||
from app.db import get_db
|
||||
from app.models import Invite, User
|
||||
@@ -98,11 +100,21 @@ def get_me(
|
||||
if user is None:
|
||||
return {"authenticated": False}
|
||||
pending_invites = 0
|
||||
pending_activations = 0
|
||||
# Admin-only, because it drives an operator warning: email is fail-soft, so a half-filled SMTP
|
||||
# config costs verification links, password resets and access-request notices with nothing
|
||||
# visible anywhere. Surfacing it here puts the warning on the screens where onboarding happens.
|
||||
email_configured = True
|
||||
if user.role == "admin":
|
||||
pending_invites = (
|
||||
db.scalar(select(func.count()).select_from(Invite).where(Invite.status == "pending"))
|
||||
or 0
|
||||
)
|
||||
# Counted separately from pending_invites rather than folded into it: the two are different
|
||||
# facts (a request awaiting a decision vs a decided address whose sign-up never got let in),
|
||||
# and quietly widening one number would make an existing field mean something new.
|
||||
pending_activations = count_pending_activations(db)
|
||||
email_configured = email_mod.email_enabled(db) # this request's session; no second connect
|
||||
return {
|
||||
"authenticated": True,
|
||||
"id": user.id,
|
||||
@@ -112,6 +124,12 @@ def get_me(
|
||||
"role": user.role,
|
||||
"is_demo": user.is_demo,
|
||||
"has_google": user.google_sub is not None,
|
||||
# The address Google answers with, when it DIFFERS from the account's own. Same string for
|
||||
# most people, and then this is None so no surface has to decide whether to show it — the
|
||||
# comparison lives here, once, instead of in every consumer.
|
||||
"google_email": (
|
||||
user.google_email if user.google_email and user.google_email != user.email else None
|
||||
),
|
||||
"has_password": user.password_hash is not None,
|
||||
# Instance-wide: whether Google sign-in / YouTube connect is available at all (the UI hides
|
||||
# YouTube-access affordances and the onboarding nudge when it isn't configured).
|
||||
@@ -121,6 +139,10 @@ def get_me(
|
||||
"can_read": has_read_scope(user),
|
||||
"can_write": has_write_scope(user),
|
||||
"pending_invites": pending_invites,
|
||||
# Sign-ups an admin still has to let in (roadmap UX-25): approved BEFORE they registered, so
|
||||
# nothing ever reached the Access-requests queue and the badge used to read zero.
|
||||
"pending_activations": pending_activations,
|
||||
"email_configured": email_configured,
|
||||
"preferences": user.preferences or {},
|
||||
}
|
||||
|
||||
@@ -137,8 +159,13 @@ def delete_my_account(
|
||||
which all cascade on the users row. Quota-audit events are kept but anonymised (FK SET NULL).
|
||||
The shared demo account can't be deleted, and the last remaining admin can't delete itself
|
||||
(that would lock everyone out of the admin surfaces)."""
|
||||
# Structured, like the other three demo refusals: `req` leaves 403 to the caller and
|
||||
# `notifyActionError` discards a plain-string detail in favour of the caller's translated
|
||||
# fallback, so a hand-written English sentence here reaches a Hungarian UI as a generic
|
||||
# "couldn't do that" — or, on this affordance, as nothing at all. `reason` distinguishes it
|
||||
# from the other sites' bare `{code: "demo"}` so the UI can say what specifically is refused.
|
||||
if user.is_demo:
|
||||
raise HTTPException(status_code=403, detail="The shared demo account can't be deleted.")
|
||||
raise HTTPException(status_code=403, detail={"code": "demo", "reason": "delete_account"})
|
||||
if user.role == "admin" and count_admins(db) <= 1:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
|
||||
+123
-32
@@ -3,6 +3,8 @@ CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-w
|
||||
mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
|
||||
out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import func, select, and_
|
||||
@@ -20,16 +22,50 @@ from app.sync.playlists import (
|
||||
repull_playlist,
|
||||
sync_user_playlists,
|
||||
)
|
||||
from app.ratelimit import RateLimiter
|
||||
from app.routes._common import get_or_404, owned_or_404
|
||||
from app.youtube.client import YouTubeClient, YouTubeError
|
||||
|
||||
router = APIRouter(prefix="/api/playlists", tags=["playlists"])
|
||||
|
||||
# SEC-14, same reasoning as the sync module's limiter: a playlist pull lists every playlist the
|
||||
# caller owns and then every item in each, against the instance-wide YouTube budget. Its own
|
||||
# limiter rather than a shared one — spending the manual subscription-import allowance must not
|
||||
# also close the playlist button, and vice versa.
|
||||
#
|
||||
# It guards EVERY route that spends under QuotaAction.PLAYLISTS_PULL, not just the bulk one: a
|
||||
# scripted loop over the single-playlist re-pull drains the same shared budget, and limiting one of
|
||||
# two routes with the same quota action is the asymmetric fix this codebase keeps re-learning.
|
||||
# Deliberately NOT on the push side (`push-plan` / `push`): those run on every push attempt,
|
||||
# including the no-op "already up to date" path, so a 3-per-10-minute allowance would refuse
|
||||
# legitimate work. Their exposure is tracked as SEC-18 instead.
|
||||
_playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600)
|
||||
|
||||
|
||||
def _check_pull_cooldown(user: User) -> None:
|
||||
"""Shared gate so the two PLAYLISTS_PULL routes cannot drift apart again."""
|
||||
if not _playlist_pull_limiter.allow(str(user.id)):
|
||||
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
|
||||
|
||||
|
||||
def _own_playlist(db: Session, user: User, playlist_id: int) -> Playlist:
|
||||
pl = db.get(Playlist, playlist_id)
|
||||
if pl is None or pl.user_id != user.id:
|
||||
raise HTTPException(status_code=404, detail="Unknown playlist")
|
||||
return pl
|
||||
return owned_or_404(db, Playlist, playlist_id, user, what="playlist")
|
||||
|
||||
|
||||
def _refused(status: int, reason: str) -> HTTPException:
|
||||
"""A playlist action the server refuses, as a reason the UI translates
|
||||
(`playlists.errors.<reason>`) — a bare English detail reached the Hungarian UI verbatim."""
|
||||
return HTTPException(status_code=status, detail={"code": "playlist", "reason": reason})
|
||||
|
||||
|
||||
def _touch(pl: Playlist) -> None:
|
||||
"""Bump the playlist's `updated_at` for an edit that only writes `playlist_items`.
|
||||
|
||||
`UpdatedAtMixin`'s `onupdate` fires on an UPDATE of the PLAYLIST row, and add / remove /
|
||||
reorder / move write the item rows instead — so without this the rail's "recently changed"
|
||||
sort would only ever see renames. The assignment is what makes the column mean "last time
|
||||
this list changed", which is what the rail promises the reader."""
|
||||
pl.updated_at = datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _next_playlist_position(db: Session, user_id: int) -> int:
|
||||
@@ -71,6 +107,7 @@ def _add_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) ->
|
||||
)
|
||||
if mark_dirty:
|
||||
recompute_dirty(db, pl)
|
||||
_touch(pl)
|
||||
db.commit()
|
||||
return True
|
||||
|
||||
@@ -88,6 +125,7 @@ def _remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool)
|
||||
db.delete(item)
|
||||
if mark_dirty:
|
||||
recompute_dirty(db, pl)
|
||||
_touch(pl)
|
||||
db.commit()
|
||||
return True
|
||||
|
||||
@@ -109,6 +147,9 @@ def _summary(
|
||||
"item_count": count,
|
||||
"total_duration_seconds": total_duration,
|
||||
"cover_thumbnail": cover,
|
||||
# When this list last CHANGED (rename, add/remove/move/reorder) — the rail's
|
||||
# "recently changed first" sort reads it. See _touch for why it isn't automatic.
|
||||
"updated_at": pl.updated_at,
|
||||
}
|
||||
if has_video is not None:
|
||||
out["has_video"] = has_video
|
||||
@@ -219,7 +260,7 @@ def create_playlist(
|
||||
) -> dict:
|
||||
name = body.name.strip()
|
||||
if not name:
|
||||
raise HTTPException(status_code=400, detail="name is required")
|
||||
raise _refused(400, "name_required")
|
||||
pl = Playlist(user_id=user.id, name=name, position=_next_playlist_position(db, user.id))
|
||||
db.add(pl)
|
||||
db.commit()
|
||||
@@ -251,8 +292,7 @@ def add_watch_later(
|
||||
) -> dict:
|
||||
"""Bookmark shortcut: add a video to the built-in Watch later playlist."""
|
||||
video_id = body.video_id
|
||||
if db.get(Video, video_id) is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown video")
|
||||
get_or_404(db, Video, video_id, what="video")
|
||||
pl = _watch_later(db, user)
|
||||
_add_item(db, pl, video_id, mark_dirty=False)
|
||||
return {"saved": True, "playlist_id": pl.id}
|
||||
@@ -280,10 +320,8 @@ def sync_youtube(
|
||||
"""Pull the user's YouTube playlists into local mirrors now (read-only direction).
|
||||
Also runs automatically on the scheduler; this is the manual 'sync now'."""
|
||||
if not has_read_scope(user):
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Connect YouTube (read access) to sync your playlists.",
|
||||
)
|
||||
raise _refused(403, "youtube_read_scope")
|
||||
_check_pull_cooldown(user)
|
||||
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
|
||||
return sync_user_playlists(db, user)
|
||||
|
||||
@@ -299,17 +337,16 @@ def revert_youtube(
|
||||
once the in-session undo history is gone."""
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
if not pl.yt_playlist_id:
|
||||
raise HTTPException(status_code=400, detail="This playlist isn't linked to YouTube.")
|
||||
raise _refused(400, "not_on_youtube")
|
||||
if not has_read_scope(user):
|
||||
raise HTTPException(
|
||||
status_code=403, detail="Connect YouTube (read access) to sync your playlists."
|
||||
)
|
||||
raise _refused(403, "youtube_read_scope")
|
||||
_check_pull_cooldown(user)
|
||||
try:
|
||||
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
|
||||
return repull_playlist(db, user, pl)
|
||||
except YouTubeError:
|
||||
db.rollback()
|
||||
raise HTTPException(status_code=422, detail="Couldn't refresh from YouTube.")
|
||||
raise _refused(422, "youtube_failed")
|
||||
|
||||
|
||||
def _pushable(db: Session, user: User, playlist_id: int) -> Playlist:
|
||||
@@ -317,9 +354,7 @@ def _pushable(db: Session, user: User, playlist_id: int) -> Playlist:
|
||||
Watch later and read-only YouTube mirrors are never pushable."""
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
if pl.kind == "watch_later":
|
||||
raise HTTPException(
|
||||
status_code=400, detail="The Watch later list can't be sent to YouTube."
|
||||
)
|
||||
raise _refused(400, "watch_later_not_pushable")
|
||||
return pl
|
||||
|
||||
|
||||
@@ -337,7 +372,7 @@ def push_plan(
|
||||
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
|
||||
plan = plan_merge(db, user, pl)
|
||||
except YouTubeError:
|
||||
raise HTTPException(status_code=422, detail="Couldn't read the playlist on YouTube.")
|
||||
raise _refused(422, "youtube_failed")
|
||||
plan["remaining_today"] = quota.remaining_today(db)
|
||||
plan["affordable"] = plan["units_estimate"] <= plan["remaining_today"]
|
||||
return plan
|
||||
@@ -367,14 +402,11 @@ def push(
|
||||
)
|
||||
plan = plan_merge(db, user, pl, live=live)
|
||||
if plan["units_estimate"] > quota.remaining_today(db):
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail="Not enough YouTube quota left today for this sync.",
|
||||
)
|
||||
raise _refused(429, "quota_exceeded")
|
||||
return merge_playlist(db, user, pl, live=live)
|
||||
except YouTubeError:
|
||||
db.rollback()
|
||||
raise HTTPException(status_code=422, detail="YouTube playlist sync failed.")
|
||||
raise _refused(422, "youtube_failed")
|
||||
|
||||
|
||||
@router.get("/{playlist_id}")
|
||||
@@ -419,7 +451,7 @@ def rename_playlist(
|
||||
if "name" in body.model_fields_set:
|
||||
name = (body.name or "").strip()
|
||||
if not name:
|
||||
raise HTTPException(status_code=400, detail="name cannot be empty")
|
||||
raise _refused(400, "name_required")
|
||||
if name != pl.name:
|
||||
pl.name = name
|
||||
recompute_dirty(db, pl)
|
||||
@@ -454,18 +486,18 @@ def delete_playlist(
|
||||
) -> dict:
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
if pl.kind == "watch_later":
|
||||
raise HTTPException(status_code=400, detail="The Watch later list can't be deleted.")
|
||||
raise _refused(400, "watch_later_undeletable")
|
||||
if on_youtube:
|
||||
if not pl.yt_playlist_id:
|
||||
raise HTTPException(status_code=400, detail="This playlist isn't on YouTube.")
|
||||
raise _refused(400, "not_on_youtube")
|
||||
if not has_write_scope(user):
|
||||
raise HTTPException(status_code=403, detail="Enable YouTube editing in Settings first.")
|
||||
raise _refused(403, "youtube_editing_off")
|
||||
try:
|
||||
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_DELETE):
|
||||
with YouTubeClient(db, user) as yt:
|
||||
yt.delete_playlist(pl.yt_playlist_id)
|
||||
except YouTubeError:
|
||||
raise HTTPException(status_code=422, detail="YouTube delete failed.")
|
||||
raise _refused(422, "youtube_failed")
|
||||
db.delete(pl) # items cascade
|
||||
db.commit()
|
||||
return {"deleted": playlist_id}
|
||||
@@ -480,8 +512,7 @@ def add_item(
|
||||
) -> dict:
|
||||
pl = _own_playlist(db, user, playlist_id)
|
||||
video_id = body.video_id
|
||||
if db.get(Video, video_id) is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown video")
|
||||
get_or_404(db, Video, video_id, what="video")
|
||||
_add_item(db, pl, video_id, mark_dirty=True)
|
||||
return {"playlist_id": pl.id, "video_id": video_id}
|
||||
|
||||
@@ -498,6 +529,65 @@ def remove_item(
|
||||
return {"playlist_id": pl.id, "video_id": video_id}
|
||||
|
||||
|
||||
class MoveItemIn(BaseModel):
|
||||
target_playlist_id: int
|
||||
|
||||
|
||||
@router.post("/{playlist_id}/items/{video_id}/move")
|
||||
def move_item(
|
||||
playlist_id: int,
|
||||
video_id: str,
|
||||
body: MoveItemIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Move one video from a playlist into another of the caller's playlists, atomically.
|
||||
|
||||
ONE transaction on purpose. The obvious client-side shape — add to the target, then remove
|
||||
from the source — leaves the video in BOTH lists whenever the second call fails, which is
|
||||
exactly the kind of half-applied state this codebase treats as a bug rather than an edge
|
||||
case. Idempotent: if the video is already in the target, the source row is simply dropped
|
||||
(moving into a list that already has it is a removal, not a duplicate)."""
|
||||
src = _own_playlist(db, user, playlist_id)
|
||||
# Its own reason: "this playlist doesn't exist any more" about a TARGET that is gone was read as
|
||||
# the list on screen — which is still there.
|
||||
dst = owned_or_404(db, Playlist, body.target_playlist_id, user, what="move_target")
|
||||
if src.id == dst.id:
|
||||
raise HTTPException(
|
||||
status_code=400, detail={"code": "playlist_move", "reason": "same_playlist"}
|
||||
)
|
||||
item = db.scalar(
|
||||
select(PlaylistItem).where(
|
||||
PlaylistItem.playlist_id == src.id, PlaylistItem.video_id == video_id
|
||||
)
|
||||
)
|
||||
if item is None:
|
||||
# A concurrent edit (another tab, or a YouTube re-pull) already took it out.
|
||||
raise HTTPException(
|
||||
status_code=404, detail={"code": "playlist_move", "reason": "not_in_playlist"}
|
||||
)
|
||||
already = db.scalar(
|
||||
select(PlaylistItem).where(
|
||||
PlaylistItem.playlist_id == dst.id, PlaylistItem.video_id == video_id
|
||||
)
|
||||
)
|
||||
if already is not None:
|
||||
db.delete(item) # the uq_playlist_video constraint would reject a second row anyway
|
||||
else:
|
||||
# Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
|
||||
# survives the move, and there is no window where the video belongs to neither list.
|
||||
item.playlist_id = dst.id
|
||||
item.position = _next_item_position(db, dst.id)
|
||||
# Both sides changed, so both get the dirty recompute and the timestamp — the asymmetric
|
||||
# version of this (touching only the source) is the recurring smell CLAUDE.md warns about.
|
||||
recompute_dirty(db, src)
|
||||
recompute_dirty(db, dst)
|
||||
_touch(src)
|
||||
_touch(dst)
|
||||
db.commit()
|
||||
return {"ok": True, "from_playlist_id": src.id, "to_playlist_id": dst.id, "video_id": video_id}
|
||||
|
||||
|
||||
class ReorderItemsIn(BaseModel):
|
||||
video_ids: list[str] = []
|
||||
|
||||
@@ -533,5 +623,6 @@ def reorder_items(
|
||||
it.position = pos
|
||||
pos += 1
|
||||
recompute_dirty(db, pl)
|
||||
_touch(pl)
|
||||
db.commit()
|
||||
return {"playlist_id": pl.id, "count": pos}
|
||||
|
||||
@@ -3,13 +3,9 @@
|
||||
import logging
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import (
|
||||
PlexItem,
|
||||
PlexLibrary,
|
||||
)
|
||||
from app.models import PlexItem
|
||||
from app.plex.lookups import by_rating_key
|
||||
|
||||
log = logging.getLogger("siftlode.plex")
|
||||
@@ -25,21 +21,3 @@ def item_or_404(db: Session, rating_key: str) -> PlexItem:
|
||||
def _norm_lang(lang: str | None) -> str:
|
||||
"""The overlay has an EN base + an optional HU twin; anything other than `hu` serves EN."""
|
||||
return "hu" if lang == "hu" else "en"
|
||||
|
||||
|
||||
def _content_libs(db: Session) -> tuple[list[int], list[int]]:
|
||||
"""The BROWSABLE content libraries, partitioned into (movie_lib_ids, show_lib_ids) — the single
|
||||
source of the browse scope. Excludes HU overlay libraries (`overlay_of` set): those are never
|
||||
browsed on their own, they only lend their Hungarian metadata to the EN twin's rows (E5 S5)."""
|
||||
libs = (
|
||||
db.execute(
|
||||
select(PlexLibrary).where(
|
||||
PlexLibrary.enabled.is_(True), PlexLibrary.overlay_of.is_(None)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
movie_lib_ids = [lb.id for lb in libs if lb.kind == "movie"]
|
||||
show_lib_ids = [lb.id for lb in libs if lb.kind == "show"]
|
||||
return movie_lib_ids, show_lib_ids
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
Integer,
|
||||
String,
|
||||
and_,
|
||||
@@ -27,15 +28,19 @@ from app.plex import rollup
|
||||
from app.plex.cards import episode_card
|
||||
from app.plex.query import (
|
||||
FIELD_COLUMNS,
|
||||
PLAYBACK_CLASSES,
|
||||
TS_CONFIG,
|
||||
LibraryFilters,
|
||||
apply_meta_filters,
|
||||
apply_movie_state_filter,
|
||||
apply_show_state_filter,
|
||||
content_libs,
|
||||
field_recheck,
|
||||
genres_col,
|
||||
meta_filter_conds,
|
||||
movie_status_case,
|
||||
playable_conds,
|
||||
playback_class_expr,
|
||||
show_status_case,
|
||||
split_csv,
|
||||
summary_col,
|
||||
@@ -43,12 +48,28 @@ from app.plex.query import (
|
||||
vector_col,
|
||||
vector_match,
|
||||
)
|
||||
from app.routes.plex._shared import _content_libs, _norm_lang
|
||||
from app.routes.plex._shared import _norm_lang
|
||||
from app.utils import to_tsquery_str
|
||||
|
||||
router = APIRouter(prefix="/api/plex", tags=["plex"])
|
||||
|
||||
|
||||
def sort_column(u, sort: str):
|
||||
"""The browse grid's sort key -> orderable column on the union subquery. Module-level (not inline
|
||||
in the handler) so a test can pin its vocabulary to `PLEX_SORT_KEYS` — a key the Sort control
|
||||
offers but this cannot order would quietly fall back to `added_at` and look like a broken control.
|
||||
An unknown key (a stale bookmark, a hand-edited URL) deliberately keeps that fallback."""
|
||||
return {
|
||||
"added": u.c.added_at,
|
||||
"title": func.lower(u.c.title),
|
||||
"year": u.c.year,
|
||||
"rating": u.c.rating,
|
||||
"duration": u.c.duration_s,
|
||||
"release": u.c.rel,
|
||||
"lastwatched": u.c.last_watched,
|
||||
}.get(sort, u.c.added_at)
|
||||
|
||||
|
||||
@router.get("/libraries")
|
||||
def list_libraries(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
|
||||
"""The mirrored libraries the user can browse (drives the Plex scope selector). HU overlay
|
||||
@@ -105,7 +126,7 @@ def unified_library(
|
||||
offset = max(0, offset)
|
||||
lang = _norm_lang(lang)
|
||||
p = f.as_dict()
|
||||
movie_lib_ids, show_lib_ids = _content_libs(db)
|
||||
movie_lib_ids, show_lib_ids = content_libs(db)
|
||||
want_movies = scope in ("movie", "both") and bool(movie_lib_ids)
|
||||
want_shows = scope in ("show", "both") and bool(show_lib_ids)
|
||||
|
||||
@@ -135,6 +156,7 @@ def unified_library(
|
||||
PlexItem.originally_available_at.label("rel"),
|
||||
PlexItem.duration_s.label("duration_s"),
|
||||
PlexItem.playable.label("playable"),
|
||||
PlexItem.transcode_hw.label("transcode_hw"),
|
||||
cast(null(), Integer).label("season_count"),
|
||||
func.coalesce(st.position_seconds, 0).label("position_seconds"),
|
||||
m_status.label("status"),
|
||||
@@ -156,6 +178,8 @@ def unified_library(
|
||||
mq = mq.where(PlexItem.duration_s >= f.duration_min)
|
||||
if f.duration_max is not None:
|
||||
mq = mq.where(PlexItem.duration_s <= f.duration_max)
|
||||
# Playback-class filter (movie-only — a show is an aggregate with no single class).
|
||||
mq = mq.where(*playable_conds(split_csv(f.playable)))
|
||||
if tsq is not None:
|
||||
mq = mq.where(vector_match(PlexItem, lang, tsq))
|
||||
if recheck is not None:
|
||||
@@ -179,6 +203,7 @@ def unified_library(
|
||||
PlexShow.originally_available_at.label("rel"),
|
||||
cast(null(), Integer).label("duration_s"),
|
||||
cast(null(), String).label("playable"),
|
||||
cast(null(), Boolean).label("transcode_hw"),
|
||||
PlexShow.child_count.label("season_count"),
|
||||
literal(0).label("position_seconds"),
|
||||
s_status.label("status"),
|
||||
@@ -212,14 +237,7 @@ def unified_library(
|
||||
if tsq is not None:
|
||||
order = [u.c.rank.desc()]
|
||||
else:
|
||||
col = {
|
||||
"title": func.lower(u.c.title),
|
||||
"year": u.c.year,
|
||||
"rating": u.c.rating,
|
||||
"duration": u.c.duration_s,
|
||||
"release": u.c.rel,
|
||||
"lastwatched": u.c.last_watched,
|
||||
}.get(sort, u.c.added_at)
|
||||
col = sort_column(u, sort)
|
||||
order = [(col.asc() if sort_dir == "asc" else col.desc()).nullslast()]
|
||||
order.append(u.c.rk.desc())
|
||||
rows = db.execute(select(u).order_by(*order).offset(offset).limit(limit)).all()
|
||||
@@ -237,6 +255,7 @@ def unified_library(
|
||||
"duration_seconds": r.duration_s,
|
||||
"thumb": f"/api/plex/image/{r.rk}",
|
||||
"playable": r.playable,
|
||||
"transcode_hw": r.transcode_hw,
|
||||
"status": r.status,
|
||||
"position_seconds": r.position_seconds,
|
||||
}
|
||||
@@ -312,13 +331,14 @@ def facets(
|
||||
"genres": [],
|
||||
"content_ratings": [],
|
||||
"studios": [],
|
||||
"playable": [],
|
||||
"year_min": None,
|
||||
"year_max": None,
|
||||
"rating_max": None,
|
||||
"duration_min": None,
|
||||
"duration_max": None,
|
||||
}
|
||||
all_movie_ids, all_show_ids = _content_libs(db)
|
||||
all_movie_ids, all_show_ids = content_libs(db)
|
||||
movie_ids = all_movie_ids if scope in ("movie", "both") else []
|
||||
show_ids = all_show_ids if scope in ("show", "both") else []
|
||||
if not movie_ids and not show_ids:
|
||||
@@ -338,6 +358,9 @@ def facets(
|
||||
query = query.where(model.duration_s >= pp["duration_min"])
|
||||
if with_dur and pp.get("duration_max") is not None:
|
||||
query = query.where(model.duration_s <= pp["duration_max"])
|
||||
# Playback-class filter (movie-only) — applied for the OTHER groups so an active class
|
||||
# narrows them; the playable facet itself passes pp with playable=None (self-exclusion).
|
||||
query = query.where(*playable_conds(split_csv(pp.get("playable"))))
|
||||
# Join watch-state and reuse the grid's exact movie state filter (unified_library) —
|
||||
# hidden movies never appear in the grid, so they must not inflate facet counts/bounds either.
|
||||
st = aliased(PlexState)
|
||||
@@ -356,6 +379,7 @@ def facets(
|
||||
genre_counts: dict[str, int] = {}
|
||||
cr_counts: dict[str, int] = {}
|
||||
studio_counts: dict[str, int] = {}
|
||||
playable_counts: dict[str, int] = {}
|
||||
years: list[int] = []
|
||||
ratings: list[float] = []
|
||||
durs: list[int] = []
|
||||
@@ -420,6 +444,23 @@ def facets(
|
||||
)
|
||||
for s, c in db.execute(stq).all():
|
||||
studio_counts[s] = studio_counts.get(s, 0) + c
|
||||
# Playback class (movie-only) — exclude its own so each class stays an addable OR option.
|
||||
if kind_movie:
|
||||
pcls = playback_class_expr()
|
||||
pcq = (
|
||||
wsq(
|
||||
select(pcls.label("cls"), func.count()),
|
||||
model,
|
||||
ids,
|
||||
{**p, "playable": None},
|
||||
kind_movie,
|
||||
True,
|
||||
)
|
||||
.where(pcls.isnot(None))
|
||||
.group_by(pcls)
|
||||
)
|
||||
for cls, c in db.execute(pcq).all():
|
||||
playable_counts[cls] = playable_counts.get(cls, 0) + c
|
||||
# Year bounds — exclude the year filter.
|
||||
by = db.execute(
|
||||
wsq(
|
||||
@@ -477,6 +518,11 @@ def facets(
|
||||
# Same for the actively-selected studios, so a selected studio stays removable even at zero matches.
|
||||
for s in split_csv(f.studios):
|
||||
studio_counts.setdefault(s, 0)
|
||||
# Keep an actively-selected playback class visible even at zero matches (the sidebar hides the whole
|
||||
# section when its facet list is empty, which would trap a zero-result selection with no way to undo).
|
||||
for c in split_csv(f.playable):
|
||||
if c in PLAYBACK_CLASSES:
|
||||
playable_counts.setdefault(c, 0)
|
||||
return {
|
||||
"genres": [
|
||||
{"value": g, "count": c} for g, c in sorted(genre_counts.items(), key=lambda kv: kv[0])
|
||||
@@ -488,6 +534,12 @@ def facets(
|
||||
{"value": s, "count": c}
|
||||
for s, c in sorted(studio_counts.items(), key=lambda kv: (-kv[1], kv[0]))
|
||||
],
|
||||
# Fixed good→heavy order (not by count), so the class list reads as a severity scale.
|
||||
"playable": [
|
||||
{"value": c, "count": playable_counts[c]}
|
||||
for c in PLAYBACK_CLASSES
|
||||
if c in playable_counts
|
||||
],
|
||||
"year_min": min(years) if years else None,
|
||||
"year_max": max(years) if years else None,
|
||||
"rating_max": max(ratings) if ratings else None,
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
"""Plex collections: browse mirrored collections + the admin manual-collection editor."""
|
||||
|
||||
from collections.abc import Callable
|
||||
from datetime import date, datetime
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import func, select
|
||||
@@ -12,6 +16,7 @@ from app.models import (
|
||||
PlexItem,
|
||||
PlexLibrary,
|
||||
PlexShow,
|
||||
PlexState,
|
||||
User,
|
||||
)
|
||||
from app.plex import rollup
|
||||
@@ -19,8 +24,7 @@ from app.plex import sync as plex_sync
|
||||
from app.plex.cards import movie_card, show_card
|
||||
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
|
||||
from app.plex.lookups import by_rating_key
|
||||
from app.plex.query import like_escape, runtime_by_key, top_art_keys
|
||||
from app.routes.plex._shared import _content_libs
|
||||
from app.plex.query import content_libs, like_escape, runtime_by_key, top_art_keys
|
||||
|
||||
# The most-restrictive → least ordering used to pick a collection's effective content rating from its
|
||||
# members (Plex shows one cert on the collection; we don't store it, so derive it).
|
||||
@@ -84,13 +88,20 @@ def _collection_card(c: PlexCollection) -> dict:
|
||||
def collections(
|
||||
library: str | None = None,
|
||||
q: str | None = None,
|
||||
keys: str | None = None,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Mirrored collections as cards (most-populated first), optionally name-filtered by `q`. With a
|
||||
`library` plex_key → that one library's collections (used by the collection editor). WITHOUT one →
|
||||
the UNION across all enabled libraries (the unified-scope sidebar picker, which has no single
|
||||
library selected)."""
|
||||
library selected).
|
||||
|
||||
`keys` (CSV of rating_keys) narrows to just those. It exists because a Plex link carries the
|
||||
collection FILTER as bare rating_keys — nothing in the URL says what "17045" is called — so the
|
||||
sidebar token and the applied-filter chip had nothing to render but the raw id. Resolving them
|
||||
through this endpoint costs one small row each; pulling the whole list to find two titles is
|
||||
~200 KB."""
|
||||
if library is not None:
|
||||
lib = (
|
||||
db.execute(select(PlexLibrary).where(PlexLibrary.plex_key == str(library)))
|
||||
@@ -114,6 +125,13 @@ def collections(
|
||||
term = (q or "").strip()
|
||||
if term:
|
||||
query = query.where(PlexCollection.title.ilike(f"%{like_escape(term)}%"))
|
||||
if keys is not None:
|
||||
wanted = [k for k in (keys or "").split(",") if k]
|
||||
# An explicit but empty `keys=` asks for nothing — answer nothing, rather than falling through
|
||||
# to "every collection" (which is what an unguarded empty IN () would have to become).
|
||||
if not wanted:
|
||||
return {"collections": []}
|
||||
query = query.where(PlexCollection.rating_key.in_(wanted))
|
||||
rows = (
|
||||
db.execute(
|
||||
query.order_by(
|
||||
@@ -128,7 +146,7 @@ def collections(
|
||||
# members (a collection lives in one library, so exactly one side has art) — a show collection would
|
||||
# otherwise render blank; the collection's own poster is the next fallback, then the colour hash.
|
||||
# Runtime is movie-only (shows have no single duration).
|
||||
movie_ids, show_ids = _content_libs(db)
|
||||
movie_ids, show_ids = content_libs(db)
|
||||
keys = [c.rating_key for c in rows]
|
||||
art_m = top_art_keys(db, PlexItem, movie_ids, PlexItem.collection_keys, keys, per=2)
|
||||
art_s = top_art_keys(db, PlexShow, show_ids, PlexShow.collection_keys, keys, per=2)
|
||||
@@ -150,14 +168,69 @@ def collections(
|
||||
}
|
||||
|
||||
|
||||
class Member(NamedTuple):
|
||||
"""One collection member flattened to everything the Sort control can order by. The page's two
|
||||
legs are different tables (movies and shows), so there is no single query to ORDER BY — the
|
||||
ordering is a pure list sort over these, which is also what makes it unit-testable."""
|
||||
|
||||
kind: str # "movie" | "show"
|
||||
obj: Any # PlexItem | PlexShow, carried through to the card serializer
|
||||
title: str # lower-cased; never NULL (the column is not nullable)
|
||||
year: int | None
|
||||
rating: float | None
|
||||
added: datetime | None
|
||||
release: date | None
|
||||
duration: int | None # movies only — a show has no single runtime
|
||||
last_watched: datetime | None
|
||||
|
||||
|
||||
# Sort key -> what to read off a Member. Pinned to `PLEX_SORT_KEYS` by a test: the collection page
|
||||
# shows the SAME Sort control as the browse grid, so a key this map is missing would be an option
|
||||
# that silently does nothing here.
|
||||
MEMBER_SORT_KEYS: dict[str, Callable[[Member], Any]] = {
|
||||
"added": lambda m: m.added,
|
||||
"lastwatched": lambda m: m.last_watched,
|
||||
"release": lambda m: m.release,
|
||||
"year": lambda m: m.year,
|
||||
"rating": lambda m: m.rating,
|
||||
"duration": lambda m: m.duration,
|
||||
"title": lambda m: m.title,
|
||||
}
|
||||
|
||||
# What the page ordered by before it had a control, and still its default: a film series wants to be
|
||||
# read in release order, not by whenever Plex happened to ingest it.
|
||||
DEFAULT_MEMBER_SORT = "year"
|
||||
|
||||
|
||||
def sort_members(members: list[Member], sort: str, sort_dir: str) -> list[Member]:
|
||||
"""Order a collection's members by one of `PLEX_SORT_KEYS`.
|
||||
|
||||
Two properties worth stating because they are easy to get wrong with a plain `reverse=True`:
|
||||
members with no value for the key sink to the END in BOTH directions (matching the browse grid's
|
||||
`nullslast()` — a movie with no rating is not "the worst rated"), and ties always resolve
|
||||
title-ascending, so flipping the direction reverses the ranking rather than the alphabet inside
|
||||
it. An unknown key falls back to the page's default instead of returning an arbitrary order."""
|
||||
key = MEMBER_SORT_KEYS.get(sort, MEMBER_SORT_KEYS[DEFAULT_MEMBER_SORT])
|
||||
ordered = sorted(members, key=lambda m: m.title) # stable secondary, applied first
|
||||
have = [m for m in ordered if key(m) is not None]
|
||||
missing = [m for m in ordered if key(m) is None]
|
||||
have.sort(key=key, reverse=sort_dir != "asc")
|
||||
return have + missing
|
||||
|
||||
|
||||
@router.get("/collections/{rating_key}")
|
||||
def collection_detail(
|
||||
rating_key: str,
|
||||
sort: str = DEFAULT_MEMBER_SORT,
|
||||
sort_dir: str = "asc",
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""One collection's hero data + its members (movies + shows), ordered by release year then title —
|
||||
the Collection info page. Read-only here; edits go through the admin mutation endpoints below."""
|
||||
"""One collection's hero data + its members (movies + shows) in the requested order — the
|
||||
Collection info page. The sort vocabulary is the browse grid's (`PLEX_SORT_KEYS`) so the same Sort
|
||||
control means the same thing on both surfaces; the default is release year ascending, which is
|
||||
what this page did before it had a control. Read-only here; edits go through the admin mutation
|
||||
endpoints below."""
|
||||
col = by_rating_key(db, PlexCollection, rating_key)
|
||||
if col is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown collection")
|
||||
@@ -180,19 +253,63 @@ def collection_detail(
|
||||
.all()
|
||||
)
|
||||
mstate = rollup.item_state_map(db, user.id, [m.id for m in movies])
|
||||
sstate = rollup.state_map(db, user.id, [s.id for s in shows])
|
||||
# Merge the two type-homogeneous lists into ONE member list ordered by (year, title).
|
||||
tagged = [(m.year, (m.title or "").lower(), "movie", m) for m in movies] + [
|
||||
(s.year, (s.title or "").lower(), "show", s) for s in shows
|
||||
# ONE rollup pass over the show members, for both the aggregate badge status and `resumed_at` —
|
||||
# the browse grid's "last watched" for a show (the newest resume/watched across its episodes). The
|
||||
# cheaper `rollup.state_map` gives only the status, which would leave `lastwatched` unorderable on
|
||||
# a show collection while the control still offered it.
|
||||
show_rows = (
|
||||
db.execute(
|
||||
rollup.agg_subquery(db, user.id, include_resumed=True).where(
|
||||
PlexItem.show_id.in_([s.id for s in shows])
|
||||
)
|
||||
).all()
|
||||
if shows
|
||||
else []
|
||||
)
|
||||
sstate = {r.show_id: rollup.show_status(r.total, r.watched, r.inprog) for r in show_rows}
|
||||
sresumed = {r.show_id: r.resumed_at for r in show_rows}
|
||||
|
||||
def _movie_last_watched(st: PlexState | None) -> datetime | None:
|
||||
# Same definition as the browse grid's: when it was last resumed, else when it was marked
|
||||
# watched (coalesce(progress_updated_at, watched_at)).
|
||||
return (st.progress_updated_at or st.watched_at) if st else None
|
||||
|
||||
# Merge the two type-homogeneous lists into ONE member list, then order it.
|
||||
members = [
|
||||
Member(
|
||||
kind="movie",
|
||||
obj=m,
|
||||
title=(m.title or "").lower(),
|
||||
year=m.year,
|
||||
rating=m.rating,
|
||||
added=m.added_at,
|
||||
release=m.originally_available_at,
|
||||
duration=m.duration_s,
|
||||
last_watched=_movie_last_watched(mstate.get(m.id)),
|
||||
)
|
||||
for m in movies
|
||||
] + [
|
||||
Member(
|
||||
kind="show",
|
||||
obj=sh,
|
||||
title=(sh.title or "").lower(),
|
||||
year=sh.year,
|
||||
rating=sh.rating,
|
||||
added=sh.added_at,
|
||||
release=sh.originally_available_at,
|
||||
duration=None,
|
||||
last_watched=sresumed.get(sh.id),
|
||||
)
|
||||
for sh in shows
|
||||
]
|
||||
tagged.sort(key=lambda t: (t[0] is None, t[0] or 0, t[1]))
|
||||
members = sort_members(members, sort, sort_dir)
|
||||
items = [
|
||||
movie_card(o, mstate.get(o.id))
|
||||
if kind == "movie"
|
||||
else show_card(o, sstate.get(o.id, "new"))
|
||||
for _, _, kind, o in tagged
|
||||
movie_card(m.obj, mstate.get(m.obj.id))
|
||||
if m.kind == "movie"
|
||||
else show_card(m.obj, sstate.get(m.obj.id, "new"))
|
||||
for m in members
|
||||
]
|
||||
years = [y for y, _, _, _ in tagged if y is not None]
|
||||
years = [m.year for m in members if m.year is not None]
|
||||
certs = [m.content_rating for m in movies] + [s.content_rating for s in shows]
|
||||
return {
|
||||
**_collection_card(col),
|
||||
@@ -230,7 +347,9 @@ def _reject_duplicate_name(
|
||||
.first()
|
||||
)
|
||||
if dup is not None and dup.rating_key != exclude_rk:
|
||||
raise HTTPException(status_code=409, detail="A collection with this name already exists")
|
||||
# Structured, so the bilingual UI says it in the reader's language (it was toasted in
|
||||
# Hungarian beside an English error dialog saying the same thing).
|
||||
raise HTTPException(status_code=409, detail={"code": "collection", "reason": "name_taken"})
|
||||
|
||||
|
||||
def _write_or_pending_error(wrote: bool, e: Exception) -> HTTPException:
|
||||
|
||||
@@ -411,6 +411,10 @@ def item_detail(
|
||||
"year": it.year,
|
||||
"duration_seconds": it.duration_s,
|
||||
"playable": it.playable,
|
||||
# Playback-class detail: HW/SW transcode split + the codec/bit-depth the info-page chip shows.
|
||||
"transcode_hw": it.transcode_hw,
|
||||
"codec_video": it.codec_video,
|
||||
"video_bit_depth": it.video_bit_depth,
|
||||
"thumb": f"/api/plex/image/{it.rating_key}",
|
||||
"art": f"/api/plex/image/{it.rating_key}?variant=art",
|
||||
"library": _lib_key(db, it.library_id),
|
||||
|
||||
@@ -27,12 +27,13 @@ from app.plex.query import (
|
||||
LibraryFilters,
|
||||
apply_meta_filters,
|
||||
apply_show_state_filter,
|
||||
content_libs,
|
||||
genres_col,
|
||||
meta_filter_conds,
|
||||
top_art_keys,
|
||||
)
|
||||
from app.plex.tmdb import resolve_recs
|
||||
from app.routes.plex._shared import _content_libs, _norm_lang
|
||||
from app.routes.plex._shared import _norm_lang
|
||||
|
||||
router = APIRouter(prefix="/api/plex", tags=["plex"])
|
||||
|
||||
@@ -43,7 +44,7 @@ SURPRISE_PER_KIND = 3 # Surprise-me: at most this many movies + this many shows
|
||||
|
||||
def _scope_libs(db: Session, scope: str) -> tuple[list[int], list[int]]:
|
||||
"""The movie/show library ids visible for this scope (empty list = out of scope)."""
|
||||
movie_ids, show_ids = _content_libs(db)
|
||||
movie_ids, show_ids = content_libs(db)
|
||||
return (
|
||||
movie_ids if scope in ("movie", "both") else [],
|
||||
show_ids if scope in ("show", "both") else [],
|
||||
@@ -112,10 +113,11 @@ def _recently_added(
|
||||
else []
|
||||
)
|
||||
# Merge the two type-homogeneous lists by added_at, then serialize each side in one batch so the
|
||||
# per-card state lookups stay batched (no N+1). The `(is not None, value)` key sorts a missing
|
||||
# timestamp last under reverse.
|
||||
# per-card state lookups stay batched (no N+1). Same `or NO_TS` key as the two rails below — the
|
||||
# E24 review found this one still on the older `(is not None, value)` form, i.e. one rule in two
|
||||
# spellings, with the two comments explaining it differently.
|
||||
tagged = [(m.added_at, "movie", m) for m in movies] + [(s.added_at, "show", s) for s in shows]
|
||||
tagged.sort(key=lambda t: (t[0] is not None, t[0]), reverse=True)
|
||||
tagged.sort(key=lambda t: t[0] or rollup.NO_TS, reverse=True)
|
||||
top = tagged[:RAIL_LIMIT]
|
||||
# Key the serialized cards back by rating_key (globally unique in Plex — a movie and a show never
|
||||
# share one), so the merge doesn't depend on object identity staying alive.
|
||||
@@ -170,7 +172,7 @@ def _recently_watched(
|
||||
)
|
||||
for ep, st, sh_title, sh_rk in db.execute(eq).all():
|
||||
rows.append((st.watched_at, leaf_card(ep, st, show_title=sh_title, show_rk=sh_rk)))
|
||||
rows.sort(key=lambda r: (r[0] is not None, r[0]), reverse=True)
|
||||
rows.sort(key=lambda r: r[0] or rollup.NO_TS, reverse=True)
|
||||
return [card for _, card in rows[:RAIL_LIMIT]]
|
||||
|
||||
|
||||
@@ -234,7 +236,7 @@ def _continue_watching(
|
||||
for sh, resumed_at in srows:
|
||||
rows.append((resumed_at, show_card(sh, "in_progress")))
|
||||
|
||||
rows.sort(key=lambda r: (r[0] is not None, r[0]), reverse=True)
|
||||
rows.sort(key=lambda r: r[0] or rollup.NO_TS, reverse=True)
|
||||
return [card for _, card in rows[:RAIL_LIMIT]]
|
||||
|
||||
|
||||
|
||||
@@ -19,8 +19,7 @@ from app.models import (
|
||||
from app.plex import rollup
|
||||
from app.plex import tmdb as plex_tmdb
|
||||
from app.plex.cards import movie_card, show_card
|
||||
from app.plex.query import split_csv
|
||||
from app.routes.plex._shared import _content_libs
|
||||
from app.plex.query import content_libs, split_csv
|
||||
|
||||
router = APIRouter(prefix="/api/plex", tags=["plex"])
|
||||
log = logging.getLogger("siftlode.plex")
|
||||
@@ -71,7 +70,7 @@ def people_cards(
|
||||
wanted = split_csv(names)[:16]
|
||||
if not wanted:
|
||||
return {"people": []}
|
||||
movie_lib_ids, show_lib_ids = _content_libs(db)
|
||||
movie_lib_ids, show_lib_ids = content_libs(db)
|
||||
want_movies = scope in ("movie", "both") and bool(movie_lib_ids)
|
||||
want_shows = scope in ("show", "both") and bool(show_lib_ids)
|
||||
photos = {
|
||||
@@ -185,7 +184,7 @@ def person_detail(
|
||||
and _should_schedule_enrich(person.id)
|
||||
):
|
||||
background_tasks.add_task(_enrich_person_bg, person.id)
|
||||
movie_lib_ids, show_lib_ids = _content_libs(db)
|
||||
movie_lib_ids, show_lib_ids = content_libs(db)
|
||||
|
||||
movies: list[dict] = []
|
||||
if movie_lib_ids:
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
"""Plex playback: image/subtitle proxies, HLS streaming, and watch-state writes."""
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
@@ -12,17 +10,20 @@ from fastapi import (
|
||||
BackgroundTasks,
|
||||
Depends,
|
||||
HTTPException,
|
||||
Path,
|
||||
Query,
|
||||
Request,
|
||||
Response,
|
||||
)
|
||||
from fastapi.responses import FileResponse
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.auth import current_user, resolved_user_id
|
||||
from app.db import SessionLocal, get_db
|
||||
from app.models import (
|
||||
POSITION_MAX_S,
|
||||
PlexItem,
|
||||
PlexSeason,
|
||||
PlexShow,
|
||||
@@ -89,62 +90,70 @@ def image(rating_key: str, request: Request, variant: str = "thumb") -> Response
|
||||
return imgcache.response(data, ct)
|
||||
|
||||
|
||||
# Both subtitle routes authenticate with the full `current_user`, not the cookie-only
|
||||
# `_require_session` the image proxies use. That lighter gate exists for bursts of low-sensitivity
|
||||
# poster fetches and skips the is_active / is_suspended / session-epoch checks — acceptable for a
|
||||
# poster, not for a route that starts reads of the media library: a suspended or "log out
|
||||
# everywhere"-revoked cookie kept scheduling them. They are not bursty (one poll a second, one fetch
|
||||
# per track), so the DB user-load costs nothing that matters.
|
||||
#
|
||||
# Both path parameters are bounded, the same way on both routes: they become a cache FILE name
|
||||
# (`sub_<rating_key>_<ord>`), and an absurd one — a 250-digit key or ordinal — outgrew the OS's name
|
||||
# limit and turned the status poll into a 500. A Plex rating_key is a short number; the bound only
|
||||
# has to be generous.
|
||||
_RatingKey = Path(max_length=32)
|
||||
_SubOrd = Path(ge=0, le=9999)
|
||||
|
||||
|
||||
@router.get("/subtitle/{rating_key}/{sub_ord}/status")
|
||||
def subtitle_status(
|
||||
rating_key: str = _RatingKey,
|
||||
sub_ord: int = _SubOrd,
|
||||
user: User = Depends(current_user),
|
||||
# Bounded both ways: a non-finite position used to reach the scheduler's arithmetic and 500, and
|
||||
# an absurd one reached ffmpeg's `-ss`, which refused it.
|
||||
at: float = Query(0.0, ge=0.0, le=subtitles.MAX_POSITION_S, allow_inf_nan=False),
|
||||
) -> dict:
|
||||
"""Is this subtitle track showable yet — and if not, get it moving. Extraction runs in the
|
||||
background (see plex/subtitles.py for the measurements that forced it there), so the player polls
|
||||
this while it waits instead of hanging on a <track> that will not load for a minute or more.
|
||||
|
||||
`at` is the viewer's absolute position: the extractor windows around it, so what gets read first
|
||||
is the part actually about to be watched."""
|
||||
return subtitles.ensure(rating_key, sub_ord, at)
|
||||
|
||||
|
||||
@router.get("/subtitle/{rating_key}/{sub_ord}")
|
||||
def subtitle(rating_key: str, sub_ord: int, request: Request, offset: float = 0.0) -> Response:
|
||||
def subtitle(
|
||||
rating_key: str = _RatingKey,
|
||||
sub_ord: int = _SubOrd,
|
||||
user: User = Depends(current_user),
|
||||
# Finite, like every float this router takes: `-inf` crashed the shift (a 500), `nan` stacked
|
||||
# every cue at 0. It can legitimately be negative (a subtitle nudged later than the session start).
|
||||
offset: float = Query(0.0, allow_inf_nan=False),
|
||||
) -> Response:
|
||||
"""Serve the ord-th subtitle track as WebVTT for the browser's <track> — so choosing a subtitle
|
||||
overlays it directly (no HLS re-mux / session restart). Handles EXTERNAL sidecar subs (fetched
|
||||
from Plex via the stream `key` — the old `-map 0:s` broke on these since they're not in the file)
|
||||
AND embedded text subs (extracted from the local file). Image subs (PGS/VobSub) → 415. Disk-cached."""
|
||||
_require_session(request)
|
||||
# Cache the ABSOLUTE-time VTT once (per rk+ord); the per-session cue-shift is applied on the way
|
||||
# out, so a resume/seek to any offset reuses the same cached track (no cache-per-offset explosion).
|
||||
cache_key = f"sub_{rating_key}_{sub_ord}"
|
||||
hit = subtitles.cache_read(cache_key)
|
||||
if hit is not None:
|
||||
return Response(subtitles.shift_vtt(hit, offset), media_type="text/vtt")
|
||||
try:
|
||||
with SessionLocal() as db:
|
||||
it = item_or_404(db, rating_key)
|
||||
with PlexClient(db) as plex:
|
||||
part = ((plex.metadata(it.rating_key) or {}).get("Media") or [{}])[0].get(
|
||||
"Part"
|
||||
) or [{}]
|
||||
subs = [s for s in (part[0].get("Stream") or []) if s.get("streamType") == 3]
|
||||
if sub_ord < 0 or sub_ord >= len(subs):
|
||||
raise HTTPException(status_code=404, detail="No such subtitle")
|
||||
s = subs[sub_ord]
|
||||
codec = (s.get("codec") or s.get("format") or "").lower()
|
||||
if codec in subtitles.BITMAP_SUBS:
|
||||
raise HTTPException(
|
||||
status_code=415,
|
||||
detail="Image-based subtitle can't be shown as text",
|
||||
)
|
||||
if s.get("key"): # external sidecar → fetch from Plex, convert to VTT
|
||||
raw = plex.raw_get(s["key"])
|
||||
if codec in ("srt", "subrip", ""):
|
||||
vtt = subtitles.srt_to_vtt(raw.decode("utf-8", "replace"))
|
||||
else:
|
||||
fd, tmp = tempfile.mkstemp(suffix="." + (codec or "sub"))
|
||||
try:
|
||||
os.write(fd, raw)
|
||||
os.close(fd)
|
||||
vtt = subtitles.ffmpeg_to_vtt(tmp, None)
|
||||
finally:
|
||||
os.unlink(tmp)
|
||||
elif s.get("index") is not None: # embedded → extract from the local file
|
||||
src = plex_paths.local_media_path(db, it.file_path)
|
||||
if src is None:
|
||||
raise HTTPException(status_code=404, detail="Media file not found on disk")
|
||||
emb_ord = sum(1 for x in subs[:sub_ord] if x.get("index") is not None)
|
||||
vtt = subtitles.ffmpeg_to_vtt(str(src), emb_ord)
|
||||
else:
|
||||
raise HTTPException(status_code=404, detail="Subtitle not fetchable")
|
||||
except PlexNotConfigured as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except PlexError as e:
|
||||
raise HTTPException(status_code=502, detail=f"Subtitle fetch failed: {e}")
|
||||
subtitles.cache_write(cache_key, vtt)
|
||||
return Response(subtitles.shift_vtt(vtt, offset), media_type="text/vtt")
|
||||
AND embedded text subs (extracted from the local file). Image subs (PGS/VobSub) → 415. Disk-cached.
|
||||
|
||||
A pure READ: it serves whatever is cached and schedules nothing — the status poll owns that,
|
||||
because only it knows where the viewer is (see `subtitles.cached_track`). The player mounts the
|
||||
<track> only once /status says there is something to show, so in practice this is only ever hit
|
||||
warm; on a cache eviction race the 202 sends the player back to polling."""
|
||||
# The cached VTT holds ABSOLUTE cue times (per rk+ord); the per-session cue-shift is applied on
|
||||
# the way out, so a resume/seek to any offset reuses the same track (no cache-per-offset explosion).
|
||||
hit, code = subtitles.cached_track(rating_key, sub_ord)
|
||||
if hit is None and code:
|
||||
raise HTTPException(
|
||||
status_code=subtitles.UNAVAILABLE_STATUS.get(code, 422),
|
||||
detail={"code": "subtitle", "reason": code},
|
||||
)
|
||||
if hit is None:
|
||||
return Response(
|
||||
status_code=202, media_type="application/json", content='{"state":"preparing"}'
|
||||
)
|
||||
return Response(subtitles.shift_vtt(hit, offset), media_type="text/vtt")
|
||||
|
||||
|
||||
# Resume/watched thresholds. _PROGRESS_MIN_S matches the feed's PROGRESS_MIN_SECONDS (5), but
|
||||
@@ -287,10 +296,15 @@ def _push_watch(
|
||||
|
||||
|
||||
class ItemProgressIn(BaseModel):
|
||||
# Finite only: JSON parsing here accepts `Infinity`/`NaN`, and the handler's `int()` turned
|
||||
# one into a 500.
|
||||
model_config = ConfigDict(allow_inf_nan=False)
|
||||
|
||||
# float, truncated in the handler — the player may post a raw currentTime (a float); a plain
|
||||
# `int` field would 422 it on the periodic progress checkpoint (the old `int(...)` truncated).
|
||||
position_seconds: float = 0
|
||||
duration_seconds: float = 0
|
||||
# Bounded above by the int4 column (see POSITION_MAX_S): past it the flush was a 500.
|
||||
position_seconds: float = Field(0, le=POSITION_MAX_S)
|
||||
duration_seconds: float = Field(0, le=POSITION_MAX_S)
|
||||
final: bool = False
|
||||
|
||||
|
||||
@@ -309,6 +323,14 @@ def item_progress(
|
||||
`_should_push_resume`). `final` (sent on pause / seek / pagehide / unmount) pushes immediately and
|
||||
resets the throttle window, so Plex sees a settled position the instant the user pauses or seeks."""
|
||||
it = item_or_404(db, rating_key)
|
||||
# A player parked mid-item — playing or PAUSED — checkpoints every 10 s, so this is the signal
|
||||
# that such a viewer still wants their HLS session: the one thing `last_access` could not see,
|
||||
# since it moved only on a file fetch and a paused player fetches nothing (see
|
||||
# stream.reap_idle). It is NOT every open player: `usePlexProgress.flush` sends nothing once the
|
||||
# finish latch is up, or before the first position is known. Those are the states in which
|
||||
# reaping is the right answer anyway, and a viewer who scrubs back into a reaped session gets
|
||||
# the same 404 → reclaim the player already handles.
|
||||
plex_stream.touch_session(str(rating_key))
|
||||
position = max(0, int(body.position_seconds))
|
||||
duration = int(body.duration_seconds)
|
||||
final = body.final
|
||||
@@ -406,10 +428,20 @@ def item_state(
|
||||
)
|
||||
prev_status = st.status if st is not None else "new"
|
||||
if status == "new":
|
||||
# Does Plex hold anything for this item that the un-mark has to clear? Two things qualify: a
|
||||
# view count, and a resume point. An IN-PROGRESS item carries status "new" WITH a position —
|
||||
# that is how the importer maps a bare `viewOffset` (`_plex_watch_to_state`) — so keying the
|
||||
# push on the status alone silently skipped the single case "Clear resume point" exists for:
|
||||
# the un-mark never reached Plex, the offset stayed, and the next pull imported it straight
|
||||
# back (Protector, cleared twice by the user and back at 92s both times). `hidden` is
|
||||
# Siftlode-only and still pushes nothing, exactly as this endpoint's docstring says.
|
||||
mirrored_on_plex = st is not None and (
|
||||
st.status == "watched" or (st.position_seconds or 0) > 0
|
||||
)
|
||||
if st is not None:
|
||||
db.delete(st)
|
||||
db.commit()
|
||||
if prev_status != "new": # something → unwatched: mirror the un-mark to Plex
|
||||
if mirrored_on_plex:
|
||||
_push_watch(background, db, user.id, it.id, rating_key, "unwatched")
|
||||
return {"status": "new"}
|
||||
if st is None:
|
||||
@@ -524,9 +556,10 @@ def season_state(
|
||||
@router.post("/stream/{rating_key}/session")
|
||||
def stream_session(
|
||||
rating_key: str,
|
||||
start: float = 0.0,
|
||||
# Finite: `start=inf` reached `int()` in the session builder and came back as a 500.
|
||||
start: float = Query(0.0, allow_inf_nan=False),
|
||||
audio: int | None = None,
|
||||
aoff: float = 0.0,
|
||||
aoff: float = Query(0.0, allow_inf_nan=False),
|
||||
multi: bool = False,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
@@ -536,7 +569,6 @@ def stream_session(
|
||||
HEVC/VP9/AV1/…) take the same HLS path but with the video re-encoded to H.264 (VAAPI or libx264).
|
||||
Selecting a non-default audio track (`audio` = stream ordinal) forces the HLS path so the chosen audio
|
||||
can be muxed in. Subtitles are independent WebVTT tracks (GET /subtitle) — no session restart."""
|
||||
plex_stream.reap_idle() # opportunistic cleanup of idle sessions on each new playback
|
||||
it = item_or_404(db, rating_key)
|
||||
transcode = it.playable == "transcode"
|
||||
# `multi` (item has ≥2 audio tracks) forces the HLS path so ALL audio tracks ship as renditions and
|
||||
@@ -554,14 +586,18 @@ def stream_session(
|
||||
raise HTTPException(status_code=404, detail="Media file not found on disk")
|
||||
# Report the REAL media start (the keyframe ffmpeg landed on, which hls.js zero-bases to), not the
|
||||
# requested offset — the client's absolute clock and the subtitle cue-shift both key off this.
|
||||
# `transcode`/`hw` let the player show an honest cold-start ("Preparing stream…") and surface the
|
||||
# active mode + encoder in the stats overlay; both segment streams are otherwise identical.
|
||||
# `transcode`/`hw`/`tonemap` let the player show an honest cold-start ("Preparing stream…") and
|
||||
# surface the active mode + encoder in the stats overlay (`tonemap` = the HW HDR→SDR path); both
|
||||
# segment streams are otherwise identical.
|
||||
return {
|
||||
"mode": "hls",
|
||||
"start_s": s.media_start_s,
|
||||
"url": f"/api/plex/stream/{it.rating_key}/hls/{s.entry}",
|
||||
# The session id is part of the path (not a query param) so the playlist's RELATIVE segment
|
||||
# names resolve back into the same session — see `stream_hls_file` for why it has to.
|
||||
"url": f"/api/plex/stream/{it.rating_key}/hls/{s.id}/{s.entry}",
|
||||
"transcode": s.transcode,
|
||||
"hw": s.hw,
|
||||
"tonemap": s.tonemap,
|
||||
}
|
||||
|
||||
|
||||
@@ -590,45 +626,82 @@ def stream_file(
|
||||
return FileResponse(str(src)) # Starlette honours the Range header (206)
|
||||
|
||||
|
||||
# What a session directory can contain, and nothing else. No `.vtt`: ffmpeg runs with `-sn` and
|
||||
# subtitles are independent tracks served by GET /subtitle, so a WebVTT name here could only ever be
|
||||
# a name that will not arrive — and accepting it cost a 20 s parked worker per request.
|
||||
_HLS_CT = {
|
||||
".m3u8": "application/vnd.apple.mpegurl",
|
||||
".ts": "video/mp2t",
|
||||
".vtt": "text/vtt",
|
||||
}
|
||||
# A real artifact name is ~20 characters. The bound is what keeps an absurd one out of the parser:
|
||||
# `int()` on an unbounded digit run raises ValueError past 4300 digits (CPython's
|
||||
# `sys.get_int_max_str_digits`), which reached the client as a 500 — the same shape of bug the
|
||||
# subtitle routes were bounded for (`_RatingKey`, above).
|
||||
_HLS_NAME_MAX = 128
|
||||
|
||||
|
||||
@router.get("/stream/{rating_key}/hls/{filename}")
|
||||
@router.get("/stream/{rating_key}/hls/{session_id}/{filename}")
|
||||
def stream_hls_file(
|
||||
rating_key: str,
|
||||
session_id: str,
|
||||
filename: str,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> Response:
|
||||
"""Serve a file (playlist / segment / subtitle) from the current HLS session directory. The
|
||||
playlist references its segments/subtitles by relative name, all resolving back here."""
|
||||
"""Serve a file (playlist / segment / subtitle) from ONE HLS session's directory. The playlist
|
||||
references its segments/subtitles by relative name, all resolving back here.
|
||||
|
||||
The session id in the path is what makes "back here" mean the same session. Sessions are keyed by
|
||||
rating_key alone and a seek-restart or a second viewer of the same item replaces the registered
|
||||
one, while the DIRECTORY is per (key, offset, tag) — so resolving by rating_key handed the old
|
||||
viewer the new session's `seg_12.ts`, produced from a different offset under `-copyts`: wrong
|
||||
bytes with a 200, which the player has no way to notice (deep-review round 5). A request for a
|
||||
session that is no longer the registered one is a 404, and the player reclaims at the playhead.
|
||||
|
||||
That reclaim is also why a request with no live session is no longer auto-started here: the
|
||||
auto-start could only ever begin at 0, silently throwing away the viewer's position."""
|
||||
# Filename must be a plain hls artifact — no path separators, only the expected extensions.
|
||||
ext = "." + filename.rsplit(".", 1)[-1] if "." in filename else ""
|
||||
if ext not in _HLS_CT or "/" in filename or "\\" in filename or ".." in filename:
|
||||
if (
|
||||
len(filename) > _HLS_NAME_MAX
|
||||
or ext not in _HLS_CT
|
||||
or "/" in filename
|
||||
or "\\" in filename
|
||||
or ".." in filename
|
||||
):
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
s = plex_stream.current_session(str(rating_key))
|
||||
if s is None:
|
||||
# No session yet (e.g. reaped): the ENTRY playlist may auto-start one at 0 — master.m3u8 for a
|
||||
# multi-audio item, index.m3u8 for a single-audio remux item.
|
||||
it = item_or_404(db, rating_key)
|
||||
if filename == "master.m3u8":
|
||||
s = plex_stream.start_session(db, it, 0.0, multi=True)
|
||||
elif filename == "index.m3u8" and it.playable == "remux":
|
||||
s = plex_stream.start_session(db, it, 0.0)
|
||||
else:
|
||||
raise HTTPException(status_code=404, detail="No active playback session")
|
||||
if s is None:
|
||||
raise HTTPException(status_code=404, detail="Media file not found on disk")
|
||||
if s is None or s.id != session_id:
|
||||
raise HTTPException(status_code=404, detail="No active playback session")
|
||||
path = s.dir / filename
|
||||
timeout = 30 if ext == ".m3u8" else 20
|
||||
if not plex_stream.wait_for(path, timeout):
|
||||
# Only a name THIS session can actually produce is worth waiting for: its own playlists, and a
|
||||
# segment of one of its own renditions whose predecessor is already on disk (ffmpeg writes them
|
||||
# in order, so if it is not there this one is not next). The session is passed, not just its
|
||||
# directory, because the single- and multi-audio argv branches spell their names differently and
|
||||
# neither can produce the other's. Answering now instead of waiting it out is what keeps a seek
|
||||
# past the produced region — or a deliberate flood of hopeless names — from parking threadpool
|
||||
# workers and their DB connections (see `artifact_may_arrive`).
|
||||
if not plex_stream.artifact_may_arrive(s, filename):
|
||||
raise HTTPException(status_code=404, detail="Not available yet")
|
||||
timeout = 30 if ext == ".m3u8" else 20
|
||||
# `proc` so a session whose ffmpeg has already exited answers at once instead of polling out the
|
||||
# whole timeout: this is a SYNC route on the ~40-slot threadpool, holding a pooled DB connection
|
||||
# with it, so every such wait is a worker (and a connection) parked for 20 s — an ordinary seek
|
||||
# past the produced region was enough to do it (round 5).
|
||||
if not plex_stream.wait_for(path, timeout, proc=s.proc):
|
||||
raise HTTPException(status_code=404, detail="Not available yet")
|
||||
try:
|
||||
content = path.read_bytes()
|
||||
except OSError:
|
||||
# The session was torn down between the wait and the read — `_kill` rmtrees the whole
|
||||
# directory, and nothing holds `_lock` across those two steps. It has to present as the same
|
||||
# 404 a gone session does: the player's recovery is keyed on the STATUS (`code === 404`), so
|
||||
# a 500 here skipped the reclaim and ended playback with "couldn't play" instead of
|
||||
# restarting at the playhead. Ordinary trigger: `loadSession` awaits the new session — which
|
||||
# kills and rmtrees the old one — before destroying the old hls instance, so the outgoing
|
||||
# instance's in-flight segment request races the rmtree on every seek-restart.
|
||||
raise HTTPException(status_code=404, detail="No active playback session") from None
|
||||
return Response(
|
||||
content=path.read_bytes(),
|
||||
content=content,
|
||||
media_type=_HLS_CT[ext],
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
@@ -25,7 +25,7 @@ router = APIRouter(prefix="/api/plex", tags=["plex"])
|
||||
|
||||
|
||||
def _own_playlist_or_404(db: Session, pid: int, user: User) -> PlexPlaylist:
|
||||
return owned_or_404(db, PlexPlaylist, pid, user, detail="Playlist not found")
|
||||
return owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
|
||||
|
||||
|
||||
def _playlist_card(pl: PlexPlaylist, count: int, thumb_rk: str | None) -> dict:
|
||||
|
||||
@@ -91,7 +91,7 @@ def create_view(
|
||||
|
||||
|
||||
def _own_view(db: Session, user: User, view_id: int) -> SavedView:
|
||||
return owned_or_404(db, SavedView, view_id, user, detail="Unknown view")
|
||||
return owned_or_404(db, SavedView, view_id, user, what="view")
|
||||
|
||||
|
||||
class UpdateViewIn(BaseModel):
|
||||
|
||||
@@ -23,6 +23,7 @@ from app.scheduler import (
|
||||
trigger_job,
|
||||
)
|
||||
from app.sync.runner import estimate_deep_backfill
|
||||
from app.routes._common import not_found
|
||||
|
||||
router = APIRouter(prefix="/api/admin/scheduler", tags=["admin"])
|
||||
|
||||
@@ -41,7 +42,7 @@ def set_job_interval(
|
||||
"""Change how often a scheduler job runs (minutes). Persisted as an override and applied
|
||||
to the live scheduler immediately."""
|
||||
if job_id not in JOB_INTERVALS:
|
||||
raise HTTPException(status_code=404, detail="Unknown job")
|
||||
raise not_found("job")
|
||||
minutes = body.interval_minutes
|
||||
if not (MIN_INTERVAL <= minutes <= MAX_INTERVAL):
|
||||
raise HTTPException(
|
||||
@@ -73,7 +74,7 @@ def run_job_now(
|
||||
try:
|
||||
result = trigger_job(job_id, actor_id=user.id)
|
||||
except KeyError:
|
||||
raise HTTPException(status_code=404, detail="Unknown job")
|
||||
raise not_found("job")
|
||||
if result == "already_running":
|
||||
raise HTTPException(status_code=409, detail="Job is already running")
|
||||
return {"id": job_id, "started": True}
|
||||
|
||||
@@ -14,7 +14,9 @@ from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import email as email_mod
|
||||
from app import mailtext
|
||||
from app import state, sysconfig
|
||||
from app.utils import canonical_email
|
||||
from app.auth import PASSWORD_MIN_LEN
|
||||
from app.db import get_db
|
||||
from app.models import User
|
||||
@@ -69,7 +71,13 @@ def setup_admin(body: SetupAdminIn, db: Session = Depends(require_setup)) -> dic
|
||||
raise HTTPException(
|
||||
status_code=400, detail=f"Password must be at least {PASSWORD_MIN_LEN} characters."
|
||||
)
|
||||
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
|
||||
# Resolve canonically, not by the literal string (DX-17). The wizard can be re-run, and E9.5's
|
||||
# whole point is that two spellings of one Gmail mailbox are one person — matching literally here
|
||||
# would let a second run under the other spelling create the DUPLICATE admin row the epic exists
|
||||
# to prevent. Every other lookup went canonical in S3; this one was outside that diff.
|
||||
user = db.execute(
|
||||
select(User).where(User.email_canonical == canonical_email(email))
|
||||
).scalar_one_or_none()
|
||||
if user is None:
|
||||
user = User(email=email, role="admin")
|
||||
db.add(user)
|
||||
@@ -103,6 +111,9 @@ def setup_config(settings_map: dict[str, Any], db: Session = Depends(require_set
|
||||
|
||||
class SetupTestEmailIn(BaseModel):
|
||||
to: str = ""
|
||||
# The wizard's page language. No account exists yet on a fresh instance, so this is the only
|
||||
# thing that can say what language to write in; anything unsupported falls back to English.
|
||||
lang: str | None = None
|
||||
|
||||
|
||||
@router.post("/test-email")
|
||||
@@ -111,7 +122,7 @@ def setup_test_email(body: SetupTestEmailIn, db: Session = Depends(require_setup
|
||||
to = body.to.strip()
|
||||
if "@" not in to:
|
||||
raise HTTPException(status_code=400, detail="Enter a valid email address.")
|
||||
if not email_mod.send_test(to):
|
||||
if not email_mod.send_test(to, mailtext.normalize(body.lang)):
|
||||
raise HTTPException(status_code=400, detail="Couldn't send — check the SMTP settings.")
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@@ -7,12 +7,20 @@ from app.audit import AuditAction
|
||||
from app.auth import admin_user, current_user, has_read_scope, require_human
|
||||
from app.db import get_db
|
||||
from app.models import Channel, Subscription, User, Video
|
||||
from app.ratelimit import RateLimiter
|
||||
from app.sync.runner import estimate_deep_backfill
|
||||
from app.sync.subscriptions import import_subscriptions
|
||||
from app.scheduler import running_job_ids
|
||||
|
||||
router = APIRouter(prefix="/api/sync", tags=["sync"])
|
||||
|
||||
# SEC-14: the YouTube API budget is SHARED across the whole instance, and a subscription import
|
||||
# walks every page of the caller's subscriptions — so one user holding down a manual sync button can
|
||||
# spend the day's units for everyone. Keyed by user id, not client IP: the cost belongs to the
|
||||
# account, and the same person on a second device is the same drain. 3 per 10 minutes leaves manual
|
||||
# re-syncing entirely usable (the scheduler does the routine work anyway) while removing the loop.
|
||||
_sync_limiter = RateLimiter(max_events=3, window_seconds=600)
|
||||
|
||||
|
||||
@router.post("/subscriptions")
|
||||
def sync_subscriptions(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
|
||||
@@ -22,6 +30,8 @@ def sync_subscriptions(user: User = Depends(require_human), db: Session = Depend
|
||||
status_code=403,
|
||||
detail="Connect your YouTube account (read access) to import subscriptions.",
|
||||
)
|
||||
if not _sync_limiter.allow(str(user.id)):
|
||||
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
|
||||
with quota.measured(db, user.id, quota.QuotaAction.SUBSCRIPTIONS_PULL) as m:
|
||||
result = import_subscriptions(db, user)
|
||||
result["quota_used_estimate"] = m.used
|
||||
|
||||
@@ -80,7 +80,7 @@ def create_tag(
|
||||
|
||||
def _own_tag(db: Session, user: User, tag_id: int) -> Tag:
|
||||
# System tags (user_id NULL) are not user-editable → owned_or_404 404s them too.
|
||||
return owned_or_404(db, Tag, tag_id, user, detail="Unknown tag")
|
||||
return owned_or_404(db, Tag, tag_id, user, what="tag")
|
||||
|
||||
|
||||
class TagUpdateIn(BaseModel):
|
||||
|
||||
@@ -167,6 +167,8 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
ConfigSpec(
|
||||
"plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
|
||||
),
|
||||
# Audio loudness normalisation on the HLS path (remux + transcode alike, not direct play).
|
||||
ConfigSpec("plex_audio_loudnorm", "bool", "plex", "plex_audio_loudnorm"),
|
||||
# NOTE: plex_watch_sync_interval_min is a config.py default read by the watch-sync scheduler job
|
||||
# (scheduler.py); it is intentionally NOT a runtime admin knob, so it stays out of SPECS.
|
||||
# --- TMDB enrichment (E5 S2; optional) — the API key is the only secret ---
|
||||
|
||||
@@ -22,6 +22,39 @@ def valid_email(value: str | None) -> bool:
|
||||
return bool(value and _EMAIL_RE.match(value))
|
||||
|
||||
|
||||
# Gmail (and its googlemail.com alias) delivers to one mailbox regardless of dots in the local part
|
||||
# and of any `+tag`; nobody else is guaranteed to. So the canonical form below strips those for
|
||||
# Gmail only, and merely lowercases everything else.
|
||||
_GMAIL_DOMAINS = {"gmail.com", "googlemail.com"}
|
||||
|
||||
|
||||
def canonical_email(value: str | None) -> str:
|
||||
"""The form used to decide whether two addresses are the SAME MAILBOX. Never used for sending —
|
||||
mail always goes to the address as the user typed it.
|
||||
|
||||
Why this exists (observed on prod, 2026-09-03): someone registered as `npeter83.temu@gmail.com`
|
||||
and was approved, but Google's OAuth `email` claim came back as the account's stored primary,
|
||||
`npeter83temu@gmail.com`. The invite gate only lowercased, found no approved row for that literal
|
||||
string, and turned an approved user's sign-in into a silent SECOND access request — plus a second
|
||||
User row. The `+tag` case is worse and fails every time: a Google account cannot contain `+`, so
|
||||
OAuth can only ever return the base address, meaning anyone who registers with
|
||||
`user+siftlode@gmail.com` (a common throwaway habit) is guaranteed to be denied after approval.
|
||||
|
||||
Scope is deliberately narrow. The dot rule does NOT hold for Google Workspace custom domains
|
||||
unless the admin turned it on, and dots are significant on most providers, so anything outside
|
||||
gmail.com/googlemail.com is only lowercased. Approving one Gmail spelling always did mean
|
||||
approving that mailbox, so collapsing them is the honest reading of an invite, not a widening
|
||||
of it.
|
||||
"""
|
||||
email = (value or "").strip().lower()
|
||||
local, sep, domain = email.partition("@")
|
||||
if not sep or domain not in _GMAIL_DOMAINS:
|
||||
return email
|
||||
local = local.partition("+")[0].replace(".", "")
|
||||
# Normalise the alias domain too, so googlemail and gmail spellings meet.
|
||||
return f"{local}@gmail.com" if local else email
|
||||
|
||||
|
||||
def now_utc() -> datetime:
|
||||
"""Current time as a timezone-aware UTC datetime."""
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
@@ -189,3 +189,24 @@ def as_user(db, override_dep, _configured):
|
||||
return TestClient(app, headers={"x-test-user": str(user.id)})
|
||||
|
||||
return _as
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_rate_limiters():
|
||||
"""Every RateLimiter is a module-level singleton keyed by client IP, and TestClient always
|
||||
presents the same host — so without this the whole suite's login/register attempts pool into one
|
||||
300s window and, past the cap, some unrelated test starts failing with a 429 depending on order.
|
||||
Cleared before each test; a test that wants to exercise throttling can still exhaust one itself.
|
||||
|
||||
Sweeps every loaded `app.*` module rather than a hand-kept list, so a limiter added in a new
|
||||
module is covered the day it lands instead of silently pooling until something fails."""
|
||||
import sys
|
||||
|
||||
from app.ratelimit import RateLimiter
|
||||
|
||||
for name, mod in list(sys.modules.items()):
|
||||
if name != "app" and not name.startswith("app."):
|
||||
continue
|
||||
for obj in vars(mod).values():
|
||||
if isinstance(obj, RateLimiter):
|
||||
obj.reset()
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
"""E12 · S3 — the route-level halves of SEC-12 and SEC-14.
|
||||
|
||||
`test_urlguard.py` pins the address judgement itself; this pins that the ENQUEUE route actually
|
||||
consults it and answers in the structured `{code, reason}` shape the frontend localizes, and that a
|
||||
manual sync trigger stops spending the instance-wide YouTube budget when held down.
|
||||
|
||||
DB-backed (the `as_user` fixture needs real rows) — skipped when no database is reachable.
|
||||
"""
|
||||
|
||||
from app.downloads import urlguard
|
||||
|
||||
|
||||
def test_enqueueing_a_loopback_url_is_refused_with_a_structured_detail(
|
||||
db, as_user, make_user, monkeypatch
|
||||
):
|
||||
user = make_user()
|
||||
db.commit()
|
||||
monkeypatch.setattr(
|
||||
urlguard.socket, "getaddrinfo", lambda *a, **kw: [(None, None, None, "", ("127.0.0.1", 0))]
|
||||
)
|
||||
r = as_user(user).post("/api/downloads", json={"source": "http://redis.internal:6379/x"})
|
||||
assert r.status_code == 422
|
||||
# The shape matters as much as the refusal: a bare English string under a bilingual UI is the
|
||||
# thing the canonical-patterns table forbids.
|
||||
assert r.json()["detail"] == {"code": "url_blocked", "reason": urlguard.REASON_PRIVATE}
|
||||
|
||||
|
||||
def test_a_public_url_gets_past_the_guard(db, as_user, make_user, monkeypatch):
|
||||
# Mutation check for the test above: with a public answer the request must fail LATER (or
|
||||
# succeed), never with url_blocked — otherwise the guard would be refusing everything and the
|
||||
# first test would still pass.
|
||||
user = make_user()
|
||||
db.commit()
|
||||
monkeypatch.setattr(
|
||||
urlguard.socket,
|
||||
"getaddrinfo",
|
||||
lambda *a, **kw: [(None, None, None, "", ("93.184.216.34", 0))],
|
||||
)
|
||||
r = as_user(user).post("/api/downloads", json={"source": "https://example.com/v"})
|
||||
detail = r.json().get("detail") if r.status_code >= 400 else None
|
||||
assert not (isinstance(detail, dict) and detail.get("code") == "url_blocked")
|
||||
|
||||
|
||||
def test_youtube_sources_never_consult_the_guard(db, as_user, make_user, monkeypatch):
|
||||
# A YouTube job resolves to a fixed public host from an 11-char id, so there is no
|
||||
# user-controlled destination — resolving one would be pure latency on every enqueue.
|
||||
def boom(*a, **kw):
|
||||
raise AssertionError("the URL guard resolved DNS for a YouTube source")
|
||||
|
||||
user = make_user()
|
||||
db.commit()
|
||||
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
|
||||
as_user(user).post("/api/downloads", json={"source": "dQw4w9WgXcQ"})
|
||||
|
||||
|
||||
def test_manual_subscription_sync_goes_on_cooldown(db, as_user, make_user, monkeypatch):
|
||||
from app.routes import sync as syncroutes
|
||||
|
||||
user = make_user()
|
||||
db.commit()
|
||||
# Neither the YouTube grant nor the import itself is under test — the limiter is.
|
||||
monkeypatch.setattr(syncroutes, "has_read_scope", lambda u: True)
|
||||
monkeypatch.setattr(syncroutes, "import_subscriptions", lambda d, u: {"imported": 0})
|
||||
|
||||
c = as_user(user)
|
||||
codes = [c.post("/api/sync/subscriptions").status_code for _ in range(4)]
|
||||
# 3 per 10 minutes: the first three pass, the fourth is refused.
|
||||
assert codes[:3] == [200, 200, 200]
|
||||
assert codes[3] == 429
|
||||
assert c.post("/api/sync/subscriptions").json()["detail"] == {"code": "sync_cooldown"}
|
||||
|
||||
|
||||
def test_the_cooldown_is_per_user_not_global(db, as_user, make_user, monkeypatch):
|
||||
# Keyed by user id on purpose: the shared YouTube budget is spent per account, and one user
|
||||
# exhausting their allowance must not lock everyone else out of their own sync button.
|
||||
from app.routes import sync as syncroutes
|
||||
|
||||
a, b = make_user(), make_user()
|
||||
db.commit()
|
||||
monkeypatch.setattr(syncroutes, "has_read_scope", lambda u: True)
|
||||
monkeypatch.setattr(syncroutes, "import_subscriptions", lambda d, u: {"imported": 0})
|
||||
|
||||
ca = as_user(a)
|
||||
for _ in range(4):
|
||||
ca.post("/api/sync/subscriptions")
|
||||
assert ca.post("/api/sync/subscriptions").status_code == 429
|
||||
assert as_user(b).post("/api/sync/subscriptions").status_code == 200
|
||||
|
||||
|
||||
def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_user, monkeypatch):
|
||||
# SEC-14, the sweep half: `revert-youtube` spends under the SAME QuotaAction.PLAYLISTS_PULL key
|
||||
# as the bulk `sync-youtube`, so limiting only one of them left the identical drain open one
|
||||
# route over. Both must draw on the one allowance.
|
||||
from app.models import Playlist
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user = make_user()
|
||||
pl = Playlist(user_id=user.id, name="Linked", source="youtube", yt_playlist_id="PL123")
|
||||
db.add(pl)
|
||||
db.commit()
|
||||
monkeypatch.setattr(plroutes, "has_read_scope", lambda u: True)
|
||||
monkeypatch.setattr(plroutes, "repull_playlist", lambda d, u, p: {"ok": True})
|
||||
monkeypatch.setattr(plroutes, "sync_user_playlists", lambda d, u: {"ok": True})
|
||||
|
||||
c = as_user(user)
|
||||
codes = [c.post(f"/api/playlists/{pl.id}/revert-youtube").status_code for _ in range(4)]
|
||||
assert codes[:3] == [200, 200, 200]
|
||||
assert codes[3] == 429
|
||||
# One allowance, not one each: the bulk route is now on cooldown too.
|
||||
r = c.post("/api/playlists/sync-youtube")
|
||||
assert r.status_code == 429
|
||||
assert r.json()["detail"] == {"code": "sync_cooldown"}
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Deleting a whitelist / access-request entry (admin).
|
||||
|
||||
Deny records an auditable decision and keeps the row; delete forgets the address outright, for an
|
||||
approval added by mistake or one that is no longer wanted. Neither stops the person asking again —
|
||||
`upsert_pending_invite` reopens a denied row as pending, and a deleted one comes back as a fresh
|
||||
request. The one hard rule is that an entry backed by an account is NOT deletable here: this row is
|
||||
that account's approval gate, and orphaning it would strand someone who could then neither sign in
|
||||
nor be approved.
|
||||
"""
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.models import Invite, User
|
||||
|
||||
|
||||
def _invites(client):
|
||||
r = client.get("/api/admin/invites")
|
||||
assert r.status_code == 200
|
||||
return {i["email"]: i for i in r.json()}
|
||||
|
||||
|
||||
def test_admin_deletes_a_whitelist_entry_that_never_became_an_account(db, make_user, as_user):
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="mistake@example.com", status="approved", decided_by=admin.email))
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
listed = _invites(client)["mistake@example.com"]
|
||||
assert listed["has_account"] is False # drives the delete affordance in the UI
|
||||
|
||||
r = client.delete(f"/api/admin/invites/{listed['id']}")
|
||||
|
||||
assert r.status_code == 200 and r.json()["deleted"] == listed["id"]
|
||||
db.expire_all()
|
||||
assert (
|
||||
db.execute(select(Invite).where(Invite.email == "mistake@example.com")).scalar_one_or_none()
|
||||
is None
|
||||
)
|
||||
|
||||
|
||||
def test_deleting_an_entry_that_has_an_account_is_refused(db, make_user, as_user):
|
||||
# The guard that matters: this row is what is_allowed() reads, so deleting it would revoke the
|
||||
# user's access without touching the account or saying so.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
member = make_user(email="member@example.com")
|
||||
db.add(Invite(email=member.email, status="approved", decided_by=admin.email))
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
listed = _invites(client)[member.email]
|
||||
assert listed["has_account"] is True
|
||||
|
||||
r = client.delete(f"/api/admin/invites/{listed['id']}")
|
||||
|
||||
assert r.status_code == 409
|
||||
db.expire_all()
|
||||
assert (
|
||||
db.execute(select(Invite).where(Invite.email == member.email)).scalar_one_or_none()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def test_a_pending_request_from_a_registration_is_not_deletable(db, make_user, as_user):
|
||||
# The shape the ordinary sign-up produces: _register_account writes the User (inactive) AND the
|
||||
# pending Invite together. The invite is that account's approval gate, so deleting it would
|
||||
# strand the person — they could neither sign in nor be approved, and re-registering is a silent
|
||||
# no-op. Managed from Users & roles instead, which is what the UI and the 409 both say.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
make_user(email="signup@example.com", is_active=False, email_verified=False)
|
||||
db.add(Invite(email="signup@example.com", status="pending"))
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
listed = _invites(client)["signup@example.com"]
|
||||
assert listed["has_account"] is True # the UI shows a disabled control, not a missing one
|
||||
|
||||
r = client.delete(f"/api/admin/invites/{listed['id']}")
|
||||
|
||||
assert r.status_code == 409
|
||||
db.expire_all()
|
||||
assert (
|
||||
db.execute(select(Invite).where(Invite.email == "signup@example.com")).scalar_one_or_none()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def test_deleting_an_env_allow_listed_address_reports_that_it_kept_access(
|
||||
db, make_user, as_user, monkeypatch
|
||||
):
|
||||
# Removing the row is NOT always a revoke: is_allowed falls back to the env bootstrap lists once
|
||||
# no invite exists. The admin has to be told, or they think they removed access and did not.
|
||||
from app.config import settings
|
||||
|
||||
# Patch the raw env field, not the derived property: that keeps the split/strip/lower parsing
|
||||
# under test, and matches how the rest of the suite stubs settings.
|
||||
monkeypatch.setattr(settings, "allowed_emails", " OPS@Example.com , someone@else.test ")
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="ops@example.com", status="approved", decided_by=admin.email))
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
listed = _invites(client)["ops@example.com"]
|
||||
# The LIST carries it too, so the confirm dialog can warn before the delete happens.
|
||||
assert listed["still_allowed_by_env"] is True
|
||||
|
||||
r = client.delete(f"/api/admin/invites/{listed['id']}")
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json()["still_allowed_by_env"] is True
|
||||
|
||||
|
||||
def test_an_admin_emails_address_is_also_reported(db, make_user, as_user, monkeypatch):
|
||||
# ADMIN_EMAILS is the sharper case: deleting the row not only keeps access, the next Google
|
||||
# sign-in re-promotes the address to admin.
|
||||
from app.config import settings
|
||||
|
||||
monkeypatch.setattr(settings, "admin_emails", "boss@example.com")
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="boss@example.com", status="approved", decided_by=admin.email))
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
|
||||
assert _invites(client)["boss@example.com"]["still_allowed_by_env"] is True
|
||||
|
||||
|
||||
def test_deleting_a_pending_request_forgets_it(db, make_user, as_user):
|
||||
# A pending row is deletable too — "drop this without recording a decision" is a different
|
||||
# choice from Deny, which stores a decision (status, decided_by) and mails the requester.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="asker@example.com", status="pending"))
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
|
||||
r = client.delete(f"/api/admin/invites/{_invites(client)['asker@example.com']['id']}")
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json()["still_allowed_by_env"] is False # nothing in the env allow-lists
|
||||
assert "asker@example.com" not in _invites(client)
|
||||
|
||||
|
||||
def test_deleting_a_missing_invite_is_404(db, make_user, as_user):
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
assert as_user(admin).delete("/api/admin/invites/9999").status_code == 404
|
||||
|
||||
|
||||
def test_a_regular_user_cannot_delete_an_invite(db, make_user, as_user):
|
||||
make_user(email="admin@example.com", role="admin")
|
||||
plain = make_user(email="plain@example.com")
|
||||
db.add(Invite(email="mistake@example.com", status="approved"))
|
||||
db.commit()
|
||||
inv_id = db.execute(select(Invite.id).where(Invite.email == "mistake@example.com")).scalar_one()
|
||||
|
||||
assert as_user(plain).delete(f"/api/admin/invites/{inv_id}").status_code == 403
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(Invite, inv_id) is not None
|
||||
|
||||
|
||||
def test_deleting_the_user_also_clears_the_way_to_re_invite(db, make_user, as_user):
|
||||
# The refusal above tells the admin to delete the account instead; purge_user takes the invite
|
||||
# with it, so the address really is forgotten and a fresh request starts over as pending.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
member = make_user(email="member@example.com")
|
||||
db.add(Invite(email=member.email, status="approved", decided_by=admin.email))
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
|
||||
assert client.delete(f"/api/admin/users/{member.id}").status_code == 200
|
||||
|
||||
db.expire_all()
|
||||
assert db.execute(select(User).where(User.email == member.email)).scalar_one_or_none() is None
|
||||
assert (
|
||||
db.execute(select(Invite).where(Invite.email == member.email)).scalar_one_or_none() is None
|
||||
)
|
||||
@@ -0,0 +1,203 @@
|
||||
"""The admin lifeline for a sign-up that is stuck outside (E9.5 S5, roadmap UX-25), DB-lane.
|
||||
|
||||
Approve an address BEFORE the person registers and their sign-up produces no queue row at all —
|
||||
`_decide` only takes a still-pending invite — so the Access-requests screen stayed empty while they
|
||||
sat locked out, and the badge read zero. The only working move was re-typing the address into the
|
||||
whitelist box, which nobody would guess.
|
||||
|
||||
Two earlier attempts at this were reverted after review, and both moved AUTHORITY rather than
|
||||
visibility: auto-activating on registration, and reopening the approved invite to pending. So the
|
||||
tests here pin the boundary as much as the feature — an admin click, gated on an address that is
|
||||
already allowed, and nothing an unauthenticated caller can reach.
|
||||
"""
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.models import AuditLog, AuthToken, Invite, User
|
||||
|
||||
_PW = "correct horse battery"
|
||||
|
||||
|
||||
def _pending_access(client):
|
||||
r = client.get("/api/admin/pending-access")
|
||||
assert r.status_code == 200
|
||||
return {u["email"]: u for u in r.json()}
|
||||
|
||||
|
||||
def _stuck_registrant(db, make_user, admin):
|
||||
"""The UX-25 shape: the admin approved the address first, the account arrived afterwards."""
|
||||
db.add(Invite(email="later@example.com", status="approved", decided_by=admin.email))
|
||||
db.commit()
|
||||
return make_user(email="later@example.com", password=_PW, is_active=False)
|
||||
|
||||
|
||||
def test_a_pre_approved_registrant_is_listed_and_offers_activation(db, make_user, as_user):
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
_stuck_registrant(db, make_user, admin)
|
||||
|
||||
row = _pending_access(as_user(admin))["later@example.com"]
|
||||
|
||||
assert row["is_active"] is False
|
||||
assert (
|
||||
row["can_activate"] is True
|
||||
) # the address is already allowed — the click just finishes it
|
||||
|
||||
|
||||
def test_an_ordinary_registrant_awaiting_a_decision_is_not_listed_twice(db, make_user, as_user):
|
||||
# They already have a pending invite with Approve on it in the queue above, and their address is
|
||||
# not allowed YET — the decision is the queue's to own. Caught in UAT on 2026-09-05: an earlier
|
||||
# filter keyed on "unconfirmed" put every fresh registration in both lists at once.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="normal@example.com", status="pending"))
|
||||
db.commit()
|
||||
make_user(email="normal@example.com", password=_PW, is_active=False)
|
||||
|
||||
assert "normal@example.com" not in _pending_access(as_user(admin))
|
||||
|
||||
|
||||
def test_a_fresh_registration_stays_in_the_queue_and_out_of_this_list(db, make_user, as_user):
|
||||
# The realistic version of the case above: a brand-new sign-up is inactive AND unconfirmed. It
|
||||
# still belongs only to the queue — the admin's next act is the decision, not a re-send.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="fresh@example.com", status="pending"))
|
||||
db.commit()
|
||||
make_user(email="fresh@example.com", password=_PW, is_active=False, email_verified=False)
|
||||
|
||||
assert "fresh@example.com" not in _pending_access(as_user(admin))
|
||||
|
||||
|
||||
def test_activating_lets_them_in_and_is_recorded(db, make_user, as_user):
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
stuck = _stuck_registrant(db, make_user, admin)
|
||||
|
||||
r = as_user(admin).post(f"/api/admin/users/{stuck.id}/activate")
|
||||
|
||||
assert r.status_code == 200 and r.json()["is_active"] is True
|
||||
db.expire_all()
|
||||
assert db.get(User, stuck.id).is_active is True
|
||||
actions = db.execute(select(AuditLog.action)).scalars().all()
|
||||
assert (
|
||||
"user_activate" in actions
|
||||
) # the admin acted on someone's access; it is not a silent read
|
||||
|
||||
|
||||
def test_activating_an_address_that_is_not_allowed_is_refused(db, make_user, as_user):
|
||||
# The guard that keeps this button from manufacturing an inconsistent account: password sign-in
|
||||
# only checks is_active and would let them in, while their Google sign-in would be turned away
|
||||
# and silently reopened as a fresh access request. Whitelisting is the move that says "let them
|
||||
# in"; this one only finishes a decision already made.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
nobody = make_user(email="nobody@example.com", password=_PW, is_active=False)
|
||||
|
||||
r = as_user(admin).post(f"/api/admin/users/{nobody.id}/activate")
|
||||
|
||||
assert r.status_code == 409
|
||||
assert r.json()["detail"]["reason"] == "not_allowed"
|
||||
db.expire_all()
|
||||
assert db.get(User, nobody.id).is_active is False
|
||||
|
||||
|
||||
def test_activating_twice_is_not_an_error(db, make_user, as_user):
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
stuck = _stuck_registrant(db, make_user, admin)
|
||||
client = as_user(admin)
|
||||
|
||||
assert client.post(f"/api/admin/users/{stuck.id}/activate").status_code == 200
|
||||
assert client.post(f"/api/admin/users/{stuck.id}/activate").status_code == 200
|
||||
|
||||
|
||||
def test_a_suspended_account_is_not_offered_as_stuck(db, make_user, as_user):
|
||||
# It cannot sign in either, but that is a decision already made — not a person waiting on one.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="banned@example.com", status="approved", decided_by=admin.email))
|
||||
db.commit()
|
||||
make_user(email="banned@example.com", password=_PW, is_active=False, is_suspended=True)
|
||||
|
||||
assert "banned@example.com" not in _pending_access(as_user(admin))
|
||||
|
||||
|
||||
# --- the confirmation resend ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_an_unconfirmed_account_is_listed_and_can_be_sent_a_fresh_link(
|
||||
db, make_user, as_user, monkeypatch
|
||||
):
|
||||
from app.routes import admin as admin_routes
|
||||
from app.routes import auth as auth_routes
|
||||
|
||||
monkeypatch.setattr(admin_routes.email_mod, "email_enabled", lambda db=None: True)
|
||||
sent: list[tuple[str, str]] = []
|
||||
monkeypatch.setattr(
|
||||
auth_routes.email_mod,
|
||||
"send_verify_email",
|
||||
lambda to, lang, link, **kw: sent.append((to, link)) is None,
|
||||
)
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
# Active but unconfirmed — a real state since E9.5 S1 made the two gates order-independent. It is
|
||||
# not the admin's to decide, but the mail may simply never have arrived, so it is theirs to help
|
||||
# with. The approved invite is what puts them on this screen at all: the list is "the address may
|
||||
# sign in, this account still can't", so an undecided request stays with the queue that owns it.
|
||||
db.add(Invite(email="unconfirmed@example.com", status="approved", decided_by=admin.email))
|
||||
db.commit()
|
||||
u = make_user(email="unconfirmed@example.com", password=_PW, email_verified=False)
|
||||
|
||||
assert "unconfirmed@example.com" in _pending_access(as_user(admin))
|
||||
r = as_user(admin).post(f"/api/admin/users/{u.id}/resend-verification")
|
||||
|
||||
assert r.status_code == 200 and r.json()["email"] == "unconfirmed@example.com"
|
||||
assert [to for to, _ in sent] == ["unconfirmed@example.com"]
|
||||
tokens = db.execute(select(AuthToken).where(AuthToken.user_id == u.id)).scalars().all()
|
||||
assert [t.kind for t in tokens] == ["verify"]
|
||||
|
||||
|
||||
def test_the_resend_says_so_when_the_instance_cannot_send_mail(db, make_user, as_user, monkeypatch):
|
||||
from app.routes import admin as admin_routes
|
||||
|
||||
monkeypatch.setattr(admin_routes.email_mod, "email_enabled", lambda db=None: False)
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
u = make_user(email="unconfirmed@example.com", password=_PW, email_verified=False)
|
||||
|
||||
r = as_user(admin).post(f"/api/admin/users/{u.id}/resend-verification")
|
||||
|
||||
# Mail is fail-soft everywhere else, which is exactly what left the 0.78.5 sign-ups with nothing
|
||||
# to fall back on. An admin pressing a button gets the failure, not a success toast.
|
||||
assert r.status_code == 409 and r.json()["detail"]["reason"] == "email_off"
|
||||
|
||||
|
||||
def test_resending_to_an_already_confirmed_account_is_refused(db, make_user, as_user):
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
u = make_user(email="fine@example.com", password=_PW)
|
||||
|
||||
r = as_user(admin).post(f"/api/admin/users/{u.id}/resend-verification")
|
||||
|
||||
assert r.status_code == 409 and r.json()["detail"]["reason"] == "not_applicable"
|
||||
|
||||
|
||||
# --- the badge -----------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_badge_counts_stuck_sign_ups_but_not_ones_the_queue_already_shows(
|
||||
db, make_user, client
|
||||
):
|
||||
# Signed in for real, not via the as_user override: /api/me resolves its user through
|
||||
# optional_current_user, which that override does not cover (see test_me_email_flag).
|
||||
admin = make_user(email="admin@example.com", password=_PW, role="admin")
|
||||
_stuck_registrant(db, make_user, admin) # approved first → counts
|
||||
db.add(Invite(email="normal@example.com", status="pending")) # already in the queue → doesn't
|
||||
db.commit()
|
||||
make_user(email="normal@example.com", password=_PW, is_active=False)
|
||||
make_user(email="unconfirmed@example.com", password=_PW, email_verified=False) # not ours
|
||||
|
||||
assert (
|
||||
client.post(
|
||||
"/auth/password-login", json={"email": admin.email, "password": _PW}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
me = client.get("/api/me").json()
|
||||
assert me["authenticated"] is True
|
||||
|
||||
# An unconfirmed account waits on its OWNER, not on the admin — listing it is help, badging it
|
||||
# would be nagging about something the admin did not cause.
|
||||
assert me["pending_activations"] == 1
|
||||
assert me["pending_invites"] == 1 # unchanged meaning: still just the undecided requests
|
||||
@@ -97,28 +97,85 @@ def test_reset_confirm_is_single_use_and_kills_existing_sessions(client, make_us
|
||||
assert replay.status_code == 400
|
||||
|
||||
|
||||
def test_activating_an_unverified_account_drops_its_preset_password(db, make_user):
|
||||
# SEC-10 (account pre-emption): an attacker registers victim@x with a password before the approved
|
||||
# owner arrives. The address was never verified, so admin approval must NOT hand that pre-set
|
||||
# password a working login — it is dropped and any session it minted is revoked.
|
||||
def test_approving_an_unverified_account_keeps_its_password(db, make_user):
|
||||
# E9.5 S1, inverting an earlier rule. Approval used to drop a password set while the email was
|
||||
# unverified (SEC-10, account pre-emption). It was dropped because it destroyed the password a
|
||||
# legitimate registrant had just chosen, in what is the NORMAL ordering — the admin decides
|
||||
# before the user reads their mail — making "your password was cleared for security" the default
|
||||
# experience.
|
||||
#
|
||||
# S1 ALSO argued the revoke was redundant, because password_login refuses an unverified account
|
||||
# before it reads is_active. The epic's closing review refuted that: the gate holds only while
|
||||
# the address is unverified, and S6's approval mail exists to get the real owner to end that
|
||||
# state. Do not restore that reasoning from this comment — the residual is SEC-16, and round 2
|
||||
# of the review tried and reverted a guard for it (see test_review_round.py).
|
||||
from app.routes.admin import _activate_user_by_email
|
||||
|
||||
victim = make_user(
|
||||
email="victim@example.com",
|
||||
password="attacker-chosen",
|
||||
user = make_user(
|
||||
email="approved-first@example.com",
|
||||
password="chosen at registration",
|
||||
email_verified=False,
|
||||
is_active=False,
|
||||
)
|
||||
epoch0 = victim.session_epoch or 0
|
||||
_activate_user_by_email(db, "victim@example.com")
|
||||
assert victim.is_active is True
|
||||
assert victim.password_hash is None
|
||||
assert (victim.session_epoch or 0) == epoch0 + 1
|
||||
epoch0 = user.session_epoch or 0
|
||||
|
||||
_activate_user_by_email(db, "approved-first@example.com")
|
||||
|
||||
assert user.is_active is True
|
||||
assert user.password_hash is not None
|
||||
assert user.password_revoked_at is None
|
||||
assert (user.session_epoch or 0) == epoch0 # nothing to revoke, so no epoch bump either
|
||||
|
||||
|
||||
def test_an_approved_but_unverified_account_still_cannot_sign_in(client, make_user):
|
||||
# The gate that stops a pre-emptor BEFORE the address is confirmed, whatever the admin decided.
|
||||
# It is not the whole story — confirming the address opens it, which is SEC-16 — but it is the
|
||||
# invariant this test owns, and a change that let an unverified account sign in would be a
|
||||
# straightforwardly worse bug than the residual.
|
||||
make_user(
|
||||
email="notyet@example.com",
|
||||
password="attacker-chosen",
|
||||
email_verified=False,
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
r = _login(client, "notyet@example.com", "attacker-chosen")
|
||||
|
||||
assert r.status_code == 403
|
||||
assert r.json()["detail"]["reason"] == "email_unverified"
|
||||
|
||||
|
||||
def test_verifying_afterwards_lets_the_original_password_in(client, db, make_user):
|
||||
# The payoff for the legitimate registrant: the password chosen at registration still works once
|
||||
# the mailbox is proven — no "your password was cleared for security" detour, no reset round-trip.
|
||||
from app.routes.auth import _issue_token
|
||||
|
||||
user = make_user(
|
||||
email="ordering@example.com",
|
||||
password="chosen at registration",
|
||||
email_verified=False,
|
||||
is_active=False,
|
||||
)
|
||||
from app.routes.admin import _activate_user_by_email
|
||||
|
||||
_activate_user_by_email(db, "ordering@example.com") # admin decides FIRST — the common ordering
|
||||
raw = _issue_token(db, user, "verify", timedelta(hours=1))
|
||||
db.commit()
|
||||
|
||||
verified = client.post("/auth/verify", json={"token": raw})
|
||||
|
||||
assert verified.status_code == 200
|
||||
assert verified.json() == {
|
||||
"ok": True,
|
||||
"password_reset_required": False,
|
||||
"email": "ordering@example.com",
|
||||
}
|
||||
assert _login(client, "ordering@example.com", "chosen at registration").status_code == 200
|
||||
|
||||
|
||||
def test_activating_a_verified_account_keeps_its_password(db, make_user):
|
||||
# A legit account whose email IS verified (e.g. a no-SMTP registration) keeps its password on
|
||||
# approval — the guard must not break the normal flow.
|
||||
# The no-SMTP shape: registration marks such an account verified up front, so it was never in
|
||||
# scope for the SEC-10 guard even before E9.5 S1. Kept as the other half of the pair.
|
||||
from app.routes.admin import _activate_user_by_email
|
||||
|
||||
user = make_user(
|
||||
@@ -127,3 +184,374 @@ def test_activating_a_verified_account_keeps_its_password(db, make_user):
|
||||
_activate_user_by_email(db, "legit@example.com")
|
||||
assert user.is_active is True
|
||||
assert user.password_hash is not None
|
||||
|
||||
|
||||
def _register_and_fetch(db, email: str):
|
||||
"""Run the /register background worker, then read the users/invites rows back through a FRESH
|
||||
query. The worker owns its own session, so anything it failed to commit is simply not there."""
|
||||
from sqlalchemy import select as _select
|
||||
|
||||
from app.models import Invite, User
|
||||
from app.routes.auth import _register_account
|
||||
|
||||
_register_account(email, _PW)
|
||||
db.expire_all()
|
||||
user = db.execute(_select(User).where(User.email == email)).scalar_one_or_none()
|
||||
invite = db.execute(_select(Invite).where(Invite.email == email)).scalar_one_or_none()
|
||||
return user, invite
|
||||
|
||||
|
||||
def _no_smtp(monkeypatch):
|
||||
"""Put the worker on the no-SMTP path and capture the admin notice instead of sending it."""
|
||||
from app.routes import auth as auth_mod
|
||||
|
||||
notified: list[str] = []
|
||||
monkeypatch.setattr(auth_mod.email_mod, "email_enabled", lambda: False)
|
||||
monkeypatch.setattr(
|
||||
auth_mod.email_mod,
|
||||
"send_admin_new_request",
|
||||
lambda admins, lang, to: notified.append(to) or True,
|
||||
)
|
||||
return notified
|
||||
|
||||
|
||||
def test_registration_of_a_pre_approved_email_persists_the_account_without_smtp(
|
||||
db, make_user, monkeypatch
|
||||
):
|
||||
# THE regression. An admin whitelists an address, the person registers, and the account silently
|
||||
# never existed: upsert_pending_invite leaves an already-approved row alone and returns WITHOUT
|
||||
# committing, and the only other commit on this path (_issue_token) is gated on SMTP — so with no
|
||||
# working mail the flushed user was rolled back at session close. The sign-up answered "ok",
|
||||
# created nothing, and re-registering did the same again.
|
||||
from app.models import Invite
|
||||
|
||||
notified = _no_smtp(monkeypatch)
|
||||
make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="roland@example.com", status="approved", decided_by="admin@example.com"))
|
||||
db.commit()
|
||||
|
||||
user, invite = _register_and_fetch(db, "roland@example.com")
|
||||
|
||||
assert user is not None, "pre-approved registration was rolled back — the account is gone"
|
||||
assert user.password_hash is not None
|
||||
assert user.email_verified is True # no SMTP => ownership can't be proven by link
|
||||
assert (
|
||||
user.is_active is False
|
||||
) # approval is bound to the account by an admin, not by signing up
|
||||
# The admin's standing decision is left exactly as it was — is_allowed() still reads "approved",
|
||||
# so their Google sign-in keeps working and an anonymous sign-up cannot revoke access.
|
||||
assert invite.status == "approved"
|
||||
assert invite.decided_by == "admin@example.com"
|
||||
# The notify path still fires for an already-approved address — nothing lands in the queue, so
|
||||
# it is the only announcement there is. (Whether it is DELIVERED is separate: on a real no-SMTP
|
||||
# instance `_send` drops it, which is the open half of the problem, roadmap UX-25.)
|
||||
assert notified == ["roland@example.com"]
|
||||
|
||||
|
||||
def test_registration_with_an_already_pending_request_persists_the_account(
|
||||
db, make_user, monkeypatch
|
||||
):
|
||||
# The same rollback, reached the other way: someone uses Request access, then registers before
|
||||
# the admin has decided. upsert_pending_invite leaves the pending row alone and does not commit.
|
||||
from app.models import Invite
|
||||
|
||||
notified = _no_smtp(monkeypatch)
|
||||
make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="pia@example.com", status="pending"))
|
||||
db.commit()
|
||||
|
||||
user, invite = _register_and_fetch(db, "pia@example.com")
|
||||
|
||||
assert user is not None, "registration was rolled back — the account is gone"
|
||||
assert user.is_active is False
|
||||
assert invite.status == "pending"
|
||||
assert notified == ["pia@example.com"]
|
||||
|
||||
|
||||
def test_an_admin_can_activate_a_pre_approved_registrant(db, monkeypatch):
|
||||
# The recovery path for the account above, pinned so it cannot quietly break: re-entering the
|
||||
# address in the Access-requests whitelist box (POST /api/admin/invites) runs
|
||||
# _activate_user_by_email.
|
||||
from app.models import Invite
|
||||
from app.routes.admin import _activate_user_by_email
|
||||
|
||||
_no_smtp(monkeypatch)
|
||||
db.add(Invite(email="roland@example.com", status="approved", decided_by="admin@example.com"))
|
||||
db.commit()
|
||||
user, _ = _register_and_fetch(db, "roland@example.com")
|
||||
|
||||
_activate_user_by_email(db, "roland@example.com")
|
||||
|
||||
db.expire_all()
|
||||
assert user.is_active is True
|
||||
assert user.password_hash is not None # approval never touches the password (E9.5 S1)
|
||||
|
||||
|
||||
def test_registration_of_an_unknown_email_still_persists_account_and_pending_invite(
|
||||
db, make_user, monkeypatch
|
||||
):
|
||||
# Guard, not a regression test: this branch always worked (upsert INSERTs the invite and commits
|
||||
# the same transaction). It must keep working — and it is the case that DOES notify the admin.
|
||||
notified = _no_smtp(monkeypatch)
|
||||
make_user(email="admin@example.com", role="admin")
|
||||
|
||||
user, invite = _register_and_fetch(db, "stranger@example.com")
|
||||
|
||||
assert user is not None
|
||||
assert user.is_active is False
|
||||
assert invite is not None and invite.status == "pending"
|
||||
assert notified == ["stranger@example.com"]
|
||||
|
||||
|
||||
def test_registration_of_a_previously_denied_email_revives_the_request(db, make_user, monkeypatch):
|
||||
# Guard: a denied person may ask again — the invite flips back to pending (and commits) so the
|
||||
# admin sees a fresh request. The stale decision must not leak into the new account's state.
|
||||
from app.models import Invite
|
||||
|
||||
notified = _no_smtp(monkeypatch)
|
||||
make_user(email="admin@example.com", role="admin")
|
||||
db.add(Invite(email="denis@example.com", status="denied", decided_by="admin@example.com"))
|
||||
db.commit()
|
||||
|
||||
user, invite = _register_and_fetch(db, "denis@example.com")
|
||||
|
||||
assert user is not None
|
||||
assert user.is_active is False
|
||||
assert invite.status == "pending"
|
||||
assert invite.decided_at is None and invite.decided_by is None
|
||||
assert notified == ["denis@example.com"]
|
||||
|
||||
|
||||
def test_registration_of_a_pre_approved_email_persists_the_account_with_smtp(db, monkeypatch):
|
||||
# Guard, not a regression test: with SMTP on, _issue_token's own commit already persisted the
|
||||
# user, so this branch was never broken and must stay unbroken.
|
||||
from app.models import Invite
|
||||
from app.routes import auth as auth_mod
|
||||
|
||||
sent: list[str] = []
|
||||
monkeypatch.setattr(auth_mod.email_mod, "email_enabled", lambda: True)
|
||||
monkeypatch.setattr(
|
||||
auth_mod.email_mod, "send_verify_email", lambda to, lang, link, **kw: sent.append(to)
|
||||
)
|
||||
monkeypatch.setattr(auth_mod.email_mod, "send_admin_new_request", lambda admins, lang, to: True)
|
||||
db.add(Invite(email="rita@example.com", status="approved", decided_by="admin@example.com"))
|
||||
db.commit()
|
||||
|
||||
user, invite = _register_and_fetch(db, "rita@example.com")
|
||||
|
||||
assert user is not None
|
||||
assert user.email_verified is False # must confirm the address
|
||||
assert user.is_active is False
|
||||
assert invite.status == "approved"
|
||||
assert sent == ["rita@example.com"]
|
||||
|
||||
|
||||
def test_verifying_an_email_never_grants_access_on_its_own(client, db, make_user):
|
||||
# Verification attests the address; it does not decide access. The two gates stay separate.
|
||||
from app.models import Invite
|
||||
from app.routes.auth import _issue_token
|
||||
|
||||
user = make_user(email="rita@example.com", password=_PW, email_verified=False, is_active=False)
|
||||
db.add(Invite(email="rita@example.com", status="approved", decided_by="admin@example.com"))
|
||||
db.commit()
|
||||
raw = _issue_token(db, user, "verify", timedelta(hours=1))
|
||||
|
||||
assert client.post("/auth/verify", json={"token": raw}).status_code == 200
|
||||
|
||||
db.expire_all()
|
||||
assert user.email_verified is True
|
||||
assert user.is_active is False
|
||||
assert _login(client, "rita@example.com", _PW).status_code == 403
|
||||
|
||||
|
||||
def test_register_route_creates_the_account_end_to_end(client, db, monkeypatch):
|
||||
# The tests above call _register_account directly, so nothing covered the ROUTE: its
|
||||
# allow_registration gate, its rate limiter and the background.add_task wiring that makes the
|
||||
# whole flow reachable at all. TestClient runs BackgroundTasks after the response, so a single
|
||||
# POST exercises the real path end to end.
|
||||
from sqlalchemy import select as _select
|
||||
|
||||
from app.models import User
|
||||
|
||||
_no_smtp(monkeypatch)
|
||||
|
||||
r = client.post("/auth/register", json={"email": "newbie@example.com", "password": _PW})
|
||||
|
||||
assert r.status_code == 200 and r.json() == {"status": "ok"}
|
||||
db.expire_all()
|
||||
user = db.execute(_select(User).where(User.email == "newbie@example.com")).scalar_one_or_none()
|
||||
assert user is not None, "POST /auth/register answered ok but created nothing"
|
||||
assert user.password_hash is not None
|
||||
|
||||
|
||||
def test_register_route_is_refused_when_registration_is_closed(client, db, monkeypatch):
|
||||
# The gate is an explicit 403 with nothing created — not a uniform "ok" that silently drops it.
|
||||
from sqlalchemy import select as _select
|
||||
|
||||
from app import sysconfig
|
||||
from app.models import User
|
||||
|
||||
_no_smtp(monkeypatch)
|
||||
sysconfig.set_value(db, "allow_registration", False)
|
||||
|
||||
r = client.post("/auth/register", json={"email": "closed@example.com", "password": _PW})
|
||||
|
||||
assert r.status_code == 403
|
||||
db.expire_all()
|
||||
assert (
|
||||
db.execute(_select(User).where(User.email == "closed@example.com")).scalar_one_or_none()
|
||||
is None
|
||||
)
|
||||
|
||||
|
||||
# --- SEC-10 lockout: a revoked preset password must stay RECOVERABLE ---------------------------
|
||||
#
|
||||
# Regression pack for the prod incident of 2026-09-02. An account registered with a password, then
|
||||
# stripped of it by the SEC-10 guard, ended up with password_hash = NULL. That is also what a
|
||||
# Google-only account looks like, and reset eligibility keyed off `password_hash` alone — so the
|
||||
# account could neither sign in (no password) nor obtain a reset link, and every response on the way
|
||||
# was a deliberately uniform "ok"/401. `password_revoked_at` is the trace that tells the two apart;
|
||||
# these pin each step of the way back in.
|
||||
#
|
||||
# E9.5 S1 removed the admin-approval caller, so Google adoption is the only path that still produces
|
||||
# this state — plus every account stripped on prod before that change, which is why the whole recovery
|
||||
# route below must keep working. These tests manufacture the state through the surviving producer.
|
||||
|
||||
|
||||
def _revoke_as_google_adoption(db, user) -> None:
|
||||
"""Put an account into the SEC-10-revoked state the way the ONLY remaining producer does: the
|
||||
Google callback adopting an account whose email was never verified."""
|
||||
from app.routes.auth import revoke_preset_password_if_unverified
|
||||
|
||||
revoke_preset_password_if_unverified(user)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _capture_reset_links(monkeypatch) -> list[str]:
|
||||
"""Patch the reset mailer and collect the recipients it is asked to write to."""
|
||||
from app.routes import auth as auth_mod
|
||||
|
||||
sent: list[str] = []
|
||||
monkeypatch.setattr(
|
||||
auth_mod.email_mod, "send_password_reset", lambda to, lang, link: sent.append(to)
|
||||
)
|
||||
return sent
|
||||
|
||||
|
||||
def test_the_google_adoption_revoke_stamps_password_revoked_at(db, make_user):
|
||||
# The guard still fires where it earns its keep: the real owner has just proven ownership through
|
||||
# Google, so a password set on the address while it was unverified must not survive.
|
||||
victim = make_user(
|
||||
email="stamped@example.com",
|
||||
password="attacker-chosen",
|
||||
email_verified=False,
|
||||
is_active=False,
|
||||
)
|
||||
assert victim.password_revoked_at is None
|
||||
epoch0 = victim.session_epoch or 0
|
||||
|
||||
_revoke_as_google_adoption(db, victim)
|
||||
|
||||
assert victim.password_hash is None
|
||||
assert victim.password_revoked_at is not None # …so the reset flow can tell it HAD a password
|
||||
assert (victim.session_epoch or 0) == epoch0 + 1 # any session that password minted is killed
|
||||
|
||||
|
||||
def test_an_account_whose_password_was_revoked_can_still_get_a_reset_link(
|
||||
db, make_user, monkeypatch
|
||||
):
|
||||
# THE lockout bug: without the stamp this sent nothing, and the account was unrecoverable.
|
||||
from app.routes.auth import _send_reset_if_eligible
|
||||
|
||||
victim = make_user(
|
||||
email="lockedout@example.com",
|
||||
password="chosen at registration",
|
||||
email_verified=False,
|
||||
is_active=False,
|
||||
)
|
||||
_revoke_as_google_adoption(db, victim)
|
||||
sent = _capture_reset_links(monkeypatch)
|
||||
|
||||
_send_reset_if_eligible("lockedout@example.com")
|
||||
|
||||
assert sent == ["lockedout@example.com"]
|
||||
|
||||
|
||||
def test_a_google_only_account_is_still_not_reset_eligible(db, make_user, monkeypatch):
|
||||
# The other side of the rule: an account that NEVER had a password gets no reset link — its owner
|
||||
# adds one from Settings while signed in (/auth/set-password), which proves identity by session.
|
||||
from app.routes.auth import _send_reset_if_eligible
|
||||
|
||||
make_user(email="googler@example.com", password=None, google_sub="sub-123")
|
||||
sent = _capture_reset_links(monkeypatch)
|
||||
|
||||
_send_reset_if_eligible("googler@example.com")
|
||||
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_completing_a_reset_clears_the_revocation_stamp(client, db, make_user):
|
||||
from app.routes.auth import _issue_token
|
||||
|
||||
user = make_user(
|
||||
email="recovered@example.com",
|
||||
password="chosen at registration",
|
||||
email_verified=False,
|
||||
is_active=True, # already approved; reset-confirm verifies the email but never activates
|
||||
)
|
||||
_revoke_as_google_adoption(db, user)
|
||||
raw = _issue_token(db, user, "reset", timedelta(hours=1))
|
||||
db.commit()
|
||||
|
||||
ok = client.post(
|
||||
"/auth/password-reset/confirm", json={"token": raw, "password": "a brand new one"}
|
||||
)
|
||||
|
||||
assert ok.status_code == 200
|
||||
db.expire_all()
|
||||
assert user.password_hash is not None
|
||||
assert user.password_revoked_at is None # spent — not a permanently reset-eligible account
|
||||
assert _login(client, "recovered@example.com", "a brand new one").status_code == 200
|
||||
|
||||
|
||||
def test_verify_tells_a_revoked_account_that_it_needs_a_new_password(client, db, make_user):
|
||||
# The banner's input. Saying so leaks nothing: the caller just proved mailbox control by holding a
|
||||
# valid single-use verify token. Without it the user meets only the uniform "Invalid email or
|
||||
# password" and has no way to learn why the password they chose stopped working.
|
||||
from app.routes.auth import _issue_token
|
||||
|
||||
user = make_user(
|
||||
email="needspw@example.com",
|
||||
password="chosen at registration",
|
||||
email_verified=False,
|
||||
is_active=False,
|
||||
)
|
||||
_revoke_as_google_adoption(db, user)
|
||||
raw = _issue_token(db, user, "verify", timedelta(hours=1))
|
||||
db.commit()
|
||||
|
||||
r = client.post("/auth/verify", json={"token": raw})
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {
|
||||
"ok": True,
|
||||
"password_reset_required": True,
|
||||
"email": "needspw@example.com",
|
||||
}
|
||||
|
||||
|
||||
def test_verify_reports_no_reset_needed_for_an_intact_account(client, db, make_user):
|
||||
from app.routes.auth import _issue_token
|
||||
|
||||
user = make_user(email="intact@example.com", email_verified=False, is_active=False)
|
||||
raw = _issue_token(db, user, "verify", timedelta(hours=1))
|
||||
db.commit()
|
||||
|
||||
r = client.post("/auth/verify", json={"token": raw})
|
||||
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {
|
||||
"ok": True,
|
||||
"password_reset_required": False,
|
||||
"email": "intact@example.com",
|
||||
}
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
"""The shared demo account's refusal says WHY, in the reader's language (deep-review round 6).
|
||||
|
||||
`require_human` answered `403 "Not available in the demo account."` — a plain string. `req` leaves
|
||||
403 to the caller, and round 5 taught `notifyActionError` to discard a plain-string detail in favour
|
||||
of the caller's already-translated fallback, so from that commit on the demo user got a generic
|
||||
"couldn't do that" and no hint that the demo account was the reason. Not every affordance behind
|
||||
this gate is UI-gated on `is_demo` either — the comfort shelf is not, so the pin button is live.
|
||||
|
||||
A structured `{code: "demo"}` is what `localizeDetail` can turn into `errors.demo` in both locales.
|
||||
Pinned at all FOUR sites: the shared dependency, the two auth routes that hand-roll the same check,
|
||||
and the account-delete route in `routes/me.py` — which round 6 missed while its docstring claimed
|
||||
all the sites were covered, so the next reader had no reason to look (round 7). An asymmetric fix is
|
||||
the smell this repo bans, and a docstring asserting symmetry it does not have is worse than none.
|
||||
|
||||
The list is derived from the source, not typed out: `test_every_is_demo_refusal_is_on_the_list`
|
||||
below fails when a fifth `is_demo` 403 appears anywhere in `app/`, so this file cannot fall behind
|
||||
the code the way round 6's did. Needs the DB.
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from app.models import User
|
||||
|
||||
|
||||
def test_every_demo_refusal_carries_the_same_structured_reason(db, make_user, as_user):
|
||||
demo = make_user(email="demo@example.com", is_demo=True)
|
||||
client = as_user(demo)
|
||||
for method, url, body in [
|
||||
# `require_human` itself, through an affordance the UI does NOT gate on is_demo.
|
||||
("post", "/api/discover/pins", {"kind": "video", "ref": "v1"}),
|
||||
# The three routes that repeat the check inline rather than depending on the gate.
|
||||
("post", "/auth/logout-others", None),
|
||||
("post", "/auth/set-password", {"password": "a long enough password"}),
|
||||
("delete", "/api/me/account", None),
|
||||
]:
|
||||
r = getattr(client, method)(url, **({"json": body} if body is not None else {}))
|
||||
assert r.status_code == 403, (url, r.text)
|
||||
detail = r.json()["detail"]
|
||||
# Every one of them is a dict whose `code` the frontend can localize; the delete route adds
|
||||
# a `reason` so its refusal can be worded specifically.
|
||||
assert isinstance(detail, dict), (url, detail)
|
||||
assert detail["code"] == "demo", (url, detail)
|
||||
|
||||
|
||||
def test_every_is_demo_refusal_is_on_the_list():
|
||||
"""A fifth site must not be able to appear unnoticed.
|
||||
|
||||
Round 6 converted three of four `is_demo` 403s and wrote a docstring saying it had converted
|
||||
them all; the fourth kept answering a bare English string for another round because nothing
|
||||
counted them. So count them here: every `is_demo`-guarded 403 in `app/` has to raise a
|
||||
structured detail, and the number of them has to match what this file exercises."""
|
||||
app_dir = Path(__file__).resolve().parents[1] / "app"
|
||||
sites = []
|
||||
for path in sorted(app_dir.rglob("*.py")):
|
||||
lines = path.read_text(encoding="utf-8").splitlines()
|
||||
for i, line in enumerate(lines):
|
||||
# `if user.is_demo:` / `if u.is_demo:` — the hand-rolled gate, not the ORM filters
|
||||
# (`User.is_demo.is_(False)`) and not the column definition.
|
||||
if not re.search(r"^\s*if\s+\w+\.is_demo\s*:", line):
|
||||
continue
|
||||
block = "\n".join(lines[i : i + 6])
|
||||
if "403" not in block:
|
||||
continue
|
||||
sites.append(f"{path.relative_to(app_dir)}:{i + 1}")
|
||||
assert 'detail={"code": "demo"' in block, (
|
||||
f"{path.relative_to(app_dir)}:{i + 1} refuses the demo account with an "
|
||||
f"unstructured detail — the frontend cannot localize it:\n{block}"
|
||||
)
|
||||
# auth.py's shared gate + routes/auth.py's two + routes/me.py's delete. A new one is welcome,
|
||||
# but it has to be added to the loop above in the same commit.
|
||||
assert len(sites) == 4, sites
|
||||
|
||||
|
||||
def test_a_real_account_is_not_refused_by_the_gate(db, make_user, as_user):
|
||||
"""The other half: the structured detail must not come from having broken the gate open the
|
||||
other way. A non-demo account gets past `require_human` and fails (or succeeds) on its own
|
||||
merits — anything but a 403 demo refusal."""
|
||||
client = as_user(make_user(email="real@example.com"))
|
||||
r = client.post("/auth/logout-others")
|
||||
assert r.status_code == 200, r.text
|
||||
assert not isinstance(r.json().get("detail"), dict)
|
||||
|
||||
|
||||
def test_the_gate_reads_the_flag_and_not_the_email(db):
|
||||
"""`is_demo` is a column, not a convention about the address — a real account that happens to
|
||||
be called demo@ must not be refused."""
|
||||
from app.auth import require_human
|
||||
|
||||
assert require_human(User(email="demo@example.com", is_demo=False)) is not None
|
||||
@@ -0,0 +1,942 @@
|
||||
"""E24 S1 — the Home surface's read-model.
|
||||
|
||||
Covers the four things that can silently rot here: the time-budget bands (pure), the cross-source
|
||||
resume rail's merge order, the hero's tier ladder, and the comfort shelf (its schema-level one-of
|
||||
invariant, its cascade, and the resolved `next` that makes one click enough).
|
||||
|
||||
DB lane (`db` fixture) except for the budget block — skipped without Postgres.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import delete, func, select, text
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from app.discover import hero, pins, queries, rails
|
||||
from app.discover.budget import BANDS, fits, remaining_seconds
|
||||
from app.plex import rollup
|
||||
from app.plex.cards import leaf_card
|
||||
from app.models import (
|
||||
BlockedChannel,
|
||||
Channel,
|
||||
ComfortPin,
|
||||
PlexItem,
|
||||
PlexLibrary,
|
||||
PlexShow,
|
||||
PlexState,
|
||||
Playlist,
|
||||
PlaylistItem,
|
||||
Subscription,
|
||||
User,
|
||||
Video,
|
||||
VideoState,
|
||||
)
|
||||
|
||||
NOW = datetime(2026, 6, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
# ── budget (pure) ─────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_bands_cover_every_runtime_and_overlap_deliberately():
|
||||
"""Every runtime has a home in `any`, and the bands overlap so "30 perc" is not answered with a
|
||||
two-minute Short just because it fits under the cap."""
|
||||
assert fits(3, "any") and fits(10_000, "any")
|
||||
assert fits(2 * 60, "short") and not fits(2 * 60, "medium") # a Short is not a 30-minute answer
|
||||
assert fits(30 * 60, "medium") and not fits(30 * 60, "short")
|
||||
assert fits(110 * 60, "long") and not fits(110 * 60, "medium")
|
||||
|
||||
|
||||
def test_unknown_duration_only_fits_the_unbounded_band():
|
||||
"""We cannot promise an unknown-length item finishes inside ten minutes, so it is not offered as
|
||||
an answer to a bounded question — the one control on the page has to mean what it says."""
|
||||
for budget in BANDS:
|
||||
assert fits(None, budget) is (budget == "any")
|
||||
|
||||
|
||||
def test_budget_is_matched_against_what_is_LEFT():
|
||||
"""A 90-minute film with 8 minutes to go is a ten-minute answer."""
|
||||
assert remaining_seconds(90 * 60, 82 * 60) == 8 * 60
|
||||
assert fits(remaining_seconds(90 * 60, 82 * 60), "short")
|
||||
assert not fits(remaining_seconds(90 * 60, 0), "short")
|
||||
assert remaining_seconds(None, 30) is None
|
||||
assert remaining_seconds(600, 900) == 0 # a position past the end never goes negative
|
||||
|
||||
|
||||
# ── fixtures ──────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _seed_yt(db) -> tuple[User, Channel]:
|
||||
"""A subscribed channel with three videos of distinct lengths, so a budget can pick between them."""
|
||||
user = User(email="discover-e24@example.com")
|
||||
ch = Channel(id="UCe24", title="Comfort Channel", handle="@comfort")
|
||||
db.add_all([user, ch])
|
||||
db.flush()
|
||||
db.add(Subscription(user_id=user.id, channel_id=ch.id))
|
||||
db.add_all(
|
||||
[
|
||||
Video(
|
||||
id="vshort",
|
||||
channel_id=ch.id,
|
||||
title="Ten minute video",
|
||||
duration_seconds=9 * 60,
|
||||
published_at=NOW - timedelta(days=1),
|
||||
),
|
||||
Video(
|
||||
id="vmed",
|
||||
channel_id=ch.id,
|
||||
title="Half hour video",
|
||||
duration_seconds=30 * 60,
|
||||
published_at=NOW - timedelta(days=2),
|
||||
),
|
||||
Video(
|
||||
id="vlong",
|
||||
channel_id=ch.id,
|
||||
title="Feature length video",
|
||||
duration_seconds=95 * 60,
|
||||
published_at=NOW - timedelta(days=3),
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
return user, ch
|
||||
|
||||
|
||||
def _seed_plex(db, user: User) -> tuple[PlexItem, PlexShow, PlexItem]:
|
||||
lib_m = PlexLibrary(plex_key="90", title="Movies", kind="movie", enabled=True)
|
||||
lib_s = PlexLibrary(plex_key="91", title="Shows", kind="show", enabled=True)
|
||||
db.add_all([lib_m, lib_s])
|
||||
db.flush()
|
||||
movie = PlexItem(
|
||||
rating_key="pm1",
|
||||
library_id=lib_m.id,
|
||||
kind="movie",
|
||||
title="A Movie",
|
||||
duration_s=100 * 60,
|
||||
playable="direct",
|
||||
)
|
||||
show = PlexShow(rating_key="ps1", library_id=lib_s.id, title="A Show", child_count=1)
|
||||
db.add_all([movie, show])
|
||||
db.flush()
|
||||
ep1 = PlexItem(
|
||||
rating_key="ps1e1",
|
||||
library_id=lib_s.id,
|
||||
kind="episode",
|
||||
title="Ep 1",
|
||||
show_id=show.id,
|
||||
season_number=1,
|
||||
episode_number=1,
|
||||
duration_s=45 * 60,
|
||||
)
|
||||
ep2 = PlexItem(
|
||||
rating_key="ps1e2",
|
||||
library_id=lib_s.id,
|
||||
kind="episode",
|
||||
title="Ep 2",
|
||||
show_id=show.id,
|
||||
season_number=1,
|
||||
episode_number=2,
|
||||
duration_s=45 * 60,
|
||||
)
|
||||
db.add_all([ep1, ep2])
|
||||
db.commit()
|
||||
return movie, show, ep1
|
||||
|
||||
|
||||
def _enable_plex(db):
|
||||
from app import sysconfig
|
||||
|
||||
sysconfig.set_value(db, "plex_enabled", True)
|
||||
db.commit()
|
||||
|
||||
|
||||
# ── continue watching ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_continue_watching_merges_both_sources_by_recency(db):
|
||||
"""One rail, not two lists side by side — and ordered by when the user last touched each thing,
|
||||
across sources. Two side-by-side rails would just move the decision up a level."""
|
||||
user, _ = _seed_yt(db)
|
||||
movie, _, _ = _seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
db.add_all(
|
||||
[
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW - timedelta(hours=5),
|
||||
),
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=movie.id,
|
||||
status="new",
|
||||
position_seconds=600,
|
||||
progress_updated_at=NOW - timedelta(hours=1),
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
|
||||
got = rails.continue_watching(db, user)
|
||||
assert [(c["source"], c["id"]) for c in got] == [("plex", "pm1"), ("youtube", "vmed")]
|
||||
assert got[0]["remaining_seconds"] == 100 * 60 - 600
|
||||
|
||||
|
||||
def test_continue_watching_is_youtube_only_when_plex_is_off(db):
|
||||
"""Plex is a deployment toggle; with it off the surface says nothing about Plex at all."""
|
||||
user, _ = _seed_yt(db)
|
||||
movie, _, _ = _seed_plex(db, user)
|
||||
db.add_all(
|
||||
[
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=movie.id,
|
||||
status="new",
|
||||
position_seconds=600,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
assert {c["source"] for c in rails.continue_watching(db, user)} == {"youtube"}
|
||||
|
||||
|
||||
def test_the_resume_rail_obeys_the_channel_blocklist(db):
|
||||
"""Blocking a channel does not touch `video_states`, so a half-watched video from it survives
|
||||
with its resume position — and this rail builds its own SELECT instead of going through
|
||||
`filtered_query`. "Never show me this channel" has to outrank "you started it", or the landing
|
||||
page answers "what should I watch" with the thing the user has just banned."""
|
||||
user, ch = _seed_yt(db)
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
assert [c["id"] for c in rails.continue_watching(db, user)] == ["vmed"]
|
||||
|
||||
db.add(BlockedChannel(user_id=user.id, channel_id=ch.id))
|
||||
db.commit()
|
||||
assert rails.continue_watching(db, user) == []
|
||||
# And so the hero, whose resume tier draws from that rail, has nothing to offer either.
|
||||
assert hero.pick(db, user, "any", 0) is None
|
||||
|
||||
|
||||
def test_a_null_timestamp_sorts_last(db):
|
||||
"""A missing progress timestamp sorts last rather than 500ing the landing page.
|
||||
|
||||
Narrower than it was written to be, and the epic-end review corrected it: the previous key,
|
||||
`(ts is not None, ts)`, does NOT raise on two nulls — tuple comparison finds the first UNEQUAL
|
||||
element via `==`, so `None < None` is never reached, and this test passes under that key too.
|
||||
What it does discriminate is the naive `key=lambda r: r[0]`, which really does raise. See
|
||||
`plex.rollup.NO_TS` for the corrected rationale."""
|
||||
user, _ = _seed_yt(db)
|
||||
db.add_all(
|
||||
[
|
||||
VideoState(
|
||||
user_id=user.id, video_id="vmed", status="new", position_seconds=1
|
||||
), # no progress_updated_at
|
||||
VideoState(
|
||||
user_id=user.id, video_id="vlong", status="new", position_seconds=1
|
||||
), # nor here
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vshort",
|
||||
status="new",
|
||||
position_seconds=1,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
got = rails.continue_watching(db, user)
|
||||
assert got[0]["id"] == "vshort"
|
||||
assert len(got) == 3
|
||||
|
||||
|
||||
# ── hero ──────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_hero_prefers_resume_and_labels_the_tier(db):
|
||||
user, _ = _seed_yt(db)
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
got = hero.pick(db, user, "any", seed=1)
|
||||
assert got["tier"] == "resume"
|
||||
assert got["card"]["id"] == "vmed"
|
||||
|
||||
|
||||
def test_hero_skips_a_resume_that_does_not_fit_the_budget(db):
|
||||
""" "I have ten minutes" must not be answered with the 80 minutes left on a film, even though
|
||||
that film is the most recent thing the user touched."""
|
||||
user, _ = _seed_yt(db)
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vlong",
|
||||
status="new",
|
||||
position_seconds=60,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
got = hero.pick(db, user, "short", seed=1)
|
||||
assert got["tier"] == "surprise"
|
||||
assert got["card"]["duration_seconds"] <= BANDS["short"][1]
|
||||
|
||||
|
||||
def test_hero_falls_back_to_the_shelf_before_guessing(db):
|
||||
"""The user's own declared fallback beats anything we could infer."""
|
||||
user, ch = _seed_yt(db)
|
||||
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
|
||||
db.commit()
|
||||
got = hero.pick(db, user, "medium", seed=1)
|
||||
assert got["tier"] == "comfort"
|
||||
assert got["card"]["id"] == "vmed"
|
||||
|
||||
|
||||
def test_hero_resolves_a_show_to_the_episode_that_would_play(db):
|
||||
"""A show has no Play button. The hero of a partially-watched show is its next episode."""
|
||||
user, _ = _seed_yt(db)
|
||||
_, show, ep1 = _seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
db.add(
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=ep1.id,
|
||||
status="watched",
|
||||
watched_at=NOW,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
got = hero.pick(db, user, "any", seed=1)
|
||||
assert got["tier"] == "resume"
|
||||
assert got["card"]["kind"] == "episode"
|
||||
assert got["card"]["id"] == "ps1e2" # ep1 is finished → ep2 is what plays
|
||||
|
||||
|
||||
def test_hero_is_none_when_nothing_fits(db):
|
||||
"""An empty answer is an honest answer: the page renders its own empty state rather than being
|
||||
handed a card that does not answer what was asked. Here the whole library is under 40 minutes,
|
||||
so "egy este" has nothing to offer."""
|
||||
user, ch = _seed_yt(db)
|
||||
db.execute(text("DELETE FROM videos WHERE id = 'vlong'"))
|
||||
db.commit()
|
||||
assert hero.pick(db, user, "long", seed=1) is None
|
||||
|
||||
|
||||
def test_surprise_is_reproducible_for_a_seed(db):
|
||||
"""The reroll button depends on this: one seed drives both sources, so the same seed must give
|
||||
the same answer and a different seed must be free to give another."""
|
||||
user, _ = _seed_yt(db)
|
||||
a = rails.surprise(db, user, "any", seed=7, limit=3)
|
||||
b = rails.surprise(db, user, "any", seed=7, limit=3)
|
||||
assert [c["id"] for c in a] == [c["id"] for c in b]
|
||||
seeds = {tuple(c["id"] for c in rails.surprise(db, user, "any", s, 3)) for s in range(12)}
|
||||
assert len(seeds) > 1 # the seed actually reaches the ordering
|
||||
|
||||
|
||||
def test_surprise_never_offers_something_already_started(db):
|
||||
"""The resume tier may have skipped an item because its REMAINING time did not fit; handing the
|
||||
same item back as "Meglepetés" — where the full runtime does fit — would contradict the question
|
||||
the user just answered."""
|
||||
user, _ = _seed_yt(db)
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vlong",
|
||||
status="new",
|
||||
position_seconds=90 * 60,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
assert "vlong" not in {c["id"] for c in rails.surprise(db, user, "long", seed=3, limit=5)}
|
||||
|
||||
|
||||
def test_the_seed_decides_which_source_leads(db):
|
||||
"""With a pool of one, a fixed source order would mean the Plex library could never answer "egy
|
||||
este" whenever a YouTube video also fit — the cross-source promise would quietly be a YouTube
|
||||
promise."""
|
||||
user, _ = _seed_yt(db)
|
||||
_seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
leaders = {rails.surprise(db, user, "any", s, limit=1)[0]["source"] for s in range(6)}
|
||||
assert leaders == {"youtube", "plex"}
|
||||
|
||||
|
||||
def test_the_reroll_must_be_able_to_move_past_the_resume_tier(db):
|
||||
"""A new seed alone only re-draws the SURPRISE tier — the resume tier deterministically returns
|
||||
the most recently progressed thing that fits, so without an exclusion the reroll handed the same
|
||||
card straight back and the button looked dead (found in the browser, 2026-09-06)."""
|
||||
user, _ = _seed_yt(db)
|
||||
db.add_all(
|
||||
[
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vshort",
|
||||
status="new",
|
||||
position_seconds=60,
|
||||
progress_updated_at=NOW - timedelta(hours=1),
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
|
||||
first = hero.pick(db, user, "any", seed=1)
|
||||
assert first["card"]["id"] == "vmed"
|
||||
# Same seed, same everything — only the exclusion differs.
|
||||
second = hero.pick(db, user, "any", seed=1, exclude={"youtube:vmed"})
|
||||
assert second["card"]["id"] == "vshort"
|
||||
|
||||
|
||||
def test_the_reroll_rotates_which_tier_answers(db):
|
||||
"""Measured 2026-09-06: exclusion alone left the button walking down the resume rail (20 rows)
|
||||
and cycling nine cards, every one of them `resume` — the shelf and the surprise tier were
|
||||
unreachable by pressing it. `start` is the press count, so consecutive presses offer a different
|
||||
KIND of answer."""
|
||||
user, _ = _seed_yt(db)
|
||||
# One of each: something half-finished, something pinned, and a third video for the draw.
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.add(ComfortPin(user_id=user.id, video_id="vlong", position=0))
|
||||
db.commit()
|
||||
tiers = [hero.pick(db, user, "any", seed=1, start=n)["tier"] for n in range(4)]
|
||||
assert tiers == ["resume", "comfort", "surprise", "resume"]
|
||||
|
||||
|
||||
def test_the_rotation_still_falls_through_an_empty_tier(db):
|
||||
"""A press must not come back empty just because the tier it happened to lead with has nothing.
|
||||
Here there is no shelf at all, so the comfort-led press has to answer from somewhere else."""
|
||||
user, _ = _seed_yt(db)
|
||||
assert db.execute(select(func.count()).select_from(ComfortPin)).scalar_one() == 0
|
||||
got = hero.pick(db, user, "any", seed=1, start=1)
|
||||
assert got is not None and got["tier"] != "comfort"
|
||||
|
||||
|
||||
def test_excluding_a_show_also_excludes_the_episode_it_resolves_to(db):
|
||||
"""The user rerolled past the SHOW they were shown; handing back its next episode under a
|
||||
different title would be the same suggestion wearing a hat."""
|
||||
user, _ = _seed_yt(db)
|
||||
_, show, ep1 = _seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
db.add(
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=ep1.id,
|
||||
status="new",
|
||||
position_seconds=120,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
shown = hero.pick(db, user, "any", seed=1)
|
||||
assert shown["card"]["id"] == "ps1e1" # the show resolved to its in-progress episode
|
||||
# Excluding what the USER saw — the show — must move the tier on, not return the episode again.
|
||||
after = hero.pick(db, user, "any", seed=1, exclude={"plex:ps1"})
|
||||
assert after is None or after["card"]["id"] != "ps1e1"
|
||||
|
||||
|
||||
# ── comfort shelf ─────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_a_pin_must_reference_exactly_one_thing(db):
|
||||
"""The union invariant lives in the schema, not in a comment at every insert site."""
|
||||
user, ch = _seed_yt(db)
|
||||
for kwargs in ({}, {"video_id": "vmed", "channel_id": ch.id}):
|
||||
db.add(ComfortPin(user_id=user.id, position=0, **kwargs))
|
||||
with pytest.raises(IntegrityError):
|
||||
db.flush()
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_the_same_thing_cannot_be_pinned_twice(db):
|
||||
"""The per-kind unique constraints are what make the route's concurrent-tab path safe (it catches
|
||||
`IntegrityError` precisely because of them) — and the migration drift check filters constraint
|
||||
diffs out as noise, so nothing else would notice them going missing. The route's own dedupe
|
||||
returns early without ever reaching the DB, so only this test exercises them."""
|
||||
user, _ = _seed_yt(db)
|
||||
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
|
||||
db.commit()
|
||||
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=1))
|
||||
with pytest.raises(IntegrityError):
|
||||
db.flush()
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_deleting_the_pinned_row_removes_the_pin(db):
|
||||
"""A film dropped from the library must not leave a dead tile on the one surface whose whole job
|
||||
is to be reliable — which is why the shelf holds foreign keys, not strings."""
|
||||
user, _ = _seed_yt(db)
|
||||
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
|
||||
db.commit()
|
||||
db.execute(text("DELETE FROM videos WHERE id = 'vmed'"))
|
||||
db.commit()
|
||||
assert db.execute(select(func.count()).select_from(ComfortPin)).scalar() == 0
|
||||
|
||||
|
||||
def test_shelf_resolves_a_channel_to_its_newest_unwatched_video(db):
|
||||
""" "Put that channel on" has to be one click, so the tile carries what would play."""
|
||||
user, ch = _seed_yt(db)
|
||||
db.add(ComfortPin(user_id=user.id, channel_id=ch.id, position=0))
|
||||
db.commit()
|
||||
(card,) = pins.shelf(db, user)
|
||||
assert card["kind"] == "channel"
|
||||
assert card["unwatched_count"] == 3
|
||||
assert card["next"]["id"] == "vshort" # newest
|
||||
|
||||
|
||||
def test_shelf_resolves_a_playlist_to_its_first_unwatched_item(db):
|
||||
user, _ = _seed_yt(db)
|
||||
pl = Playlist(user_id=user.id, name="Elalvós")
|
||||
db.add(pl)
|
||||
db.flush()
|
||||
db.add_all(
|
||||
[
|
||||
PlaylistItem(playlist_id=pl.id, video_id="vshort", position=0),
|
||||
PlaylistItem(playlist_id=pl.id, video_id="vmed", position=1),
|
||||
]
|
||||
)
|
||||
db.add(VideoState(user_id=user.id, video_id="vshort", status="watched", watched_at=NOW))
|
||||
db.add(ComfortPin(user_id=user.id, playlist_id=pl.id, position=0))
|
||||
db.commit()
|
||||
(card,) = pins.shelf(db, user)
|
||||
assert card["kind"] == "playlist"
|
||||
assert card["item_count"] == 2
|
||||
assert card["next"]["id"] == "vmed"
|
||||
|
||||
|
||||
def test_a_pinned_video_survives_leaving_the_feed_scope(db):
|
||||
"""The shelf is the fallback surface. An entry silently vanishing because the user unsubscribed
|
||||
from the channel is exactly the failure it must not have."""
|
||||
user, ch = _seed_yt(db)
|
||||
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
|
||||
db.commit()
|
||||
db.execute(delete(Subscription))
|
||||
db.commit()
|
||||
(card,) = pins.shelf(db, user)
|
||||
assert card["id"] == "vmed"
|
||||
assert card["next"]["id"] == "vmed"
|
||||
|
||||
|
||||
def test_the_plex_surprise_obeys_the_time_budget(db):
|
||||
"""The one control on the page has to mean what it says: a 100-minute film is not an answer to
|
||||
"10 perc". The YouTube half of this rule has had its own test since S1; the Plex half's duration
|
||||
bounds could be deleted with the whole suite still green, because every test that reached
|
||||
`plex_surprise` passed budget="any"."""
|
||||
user, _ = _seed_yt(db)
|
||||
_seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
drawn = lambda budget: { # noqa: E731
|
||||
(c["source"], c["id"]) for s in range(6) for c in rails.surprise(db, user, budget, s, 5)
|
||||
}
|
||||
assert ("plex", "pm1") not in drawn("short")
|
||||
# …and it IS offered where it fits, so this cannot pass by the rail being empty.
|
||||
assert ("plex", "pm1") in drawn("long")
|
||||
|
||||
|
||||
def test_the_plex_surprise_never_offers_something_already_started(db):
|
||||
"""A half-watched film is not a surprise, and the resume tier has already had its chance at it.
|
||||
The mirror of `test_surprise_never_offers_something_already_started` for the other source — the
|
||||
asymmetric half of one principle, which the house rules say to fix everywhere."""
|
||||
user, _ = _seed_yt(db)
|
||||
movie, _, _ = _seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
db.add(
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=movie.id,
|
||||
status="new",
|
||||
position_seconds=600,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
drawn = {
|
||||
(c["source"], c["id"]) for s in range(6) for c in rails.surprise(db, user, "long", s, 5)
|
||||
}
|
||||
assert ("plex", "pm1") not in drawn
|
||||
|
||||
|
||||
def test_a_new_pin_lands_at_the_end_even_after_a_hole(db):
|
||||
"""Positions are never renumbered on removal — `remove_pin` deletes the row, and every FK cascade
|
||||
silently removes a pin when its target disappears — so the append position has to come from
|
||||
`max(position)`, not from the row count. With a count, the first hole drops the next pin into the
|
||||
MIDDLE of the shelf and the second makes two pins claim one slot."""
|
||||
user, ch = _seed_yt(db)
|
||||
db.add_all(
|
||||
[
|
||||
ComfortPin(user_id=user.id, video_id="vshort", position=0),
|
||||
ComfortPin(user_id=user.id, video_id="vmed", position=1),
|
||||
ComfortPin(user_id=user.id, video_id="vlong", position=2),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
db.execute(delete(ComfortPin).where(ComfortPin.video_id.in_(("vshort", "vmed"))))
|
||||
db.commit()
|
||||
|
||||
db.add(ComfortPin(user_id=user.id, channel_id=ch.id, position=pins.next_position(db, user)))
|
||||
db.commit()
|
||||
assert [c["kind"] for c in pins.shelf(db, user)] == ["video", "channel"]
|
||||
positions = (
|
||||
db.execute(select(ComfortPin.position).order_by(ComfortPin.position)).scalars().all()
|
||||
)
|
||||
assert len(set(positions)) == len(positions) # and no two tiles claim the same slot
|
||||
|
||||
|
||||
def test_a_self_playing_pin_does_not_reference_itself(db):
|
||||
"""`next` must be a distinct dict even when the pin plays itself. Aliasing the card into its own
|
||||
`next` is a cycle, and the serializer turns the whole shelf into a 500 over it."""
|
||||
user, _ = _seed_yt(db)
|
||||
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
|
||||
db.commit()
|
||||
(card,) = pins.shelf(db, user)
|
||||
assert card["next"] is not card
|
||||
assert "next" not in card["next"]
|
||||
|
||||
|
||||
def test_the_resume_rail_says_what_would_actually_play(db):
|
||||
"""A show tile that only opens the show page moves the decision instead of removing it — the
|
||||
click lands on six seasons to choose between, which is the paralysis this whole surface exists
|
||||
for. The shelf and the hero already resolved; this rail did not (reported 2026-09-06). The tile
|
||||
stays the show (its artwork is what the user recognises); `next` is what starts."""
|
||||
user, _ = _seed_yt(db)
|
||||
_, show, ep1 = _seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
db.add(
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=ep1.id,
|
||||
status="watched",
|
||||
watched_at=NOW,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
rail = rails.continue_watching(db, user)
|
||||
tile = next(c for c in rail if c["kind"] == "show")
|
||||
assert tile["id"] == "ps1" # still the show
|
||||
assert tile["next"]["kind"] == "episode"
|
||||
assert tile["next"]["id"] == "ps1e2" # ep1 is finished → ep2 is what plays
|
||||
|
||||
|
||||
def test_only_a_show_row_pays_for_a_resolution(db):
|
||||
"""`next` is the answer to "a show has no Play button". Attaching it to a movie or a video would
|
||||
both cost a query for nothing and re-open the self-reference cycle the shelf already hit.
|
||||
|
||||
The states are what make this test exist: without them every rail query inner-joins its way to
|
||||
zero rows, the loop body never runs, and the assertion is vacuous — which is exactly how the
|
||||
epic-end review found it."""
|
||||
user, _ = _seed_yt(db)
|
||||
movie, _, ep1 = _seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
db.add_all(
|
||||
[
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=movie.id,
|
||||
status="new",
|
||||
position_seconds=600,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=ep1.id,
|
||||
status="new",
|
||||
position_seconds=120,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
|
||||
rail = rails.continue_watching(db, user)
|
||||
# A video, a movie and a show — one row of each kind that can appear here, so the assertion below
|
||||
# actually exercises both sides of the equality.
|
||||
assert {c["kind"] for c in rail} == {"video", "movie", "show"}
|
||||
for card in rail:
|
||||
assert ("next" in card) == (card["kind"] == "show"), card["id"]
|
||||
|
||||
|
||||
def test_next_episode_takes_an_episode_with_no_state_row_at_all(db):
|
||||
"""The NULL trap in the SQL rewrite: `status NOT IN ('watched','hidden')` is UNKNOWN — not true —
|
||||
for an episode the user has never touched, so a bare filter drops exactly the common case and the
|
||||
rail would offer episode 1 of a show already half-watched."""
|
||||
user, _ = _seed_yt(db)
|
||||
_, show, ep1 = _seed_plex(db, user)
|
||||
db.add(PlexState(user_id=user.id, item_id=ep1.id, status="watched", watched_at=NOW))
|
||||
db.commit() # ep2 has NO PlexState row
|
||||
assert queries.next_episode(db, user.id, show.id)["id"] == "ps1e2"
|
||||
|
||||
|
||||
def test_a_finished_show_starts_again_from_the_top(db):
|
||||
"""Nothing unwatched left is not "nothing to play" — it is "watch it again"."""
|
||||
user, _ = _seed_yt(db)
|
||||
_, show, ep1 = _seed_plex(db, user)
|
||||
ep2 = db.execute(select(PlexItem).where(PlexItem.rating_key == "ps1e2")).scalar_one()
|
||||
db.add_all(
|
||||
[
|
||||
PlexState(user_id=user.id, item_id=ep1.id, status="watched", watched_at=NOW),
|
||||
PlexState(user_id=user.id, item_id=ep2.id, status="watched", watched_at=NOW),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
assert queries.next_episode(db, user.id, show.id)["id"] == "ps1e1"
|
||||
|
||||
|
||||
def test_next_episode_agrees_with_the_show_page_when_two_are_in_progress(db):
|
||||
"""One show, one answer. `plex.rollup.rollup` — which drives the show page's Resume button —
|
||||
takes the LAST in-progress episode in broadcast order; taking the FIRST here gave the same show
|
||||
two different "continue" answers, with the rail ranking it by its most recent touch and then
|
||||
offering the episode abandoned weeks earlier. Two in-progress episodes are not exotic: a scrub to
|
||||
the end leaves a stale `status='new', position>0` row until the daily reconcile."""
|
||||
user, _ = _seed_yt(db)
|
||||
_, show, ep1 = _seed_plex(db, user)
|
||||
ep2 = db.execute(select(PlexItem).where(PlexItem.rating_key == "ps1e2")).scalar_one()
|
||||
db.add_all(
|
||||
[
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=ep1.id,
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW - timedelta(days=9),
|
||||
),
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=ep2.id,
|
||||
status="new",
|
||||
position_seconds=600,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
|
||||
got = queries.next_episode(db, user.id, show.id)
|
||||
assert got["id"] == "ps1e2"
|
||||
# Asserted against the other implementation rather than against a literal, so the two cannot
|
||||
# drift apart again without this failing.
|
||||
ordered = [
|
||||
leaf_card(ep, db.execute(select(PlexState).where(PlexState.item_id == ep.id)).scalar_one())
|
||||
for ep in (ep1, ep2)
|
||||
]
|
||||
assert rollup.rollup(ordered)["resume"]["id"] == got["id"]
|
||||
|
||||
|
||||
def test_next_episode_prefers_the_one_in_progress(db):
|
||||
"""In progress beats "next unwatched": resuming is cheaper than starting."""
|
||||
user, _ = _seed_yt(db)
|
||||
_, show, ep1 = _seed_plex(db, user)
|
||||
ep2 = db.execute(select(PlexItem).where(PlexItem.rating_key == "ps1e2")).scalar_one()
|
||||
db.add_all(
|
||||
[
|
||||
PlexState(user_id=user.id, item_id=ep1.id, status="watched", watched_at=NOW),
|
||||
PlexState(user_id=user.id, item_id=ep2.id, status="new", position_seconds=120),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
assert queries.next_episode(db, user.id, show.id)["id"] == "ps1e2"
|
||||
|
||||
|
||||
# ── HTTP surface ──────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_discover_endpoint_answers_in_one_pass(db, as_user):
|
||||
user, _ = _seed_yt(db)
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
r = as_user(user).get("/api/discover?budget=medium")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["budget"] == "medium"
|
||||
assert body["hero"]["tier"] == "resume"
|
||||
assert [c["id"] for c in body["continue_watching"]] == ["vmed"]
|
||||
assert body["comfort"] == []
|
||||
|
||||
|
||||
def test_an_unknown_budget_is_refused_rather_than_silently_widened(db, as_user):
|
||||
user, _ = _seed_yt(db)
|
||||
assert as_user(user).get("/api/discover?budget=whenever").status_code == 422
|
||||
|
||||
|
||||
def test_pinning_twice_is_the_same_shelf(db, as_user):
|
||||
"""The gesture is "make sure this is on my shelf", so a re-pin is a no-op, not a 409."""
|
||||
user, _ = _seed_yt(db)
|
||||
c = as_user(user)
|
||||
first = c.post("/api/discover/pins", json={"kind": "video", "ref": "vmed"}).json()
|
||||
second = c.post("/api/discover/pins", json={"kind": "video", "ref": "vmed"}).json()
|
||||
assert first["created"] is True and second["created"] is False
|
||||
assert first["id"] == second["id"]
|
||||
assert len(c.get("/api/discover/pins").json()["items"]) == 1
|
||||
|
||||
|
||||
def test_pinning_a_plex_show_resolves_to_the_episode_that_would_play(db, as_user):
|
||||
"""Two of the five pinnable kinds are Plex, and the rating_key → internal-id translation is
|
||||
theirs alone — a wrong column or a wrong model in either arm pins the wrong row, or 404s every
|
||||
Plex pin. Neither arm had a test before the epic-end review."""
|
||||
user, _ = _seed_yt(db)
|
||||
_, _, ep1 = _seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
db.add(PlexState(user_id=user.id, item_id=ep1.id, status="watched", watched_at=NOW))
|
||||
db.commit()
|
||||
c = as_user(user)
|
||||
assert c.post("/api/discover/pins", json={"kind": "plex_show", "ref": "ps1"}).status_code == 200
|
||||
(tile,) = c.get("/api/discover/pins").json()["items"]
|
||||
assert (tile["kind"], tile["id"]) == ("show", "ps1")
|
||||
assert tile["next"]["id"] == "ps1e2" # ep1 is finished
|
||||
|
||||
|
||||
def test_pinning_a_plex_movie_plays_itself_without_a_cycle(db, as_user):
|
||||
"""The self-reference that once 500'd the whole shelf, on the other leaf kind: `next` has to be a
|
||||
distinct dict, not the tile aliased into itself. Covered for a video since S1, never for a movie."""
|
||||
user, _ = _seed_yt(db)
|
||||
_seed_plex(db, user)
|
||||
_enable_plex(db)
|
||||
c = as_user(user)
|
||||
assert c.post("/api/discover/pins", json={"kind": "plex_item", "ref": "pm1"}).status_code == 200
|
||||
(tile,) = c.get("/api/discover/pins").json()["items"]
|
||||
assert tile["id"] == "pm1" and tile["next"]["id"] == "pm1"
|
||||
|
||||
|
||||
def test_the_reroll_endpoint_parses_its_whole_exclusion_list_and_its_press_count(db, as_user):
|
||||
"""The reroll's behaviour is covered at the domain level; the ROUTE's query-string translation —
|
||||
the comma split and the `start` passthrough — was not covered at all, so a change there could
|
||||
hand back the same card on every press with the suite green. Excluding all three candidates is
|
||||
what proves the whole list parsed: a delimiter change leaves one unsplit string in the set, and a
|
||||
hero comes back.
|
||||
|
||||
NOT the `MAX_EXCLUDE` clamp, which this sends three keys against a cap of 32 and cannot see. The
|
||||
docstring used to claim it did (round 2). The clamp is deliberately left unpinned: it guards
|
||||
nothing the route would otherwise do — see the constant's own comment — and a test that sent 33
|
||||
keys could only assert that the request still answers, which the comma split already proves."""
|
||||
user, _ = _seed_yt(db)
|
||||
db.add(
|
||||
VideoState(
|
||||
user_id=user.id,
|
||||
video_id="vmed",
|
||||
status="new",
|
||||
position_seconds=300,
|
||||
progress_updated_at=NOW,
|
||||
)
|
||||
)
|
||||
db.add(ComfortPin(user_id=user.id, video_id="vlong", position=0))
|
||||
db.commit()
|
||||
c = as_user(user)
|
||||
|
||||
# `start` rotates which tier answers first: 0 leads with resume, 1 with the shelf.
|
||||
assert c.get("/api/discover/pick?seed=1&start=0").json()["hero"]["tier"] == "resume"
|
||||
assert c.get("/api/discover/pick?seed=1&start=1").json()["hero"]["tier"] == "comfort"
|
||||
|
||||
skip = ",".join(f"youtube:{v}" for v in ("vshort", "vmed", "vlong"))
|
||||
assert c.get(f"/api/discover/pick?seed=2&exclude={skip}").json()["hero"] is None
|
||||
assert c.get("/api/discover/pick?budget=whenever").status_code == 422
|
||||
|
||||
|
||||
def test_pinning_something_that_does_not_exist_is_a_404(db, as_user):
|
||||
user, _ = _seed_yt(db)
|
||||
r = as_user(user).post("/api/discover/pins", json={"kind": "video", "ref": "nope"})
|
||||
assert r.status_code == 404
|
||||
|
||||
|
||||
def test_a_pin_whose_target_vanishes_mid_request_is_a_404_not_a_500(db, as_user, monkeypatch):
|
||||
"""`add_pin`'s `except IntegrityError` arm was reached by no test at all, so the 500→404
|
||||
narrowing could be reverted with the whole suite green (round 2). The 404 above comes from the
|
||||
`resolve_ref` PRE-CHECK, a different branch that returns before the INSERT.
|
||||
|
||||
The race the arm exists for — a Plex sync dropping the row between `resolve_ref` confirming it
|
||||
and the INSERT — needs two transactions, but the BRANCH does not: a `resolve_ref` pointing at an
|
||||
id no row carries lets the pre-check pass and trips the same FK on commit, with the re-query
|
||||
finding no winner. The sentinel has to FIT `comfort_pins.video_id` (varchar(16)) — a longer one
|
||||
raises `DataError` on the way in, which this arm does not catch and never meant to. Its sibling arm (a concurrent duplicate, where the re-query DOES find a winner
|
||||
and answers 200) genuinely needs a second connection and stays uncovered."""
|
||||
user, _ = _seed_yt(db)
|
||||
monkeypatch.setattr(pins, "resolve_ref", lambda *a, **k: ("video_id", "gone-mid-req"))
|
||||
r = as_user(user).post("/api/discover/pins", json={"kind": "video", "ref": "vshort"})
|
||||
assert r.status_code == 404
|
||||
|
||||
|
||||
def test_another_users_playlist_is_not_pinnable(db, as_user):
|
||||
"""404, not 403 — a 403 would confirm the id exists to a stranger."""
|
||||
user, _ = _seed_yt(db)
|
||||
other = User(email="other-e24@example.com")
|
||||
db.add(other)
|
||||
db.flush()
|
||||
pl = Playlist(user_id=other.id, name="Theirs")
|
||||
db.add(pl)
|
||||
db.commit()
|
||||
r = as_user(user).post("/api/discover/pins", json={"kind": "playlist", "ref": str(pl.id)})
|
||||
assert r.status_code == 404
|
||||
|
||||
|
||||
def test_reorder_ignores_ids_that_are_not_yours(db, as_user):
|
||||
"""A stale tab replaying an order that includes a since-deleted pin should reorder what still
|
||||
exists rather than fail wholesale."""
|
||||
user, ch = _seed_yt(db)
|
||||
c = as_user(user)
|
||||
a = c.post("/api/discover/pins", json={"kind": "video", "ref": "vmed"}).json()["id"]
|
||||
b = c.post("/api/discover/pins", json={"kind": "channel", "ref": ch.id}).json()["id"]
|
||||
assert c.post("/api/discover/pins/reorder", json={"ids": [b, 9999, a]}).status_code == 200
|
||||
assert [p["pin_id"] for p in c.get("/api/discover/pins").json()["items"]] == [b, a]
|
||||
|
||||
|
||||
def test_deleting_a_pin_reports_what_went(db, as_user):
|
||||
user, _ = _seed_yt(db)
|
||||
c = as_user(user)
|
||||
pin_id = c.post("/api/discover/pins", json={"kind": "video", "ref": "vmed"}).json()["id"]
|
||||
r = c.delete(f"/api/discover/pins/{pin_id}")
|
||||
assert r.status_code == 200 and r.json() == {"deleted": pin_id}
|
||||
assert c.get("/api/discover/pins").json()["items"] == []
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Category classification of raw yt-dlp / ffmpeg download-failure strings (S2)."""
|
||||
|
||||
from app.downloads import errors as dlerrors
|
||||
from app.downloads.errors import classify
|
||||
|
||||
|
||||
@@ -54,3 +55,22 @@ def test_login_beats_generic_unavailable_when_both_present():
|
||||
"This video is unavailable"
|
||||
)
|
||||
assert classify(msg) == "login_required"
|
||||
|
||||
|
||||
def test_our_own_worker_refusals_classify_as_themselves():
|
||||
# The worker raises these itself (not yt-dlp), and the rule that catches each one matches only a
|
||||
# fragment of the text. Pinning message and rule together is what stops an edit to the wording
|
||||
# from silently orphaning the category — the UI would then fall back to "Download failed".
|
||||
assert classify(dlerrors.QUOTA_FULL_MESSAGE) == "quota_exceeded"
|
||||
|
||||
|
||||
def test_a_url_jobs_network_failure_is_redacted_but_youtube_keeps_its_text():
|
||||
# SEC-12: the raw transport text for a user-chosen host:port is a port-scanner readout, so it is
|
||||
# replaced for `url` jobs. Everything else must survive verbatim, or a real diagnosis is lost.
|
||||
raw = "ERROR: Unable to download webpage: <urlopen error [Errno 111] Connection refused>"
|
||||
assert dlerrors.user_error_text("url", raw, "network") == dlerrors.NETWORK_REDACTED_MESSAGE
|
||||
# Same failure, but the destination was never user-chosen — nothing to leak, keep the detail.
|
||||
assert dlerrors.user_error_text("youtube", raw, "network") == raw
|
||||
# A user-chosen host, but a category that says something real about the CONTENT, not the network.
|
||||
login = "ERROR: [facebook] 123: You must be logged in to see this post"
|
||||
assert dlerrors.user_error_text("url", login, "login_required") == login
|
||||
|
||||
@@ -10,6 +10,8 @@ Uses the `db` fixture (real Postgres) — skipped when no database is reachable.
|
||||
its OWN session, so every test COMMITS its setup before calling it.
|
||||
"""
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import pipeline, quota
|
||||
from app.models import DownloadJob, DownloadQuota, MediaAsset, User
|
||||
from app.worker import _claim_job
|
||||
|
||||
@@ -169,3 +171,110 @@ def test_list_downloads_is_capped(db, monkeypatch):
|
||||
monkeypatch.setattr(dl, "_MAX_LIST", 2)
|
||||
rows = dl.list_downloads(user=u, db=db)
|
||||
assert len(rows) == 2 # bounded by the cap, not the 3 rows on disk
|
||||
|
||||
|
||||
# --- E12 · S3 · the worker-side max_bytes remainder ------------------------------------------
|
||||
|
||||
|
||||
def _sized_asset(db, size_bytes, ref):
|
||||
a = MediaAsset(
|
||||
source_kind="url",
|
||||
source_ref=ref,
|
||||
format_sig="s",
|
||||
status="ready",
|
||||
ref_count=1,
|
||||
size_bytes=size_bytes,
|
||||
)
|
||||
db.add(a)
|
||||
db.flush()
|
||||
return a
|
||||
|
||||
|
||||
def test_remaining_bytes_is_none_for_an_unlimited_user(db):
|
||||
u = User(email="unlimited@example.com")
|
||||
db.add(u)
|
||||
db.flush()
|
||||
_quota(db, u.id, max_concurrent=2, unlimited=True)
|
||||
db.flush()
|
||||
# None, not a number: "no cap" and "no budget left" must never be the same value downstream —
|
||||
# build_ydl_opts leaves max_filesize unset for both, so only the caller can tell them apart.
|
||||
assert quota.remaining_bytes(db, u.id) is None
|
||||
|
||||
|
||||
def test_remaining_bytes_is_the_unspent_part_of_the_cap(db):
|
||||
u = User(email="partial@example.com")
|
||||
db.add(u)
|
||||
db.flush()
|
||||
db.add(
|
||||
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False)
|
||||
)
|
||||
a = _sized_asset(db, 400, "partial-1")
|
||||
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
|
||||
db.flush()
|
||||
assert quota.remaining_bytes(db, u.id) == 600
|
||||
|
||||
|
||||
def test_remaining_bytes_is_zero_once_the_cap_is_met(db):
|
||||
# THE case this exists for: `check_enqueue` let these jobs in while the user was under the cap,
|
||||
# and an earlier one has filled it since. Zero is what makes the worker refuse; if this ever
|
||||
# returned the raw negative or None instead, the user already over quota would be the only one
|
||||
# downloading with no size ceiling at all.
|
||||
u = User(email="atcap@example.com")
|
||||
db.add(u)
|
||||
db.flush()
|
||||
db.add(
|
||||
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False)
|
||||
)
|
||||
a = _sized_asset(db, 1500, "atcap-1") # overshot: one job finished larger than the remainder
|
||||
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
|
||||
db.flush()
|
||||
assert quota.remaining_bytes(db, u.id) == 0
|
||||
|
||||
|
||||
def test_a_refusal_for_one_user_does_not_poison_the_shared_asset(db, monkeypatch, tmp_path):
|
||||
"""A user at their cap fails their own job, and another user's queued job for the same video
|
||||
still downloads.
|
||||
|
||||
MediaAssets are deduped across users, and the generic failure handler writes the message onto
|
||||
that shared row — correct for an upstream failure, wrong for "YOUR storage is full", which
|
||||
`worker._process_job` would then hand verbatim to someone with an empty library.
|
||||
"""
|
||||
monkeypatch.setattr(settings, "download_root", str(tmp_path))
|
||||
over, other = User(email="over@example.com"), User(email="other@example.com")
|
||||
db.add_all([over, other])
|
||||
db.flush()
|
||||
db.add(
|
||||
DownloadQuota(
|
||||
user_id=over.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False
|
||||
)
|
||||
)
|
||||
full = _sized_asset(db, 1500, "already-held")
|
||||
_job(db, over.id, queue_pos=0, status="done", asset_id=full.id)
|
||||
|
||||
# `downloading` is the state the worker leaves it in — `_claim_asset` compare-and-sets it before
|
||||
# the child runs — and `_reset_asset_pending` only acts on that status. Seeded as `pending` this
|
||||
# assertion would pass with the release deleted.
|
||||
shared = MediaAsset(
|
||||
source_kind="youtube",
|
||||
source_ref="x" * 11,
|
||||
format_sig="s",
|
||||
status="downloading",
|
||||
ref_count=2,
|
||||
)
|
||||
db.add(shared)
|
||||
db.flush()
|
||||
mine = _job(db, over.id, queue_pos=1, status="running", asset_id=shared.id)
|
||||
theirs = _job(db, other.id, queue_pos=2, asset_id=shared.id)
|
||||
db.commit()
|
||||
|
||||
# The refusal happens before extraction, so nothing needs stubbing — if it ever stops happening
|
||||
# first, this test starts reaching the network, which is its own signal.
|
||||
pipeline._download(mine.id, shared.id, "youtube", "x" * 11, {}, over.id)
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(DownloadJob, mine.id).status == "error"
|
||||
assert db.get(DownloadJob, mine.id).error_code == "quota_exceeded"
|
||||
# The asset is released, NOT marked error — otherwise the next line's job inherits the refusal.
|
||||
assert db.get(MediaAsset, shared.id).status == "pending"
|
||||
assert db.get(MediaAsset, shared.id).error is None
|
||||
assert db.get(DownloadJob, theirs.id).status == "queued"
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
"""An undelivered email leaves a trace in the admins' inbox.
|
||||
|
||||
Sending is deliberately fail-soft — a missing SMTP setting or a rejecting relay logs and returns
|
||||
False so nothing user-facing breaks. The cost of that used to be total silence: an instance whose
|
||||
SMTP was never filled in swallowed verification links, password resets and access-request notices,
|
||||
and the operator's first clue was somebody saying nothing arrived.
|
||||
"""
|
||||
|
||||
import smtplib
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import email as email_mod
|
||||
from app.models import Notification
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_smtp_config(monkeypatch):
|
||||
"""Every test here drives the real `_send`, so make sure it never reaches a socket."""
|
||||
monkeypatch.setattr(
|
||||
email_mod,
|
||||
"_smtp",
|
||||
lambda: {"host": "", "port": 587, "user": "", "password": "", "from": ""},
|
||||
)
|
||||
|
||||
|
||||
def _notifications(db, user_id: int) -> list[Notification]:
|
||||
return list(
|
||||
db.execute(
|
||||
select(Notification)
|
||||
.where(Notification.user_id == user_id, Notification.type == "email_undelivered")
|
||||
.order_by(Notification.id)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def test_a_skipped_email_notifies_every_admin(db, make_user):
|
||||
admin_a = make_user(email="admin-a@example.com", role="admin")
|
||||
admin_b = make_user(email="admin-b@example.com", role="admin")
|
||||
plain = make_user(email="plain@example.com")
|
||||
|
||||
assert email_mod.send_test("someone@example.com", "en") is False # fail-soft, as before
|
||||
|
||||
db.expire_all()
|
||||
for admin in (admin_a, admin_b):
|
||||
(n,) = _notifications(db, admin.id)
|
||||
assert n.data["reason"] == "not_configured"
|
||||
assert n.data["subject"] == "Siftlode SMTP test"
|
||||
assert n.data["recipients"] == ["someone@example.com"]
|
||||
assert _notifications(db, plain.id) == [] # not an operator's problem to see
|
||||
|
||||
|
||||
def test_the_notice_is_throttled_so_a_dead_relay_cannot_flood_the_inbox(db, make_user):
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
|
||||
for _ in range(5):
|
||||
email_mod.send_test("someone@example.com", "en")
|
||||
|
||||
db.expire_all()
|
||||
assert len(_notifications(db, admin.id)) == 1
|
||||
|
||||
|
||||
def test_a_send_failure_is_reported_separately_from_a_missing_config(db, make_user, monkeypatch):
|
||||
# A configured-but-rejecting relay is a different operator problem from an empty config, and the
|
||||
# inbox copy differs — so the reason has to survive to the notification.
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
monkeypatch.setattr(
|
||||
email_mod,
|
||||
"_smtp",
|
||||
lambda: {
|
||||
"host": "smtp.example.com",
|
||||
"port": 587,
|
||||
"user": "u",
|
||||
"password": "p",
|
||||
"from": "Siftlode <u@example.com>",
|
||||
},
|
||||
)
|
||||
|
||||
def _boom(*a, **kw):
|
||||
raise smtplib.SMTPException("nope")
|
||||
|
||||
monkeypatch.setattr(email_mod.smtplib, "SMTP", _boom)
|
||||
|
||||
assert email_mod.send_test("someone@example.com", "en") is False
|
||||
|
||||
db.expire_all()
|
||||
(n,) = _notifications(db, admin.id)
|
||||
assert n.data["reason"] == "send_failed"
|
||||
|
||||
|
||||
def test_no_admins_means_no_notification_and_no_error(db, make_user):
|
||||
# A fresh instance can have no admin yet (the wizard has not run). The alert must stay silent
|
||||
# rather than blowing up inside the fail-soft path it is supposed to make visible.
|
||||
plain = make_user(email="plain@example.com")
|
||||
|
||||
assert email_mod.send_test("someone@example.com", "en") is False
|
||||
|
||||
db.expire_all()
|
||||
assert _notifications(db, plain.id) == []
|
||||
|
||||
|
||||
def test_a_drop_with_nobody_to_tell_does_not_burn_the_hour(db, make_user):
|
||||
# The throttle window must only be spent on a notice that was actually written. Otherwise a drop
|
||||
# during setup — before any admin exists — silences the next hour for the admin created a minute
|
||||
# later, which is exactly the blind spot this alert exists to remove.
|
||||
email_mod.send_test("someone@example.com", "en") # no admins yet
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
|
||||
assert email_mod.send_test("someone@example.com", "en") is False
|
||||
|
||||
db.expire_all()
|
||||
assert len(_notifications(db, admin.id)) == 1
|
||||
|
||||
|
||||
def test_a_failed_alert_does_not_burn_the_hour_either(db, make_user, monkeypatch):
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
boom = {"on": True}
|
||||
|
||||
def _maybe_boom(*a, **kw):
|
||||
if boom["on"]:
|
||||
raise RuntimeError("notification store is down")
|
||||
from app.notifications import create_notification as real
|
||||
|
||||
return real(*a, **kw)
|
||||
|
||||
monkeypatch.setattr(email_mod, "create_notification", _maybe_boom)
|
||||
email_mod.send_test("someone@example.com", "en") # the write blows up; nothing recorded
|
||||
boom["on"] = False
|
||||
|
||||
assert email_mod.send_test("someone@example.com", "en") is False
|
||||
|
||||
db.expire_all()
|
||||
assert len(_notifications(db, admin.id)) == 1
|
||||
|
||||
|
||||
def test_a_broken_alert_never_breaks_the_send(db, make_user, monkeypatch):
|
||||
# The alert is the fallback path; if IT fails, the caller must still get the ordinary fail-soft
|
||||
# False rather than an exception surfacing from a background task.
|
||||
make_user(email="admin@example.com", role="admin")
|
||||
|
||||
def _boom(*a, **kw):
|
||||
raise RuntimeError("notification store is down")
|
||||
|
||||
monkeypatch.setattr(email_mod, "create_notification", _boom)
|
||||
|
||||
assert email_mod.send_test("someone@example.com", "en") is False
|
||||
@@ -0,0 +1,372 @@
|
||||
"""One mailbox, many spellings — the canonical-email identity rule (E9.5 S3).
|
||||
|
||||
Gmail delivers `x.y@gmail.com`, `xy@gmail.com` and `xy+tag@gmail.com` to ONE inbox. The invite gate
|
||||
used to compare literal strings, so on 2026-09-03 an approved user's Google sign-in became a silent
|
||||
SECOND access request: they had registered as `npeter83.temu@gmail.com`, and OAuth returned the
|
||||
account's stored primary `npeter83temu@gmail.com`. The `+tag` case fails every time — a Google account
|
||||
cannot contain `+`, so OAuth can only ever answer with the base address.
|
||||
|
||||
The rule is narrow on purpose: dots are significant on most providers, and the Gmail dot rule does not
|
||||
hold for Google Workspace custom domains unless their admin enabled it.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select, text
|
||||
|
||||
_UAT_PW = "correct horse battery"
|
||||
|
||||
from app.auth import is_allowed
|
||||
from app.models import Invite, User
|
||||
from app.utils import canonical_email
|
||||
|
||||
# (input, expected canonical) — the matrix the sprint promised, plus the shapes that must NOT fold.
|
||||
CASES = [
|
||||
# Gmail: dots and +tags are noise…
|
||||
("npeter83.temu@gmail.com", "npeter83temu@gmail.com"),
|
||||
("npeter83temu@gmail.com", "npeter83temu@gmail.com"),
|
||||
("n.p.e.t.e.r@gmail.com", "npeter@gmail.com"),
|
||||
("user+siftlode@gmail.com", "user@gmail.com"),
|
||||
("u.ser+a+b@gmail.com", "user@gmail.com"),
|
||||
("USer.Name@GMail.COM", "username@gmail.com"),
|
||||
# …and googlemail is the same mailbox as gmail.
|
||||
("user.name@googlemail.com", "username@gmail.com"),
|
||||
# Everyone else: lowercase ONLY. Dots stay significant.
|
||||
("first.last@example.com", "first.last@example.com"),
|
||||
("First.Last@Example.COM", "first.last@example.com"),
|
||||
("first+tag@example.com", "first+tag@example.com"),
|
||||
# A Workspace domain is not Gmail, however Gmail-ish it looks.
|
||||
("first.last@gmail.company.com", "first.last@gmail.company.com"),
|
||||
# Degenerate input must not explode or invent an address.
|
||||
("", ""),
|
||||
(None, ""),
|
||||
("not-an-email", "not-an-email"),
|
||||
(".@gmail.com", ".@gmail.com"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,expected", CASES)
|
||||
def test_canonical_email(raw, expected):
|
||||
assert canonical_email(raw) == expected
|
||||
|
||||
|
||||
def test_the_migration_sql_agrees_with_the_helper(db):
|
||||
"""The 0079 backfill reimplements canonical_email in SQL (a migration must not import app code
|
||||
that keeps changing). That duplication is only safe while the two agree, so pin them together —
|
||||
a drift here means rows backfilled into a form the application will never look up."""
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
path = Path(__file__).resolve().parents[1] / "alembic" / "versions" / "0079_email_canonical.py"
|
||||
spec = importlib.util.spec_from_file_location("m0079", path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
|
||||
sql = module._CANON_SQL.format(col=":addr")
|
||||
for raw, expected in CASES:
|
||||
if raw is None:
|
||||
continue # the column is NOT NULL; the backfill skips NULL emails explicitly
|
||||
got = db.execute(text(f"SELECT {sql}"), {"addr": raw}).scalar_one()
|
||||
assert got == expected, f"SQL and helper disagree on {raw!r}: {got!r} != {expected!r}"
|
||||
|
||||
|
||||
def test_the_column_is_kept_in_sync_by_writing_the_email(db):
|
||||
"""No write site has to remember: the model validator derives it. If this breaks, every lookup
|
||||
below silently stops matching."""
|
||||
user = User(email="Dotted.Name+tag@GoogleMail.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
assert user.email == "Dotted.Name+tag@GoogleMail.com" # stored AS TYPED — mail goes here
|
||||
assert user.email_canonical == "dottedname@gmail.com" # …matched on this
|
||||
|
||||
user.email = "someone.else@example.com"
|
||||
db.commit()
|
||||
assert user.email_canonical == "someone.else@example.com"
|
||||
|
||||
|
||||
def test_an_approval_covers_every_spelling_of_the_same_gmail_mailbox(db):
|
||||
# THE 2026-09-03 lockout, in one assertion: approved under the dotted spelling, signing in with
|
||||
# the primary Google returns.
|
||||
db.add(Invite(email="npeter83.temu@gmail.com", status="approved"))
|
||||
db.commit()
|
||||
|
||||
assert is_allowed(db, "npeter83temu@gmail.com") is True
|
||||
assert is_allowed(db, "npeter83.temu@gmail.com") is True
|
||||
assert is_allowed(db, "npeter83.temu+throwaway@gmail.com") is True
|
||||
|
||||
|
||||
def test_a_plus_tagged_registration_is_not_locked_out_after_approval(db):
|
||||
# The case that ALWAYS broke: a Google account cannot contain `+`, so OAuth can only return the
|
||||
# base address, and a literal match could never find the invite.
|
||||
db.add(Invite(email="someone+siftlode@gmail.com", status="approved"))
|
||||
db.commit()
|
||||
|
||||
assert is_allowed(db, "someone@gmail.com") is True
|
||||
|
||||
|
||||
def test_dots_stay_significant_outside_gmail(db):
|
||||
db.add(Invite(email="first.last@example.com", status="approved"))
|
||||
db.commit()
|
||||
|
||||
assert is_allowed(db, "first.last@example.com") is True
|
||||
assert is_allowed(db, "firstlast@example.com") is False # a DIFFERENT mailbox elsewhere
|
||||
|
||||
|
||||
def test_an_approved_row_wins_over_a_denied_sibling(db):
|
||||
# Prod carries approved pairs for one mailbox (oldest 2026-06-16). The API cannot flip an approved
|
||||
# invite to denied, so approved-beside-denied means the admin approved that mailbox under another
|
||||
# spelling — not that they revoked it. Revoking is deleting the row or suspending the account.
|
||||
db.add(Invite(email="two.spellings@gmail.com", status="denied"))
|
||||
db.add(Invite(email="twospellings@gmail.com", status="approved"))
|
||||
db.commit()
|
||||
|
||||
assert is_allowed(db, "twospellings@gmail.com") is True
|
||||
|
||||
|
||||
def test_a_denied_mailbox_stays_denied(db):
|
||||
db.add(Invite(email="no.thanks@gmail.com", status="denied"))
|
||||
db.commit()
|
||||
|
||||
assert is_allowed(db, "nothanks@gmail.com") is False
|
||||
|
||||
|
||||
def test_a_second_spelling_does_not_open_a_second_access_request(db):
|
||||
"""The other half of the 2026-09-03 report: the admin queue got a duplicate request for a user
|
||||
who was already approved."""
|
||||
from app.routes.auth import upsert_pending_invite
|
||||
|
||||
db.add(Invite(email="already.approved@gmail.com", status="approved"))
|
||||
db.commit()
|
||||
|
||||
req = upsert_pending_invite(db, "alreadyapproved@gmail.com")
|
||||
|
||||
assert req.is_new is False # nothing new to notify an admin about
|
||||
assert req.invite is not None and req.invite.email == "already.approved@gmail.com"
|
||||
assert len(db.execute(select(Invite)).scalars().all()) == 1
|
||||
|
||||
|
||||
def test_manually_whitelisting_another_spelling_reuses_the_row(db):
|
||||
"""How prod ended up with hartmann.peter1986@ AND hartmannpeter1986@ both approved on the same
|
||||
day: the admin typed the other spelling and got a second row instead of a no-op."""
|
||||
from app.routes.admin import add_invite, AddInviteIn
|
||||
|
||||
admin = User(email="admin@example.com", role="admin", is_active=True, email_verified=True)
|
||||
db.add(admin)
|
||||
db.add(Invite(email="hartmann.peter1986@gmail.com", status="approved"))
|
||||
db.commit()
|
||||
|
||||
add_invite(AddInviteIn(email="hartmannpeter1986@gmail.com"), admin=admin, db=db)
|
||||
|
||||
rows = db.execute(select(Invite)).scalars().all()
|
||||
assert len(rows) == 1
|
||||
assert rows[0].email == "hartmann.peter1986@gmail.com" # the original spelling is preserved
|
||||
|
||||
|
||||
def _capture_reset_links(monkeypatch) -> list[str]:
|
||||
"""Patch the reset mailer and collect the recipients it is asked to write to."""
|
||||
from app.routes import auth as auth_mod
|
||||
|
||||
sent: list[str] = []
|
||||
monkeypatch.setattr(
|
||||
auth_mod.email_mod, "send_password_reset", lambda to, lang, link: sent.append(to)
|
||||
)
|
||||
return sent
|
||||
|
||||
|
||||
# --- signing in and recovering an account, whichever spelling of it the person types -------------
|
||||
#
|
||||
# The three flows that look an account up BY TYPED ADDRESS used to match literally, so a Gmail user
|
||||
# who registered `x.y@` and typed `xy@` met "Invalid email or password" at sign-in — and, worse,
|
||||
# total silence from password-reset and verify-resend, which answer uniformly by design.
|
||||
|
||||
|
||||
def test_the_password_works_with_any_spelling_of_the_same_mailbox(db):
|
||||
from app.auth import find_user_by_email
|
||||
|
||||
make = User(
|
||||
email="t.e.st+tag@gmail.com", email_verified=True, is_active=True, password_hash="x"
|
||||
)
|
||||
db.add(make)
|
||||
db.commit()
|
||||
|
||||
for typed in (
|
||||
"t.e.st+tag@gmail.com", # as registered
|
||||
"test@gmail.com", # the mailbox itself
|
||||
"t.e.st@gmail.com", # dots only
|
||||
"TEST@googlemail.com", # the alias domain
|
||||
):
|
||||
assert find_user_by_email(db, typed) is make, typed
|
||||
|
||||
|
||||
def test_an_unrelated_address_still_finds_nothing(db):
|
||||
from app.auth import find_user_by_email
|
||||
|
||||
db.add(User(email="someone@gmail.com", email_verified=True, is_active=True))
|
||||
db.commit()
|
||||
|
||||
assert find_user_by_email(db, "someoneelse@gmail.com") is None
|
||||
assert find_user_by_email(db, "some.one@example.com") is None # dots matter off Gmail
|
||||
|
||||
|
||||
def test_the_exact_spelling_wins_when_two_accounts_share_a_mailbox(db):
|
||||
# Prod carries such a pair (ids 13 and 14). Whoever types their own address in full must land on
|
||||
# their own account, not on whichever row happens to come first.
|
||||
from app.auth import find_user_by_email
|
||||
|
||||
dotted = User(email="npeter83.temu@gmail.com", email_verified=True, is_active=True)
|
||||
plain = User(email="npeter83temu@gmail.com", email_verified=True, is_active=True)
|
||||
db.add_all([dotted, plain])
|
||||
db.commit()
|
||||
|
||||
assert find_user_by_email(db, "npeter83.temu@gmail.com") is dotted
|
||||
assert find_user_by_email(db, "npeter83temu@gmail.com") is plain
|
||||
|
||||
|
||||
def test_an_ambiguous_spelling_refuses_rather_than_guessing(db):
|
||||
# Neither spelling is exact, and both are the same mailbox. Handing a password sign-in one of
|
||||
# them at random would let someone into the wrong account, so this answers "no such account" —
|
||||
# the same uniform reply the caller already gives.
|
||||
from app.auth import find_user_by_email
|
||||
|
||||
db.add(User(email="npeter83.temu@gmail.com", email_verified=True, is_active=True))
|
||||
db.add(User(email="npeter83temu@gmail.com", email_verified=True, is_active=True))
|
||||
db.commit()
|
||||
|
||||
assert find_user_by_email(db, "n.peter83.temu+x@gmail.com") is None
|
||||
|
||||
|
||||
def test_a_reset_link_goes_to_the_address_on_the_account(db, monkeypatch):
|
||||
"""Typing another spelling must still reach the owner — and the mail must go to the address they
|
||||
registered with, tag and all, not to the one typed into the form."""
|
||||
from app.routes.auth import _send_reset_if_eligible
|
||||
|
||||
db.add(
|
||||
User(
|
||||
email="me+siftlode@gmail.com",
|
||||
email_verified=True,
|
||||
is_active=True,
|
||||
password_hash="x",
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
sent = _capture_reset_links(monkeypatch)
|
||||
|
||||
_send_reset_if_eligible("me@gmail.com")
|
||||
|
||||
assert sent == ["me+siftlode@gmail.com"]
|
||||
|
||||
|
||||
# --- a Google sign-in must not rewrite the address its owner chose ------------------------------
|
||||
|
||||
|
||||
def test_a_google_sign_in_keeps_a_plus_tagged_address(db):
|
||||
"""People register as `me+siftlode@gmail.com` so a later leak from this site is identifiable.
|
||||
Google returns the account's primary address, which can never contain `+`, so following it
|
||||
blindly destroyed the tag on the first Google sign-in — and, since sign-in matches the stored
|
||||
address first, stopped them signing in with the address they had chosen."""
|
||||
from app.routes.auth import _sync_google_email
|
||||
|
||||
user = User(email="me+siftlode@gmail.com", email_verified=True, is_active=True)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
|
||||
_sync_google_email(db, user, "me@gmail.com")
|
||||
|
||||
assert user.email == "me+siftlode@gmail.com"
|
||||
|
||||
|
||||
def test_a_google_sign_in_follows_a_real_mailbox_change(db):
|
||||
# The case the sync exists for: the Google account genuinely moved to a different mailbox.
|
||||
from app.routes.auth import _sync_google_email
|
||||
|
||||
user = User(email="old.address@gmail.com", email_verified=True, is_active=True)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
|
||||
_sync_google_email(db, user, "brand-new@example.com")
|
||||
|
||||
assert user.email == "brand-new@example.com"
|
||||
|
||||
|
||||
def test_a_mailbox_change_is_refused_when_another_account_owns_it(db):
|
||||
from app.routes.auth import _sync_google_email
|
||||
|
||||
db.add(User(email="taken@example.com", email_verified=True, is_active=True))
|
||||
user = User(email="mover@gmail.com", email_verified=True, is_active=True)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
|
||||
_sync_google_email(db, user, "taken@example.com")
|
||||
|
||||
assert user.email == "mover@gmail.com" # unique constraint would have 500'd the sign-in
|
||||
|
||||
|
||||
# --- showing BOTH addresses, once there are two -------------------------------------------------
|
||||
|
||||
|
||||
def test_the_google_address_is_recorded_beside_the_registered_one(db):
|
||||
"""Since the sign-in no longer overwrites the registered address, the account has two true
|
||||
addresses and only one was stored. This is the other half, for the UI to name."""
|
||||
from app.routes.auth import _sync_google_email
|
||||
|
||||
user = User(email="me+siftlode@gmail.com", email_verified=True, is_active=True)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
|
||||
user.google_email = "me@gmail.com"
|
||||
_sync_google_email(db, user, "me@gmail.com")
|
||||
db.commit()
|
||||
|
||||
assert user.email == "me+siftlode@gmail.com" # still theirs
|
||||
assert user.google_email == "me@gmail.com" # …and now we know the other one
|
||||
|
||||
|
||||
def test_me_reports_the_google_address_only_when_it_differs(client, db, make_user):
|
||||
"""The comparison lives in one place so no surface has to repeat it — and so the account menu
|
||||
stays exactly as it is for the majority, whose two addresses are the same string."""
|
||||
same = make_user(email="plain@gmail.com", password=_UAT_PW)
|
||||
same.google_email = "plain@gmail.com"
|
||||
db.commit()
|
||||
assert (
|
||||
client.post(
|
||||
"/auth/password-login", json={"email": "plain@gmail.com", "password": _UAT_PW}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
assert client.get("/api/me").json()["google_email"] is None
|
||||
|
||||
client.post("/auth/logout")
|
||||
tagged = make_user(email="me+siftlode@gmail.com", password=_UAT_PW)
|
||||
tagged.google_email = "me@gmail.com"
|
||||
db.commit()
|
||||
assert (
|
||||
client.post(
|
||||
"/auth/password-login", json={"email": "me+siftlode@gmail.com", "password": _UAT_PW}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
assert client.get("/api/me").json()["google_email"] == "me@gmail.com"
|
||||
|
||||
|
||||
# --- DX-17 · the setup wizard resolves its admin canonically ---------------------------------
|
||||
|
||||
|
||||
def test_rerunning_the_wizard_with_another_gmail_spelling_updates_the_same_admin(db):
|
||||
"""The wizard can be re-run, and E9.5's premise is that two spellings of one Gmail mailbox are one
|
||||
person. Resolving by the literal string here would create a SECOND admin row — exactly the
|
||||
duplicate the epic exists to prevent. This site was outside S3's sweep (DX-17).
|
||||
|
||||
Calls the handler directly: `require_setup` is an instance-state gate, not the behaviour under
|
||||
test, and the test fixtures present an already-configured instance."""
|
||||
from app.routes.setup import SetupAdminIn, setup_admin
|
||||
|
||||
setup_admin(SetupAdminIn(email="a.b.c@gmail.com", password=_UAT_PW), db)
|
||||
setup_admin(SetupAdminIn(email="abc@gmail.com", password="another good passphrase"), db)
|
||||
|
||||
admins = db.execute(select(User).where(User.role == "admin")).scalars().all()
|
||||
assert len(admins) == 1, [u.email for u in admins]
|
||||
# The address stays AS TYPED on the row it matched; only identity is canonical.
|
||||
assert admins[0].email == "a.b.c@gmail.com"
|
||||
# And the second run's password is the live one — it updated, it did not fork.
|
||||
from app.security import verify_password
|
||||
|
||||
assert verify_password("another good passphrase", admins[0].password_hash)
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Every outbound mail carries an HTML alternative, so its link is actually clickable.
|
||||
|
||||
The bodies are plain text and stay that way — but a bare URL in a text/plain mail is only a link if
|
||||
the CLIENT chooses to linkify it. Gmail/Outlook/Apple Mail do; lighter web clients render the body
|
||||
verbatim and leave the verification or password-reset link as dead text, which is the entire point of
|
||||
those two messages. So `_send` now attaches an HTML part generated from the same plain body.
|
||||
|
||||
These are pure-logic tests except the last one, which drives the real `_send` against a fake SMTP so
|
||||
the wire format itself is pinned, not just the helper.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from app import email as email_mod
|
||||
|
||||
_SMTP_OK = {
|
||||
"host": "smtp.example.com",
|
||||
"port": 587,
|
||||
"user": "u",
|
||||
"password": "p",
|
||||
"from": "Siftlode <u@example.com>",
|
||||
}
|
||||
|
||||
|
||||
def test_a_bare_url_becomes_a_real_link():
|
||||
# THE regression: this is what a client that does not linkify plain text was leaving as dead text.
|
||||
out = email_mod._html_body(
|
||||
"Confirm your email:\n\nhttps://siftlode.example/#verify=tok123\n\n— Siftlode"
|
||||
)
|
||||
assert '<a href="https://siftlode.example/#verify=tok123">' in out
|
||||
|
||||
|
||||
def test_interpolated_content_cannot_inject_markup():
|
||||
# Bodies carry untrusted values — a requester's address in the admin notice, the operator contact.
|
||||
# Escaping must run BEFORE linkification, or a crafted address becomes markup in every admin's mail.
|
||||
out = email_mod._html_body('A request from <script>alert("x")</script> & co')
|
||||
assert "<script>" not in out
|
||||
assert "<script>" in out
|
||||
assert "& co" in out
|
||||
|
||||
|
||||
def test_a_query_string_keeps_its_escaped_ampersand_inside_the_href():
|
||||
# `&` is escaped to `&` before the linkifier sees it, which is also what an HTML attribute
|
||||
# wants — so the href must carry the escaped form and nothing may be trimmed off it.
|
||||
out = email_mod._html_body("https://siftlode.example/x?a=1&b=2")
|
||||
assert (
|
||||
'<a href="https://siftlode.example/x?a=1&b=2">https://siftlode.example/x?a=1&b=2</a>'
|
||||
in out
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"body,href,trail",
|
||||
[
|
||||
("See https://siftlode.example/x.", "https://siftlode.example/x", "."),
|
||||
("See https://siftlode.example/x, then sign in", "https://siftlode.example/x", ","),
|
||||
("See (https://siftlode.example/x)", "https://siftlode.example/x", ")"),
|
||||
],
|
||||
)
|
||||
def test_sentence_punctuation_stays_out_of_the_href(body, href, trail):
|
||||
out = email_mod._html_body(body)
|
||||
assert f'<a href="{href}">{href}</a>{trail}' in out
|
||||
|
||||
|
||||
def test_a_token_ending_in_a_url_safe_character_is_not_trimmed():
|
||||
# token_urlsafe output can end in `-` or `_`; trimming either would hand the user a dead link.
|
||||
for tok in ("abc-", "abc_"):
|
||||
out = email_mod._html_body(f"https://siftlode.example/#reset={tok}")
|
||||
assert f'href="https://siftlode.example/#reset={tok}"' in out
|
||||
|
||||
|
||||
def test_blank_lines_become_paragraphs_and_single_breaks_stay_inline():
|
||||
out = email_mod._html_body("Hi,\n\nline one\nline two\n\n— Siftlode")
|
||||
assert out.count("<p>") == 3
|
||||
assert "<p>line one<br>line two</p>" in out
|
||||
|
||||
|
||||
def test_the_sent_message_is_multipart_with_the_plain_body_intact(monkeypatch):
|
||||
# The wire format, not just the helper: text/plain stays the primary part and the source of
|
||||
# truth, with text/html alongside it.
|
||||
monkeypatch.setattr(email_mod, "_smtp", lambda: _SMTP_OK)
|
||||
sent = []
|
||||
|
||||
class _FakeSMTP:
|
||||
def __init__(self, *a, **kw):
|
||||
pass
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
def starttls(self, *a, **kw):
|
||||
pass
|
||||
|
||||
def login(self, *a, **kw):
|
||||
pass
|
||||
|
||||
def send_message(self, msg):
|
||||
sent.append(msg)
|
||||
|
||||
monkeypatch.setattr(email_mod.smtplib, "SMTP", _FakeSMTP)
|
||||
|
||||
assert (
|
||||
email_mod.send_verify_email(
|
||||
"someone@example.com", "en", "https://siftlode.example/#verify=tok"
|
||||
)
|
||||
is True
|
||||
)
|
||||
|
||||
(msg,) = sent
|
||||
assert msg.get_content_type() == "multipart/alternative"
|
||||
plain = msg.get_body(preferencelist=("plain",))
|
||||
rich = msg.get_body(preferencelist=("html",))
|
||||
assert plain is not None and rich is not None
|
||||
# The plain part must be byte-for-byte the template's own body — the HTML part is derived from
|
||||
# it, never a second copy that could drift.
|
||||
assert "https://siftlode.example/#verify=tok" in plain.get_content()
|
||||
assert "<a href=" not in plain.get_content()
|
||||
assert '<a href="https://siftlode.example/#verify=tok">' in rich.get_content()
|
||||
@@ -109,3 +109,52 @@ def test_browser_transcode_flags_universal_playability():
|
||||
assert F("h264", "aac", False) == (True, False, False)
|
||||
# A failed probe / missing stream is never re-encoded.
|
||||
assert F(None, None, True) == (False, False, False)
|
||||
|
||||
|
||||
# --- SEC-13 · one enqueue is one bounded download -------------------------------------------
|
||||
|
||||
|
||||
def test_a_playlist_url_can_only_yield_one_item():
|
||||
# `noplaylist` alone only demotes a video-inside-a-playlist URL to that video; a URL that is
|
||||
# purely a playlist or channel has no such video, so without this it expands into every entry.
|
||||
opts = _opts({})
|
||||
assert opts["noplaylist"] is True
|
||||
assert opts["playlist_items"] == "1"
|
||||
|
||||
|
||||
def test_the_remaining_quota_becomes_a_hard_size_ceiling():
|
||||
opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000)
|
||||
assert opts["max_filesize"] == 5_000_000
|
||||
|
||||
|
||||
def test_no_budget_means_no_size_filter():
|
||||
# Unlimited users pass None; a non-positive remainder is the quota layer's refusal to make at
|
||||
# enqueue, not an impossible yt-dlp filter to express here.
|
||||
for budget in (None, 0, -1):
|
||||
assert "max_filesize" not in formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, max_bytes=budget
|
||||
)
|
||||
|
||||
|
||||
def test_a_source_adapter_cannot_raise_the_size_ceiling():
|
||||
# The adapter merge happens after max_filesize is set, so this pins that no adapter option can
|
||||
# widen a storage cap — the mirror of the cookiefile rule, in the opposite direction.
|
||||
opts = formats.build_ydl_opts(
|
||||
{}, "/tmp/x", lambda d: None, extra={"max_filesize": 999_999_999}, max_bytes=5_000_000
|
||||
)
|
||||
assert opts["max_filesize"] == 5_000_000
|
||||
|
||||
|
||||
def test_a_join_cut_list_is_capped():
|
||||
# An accurate join emits two filters per segment into one ffmpeg filter_complex, so an unbounded
|
||||
# list is an unbounded command line. The surplus is dropped; the legitimate head survives.
|
||||
segs = [{"start_s": i, "end_s": i + 1} for i in range(500)]
|
||||
out = editmod.normalize_edit_spec({"segments": segs})
|
||||
assert len(out["segments"]) == editmod._MAX_SEGMENTS
|
||||
assert out["segments"][0] == {"start_s": 0.0, "end_s": 1.0}
|
||||
|
||||
|
||||
def test_a_normal_cut_list_is_untouched():
|
||||
segs = [{"start_s": 0, "end_s": 5}, {"start_s": 10, "end_s": 12}]
|
||||
out = editmod.normalize_edit_spec({"segments": segs})
|
||||
assert len(out["segments"]) == 2
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
"""Idle HLS sessions are reaped on a timer, and a viewer who is merely PAUSED keeps theirs.
|
||||
|
||||
plex/stream.py promises "a periodic reaper kills idle ones", but `reap_idle` ran only inside
|
||||
POST /session: after the last playback, an abandoned session (its ffmpeg still working through the
|
||||
film, its directory a whole segmented copy of it) lived until somebody pressed play again
|
||||
(deep-review round 4).
|
||||
|
||||
And it must not take a PAUSED viewer's session: a remux finishes long before the film does, and a
|
||||
paused player with a full buffer fetches nothing, so "finished and unfetched for 30 s" does not mean
|
||||
"abandoned". Round 4 bought that safety by keeping EVERY finished session for the full ten-minute
|
||||
idle window — one segmented copy of a film per item, on the volume the download center shares. Round
|
||||
5 replaced the blanket retention with the signal that already existed: an open player checkpoints its
|
||||
position every 10 s whether or not it is playing, and that route says "still here".
|
||||
|
||||
Both call sites are pinned here, because the bug each time was that only one of them moved: the
|
||||
periodic pass, and the fact that starting a session no longer sweeps OTHER viewers' sessions at all.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import app.main as app_main
|
||||
import app.state as state
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.models import PlexItem, PlexLibrary, User
|
||||
from app.plex import paths as plex_paths
|
||||
from app.plex import stream
|
||||
|
||||
|
||||
class _Configured:
|
||||
def __enter__(self):
|
||||
return None
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
|
||||
def test_the_periodic_pass_is_the_one_that_runs_and_it_uses_the_short_rule(monkeypatch):
|
||||
async def _no_sweep():
|
||||
return None
|
||||
|
||||
reaped: list[int] = []
|
||||
monkeypatch.setattr(app_main, "SessionLocal", _Configured)
|
||||
monkeypatch.setattr(app_main.state, "is_configured", lambda db: True)
|
||||
monkeypatch.setattr(app_main, "_sweep_hls_orphans", _no_sweep)
|
||||
monkeypatch.setattr(app_main, "start_scheduler", lambda: None)
|
||||
monkeypatch.setattr(app_main, "shutdown_scheduler", lambda: None)
|
||||
monkeypatch.setattr(app_main.plex_subtitles, "shutdown", lambda: None)
|
||||
monkeypatch.setattr(app_main, "_HLS_REAP_EVERY_S", 0.01)
|
||||
# `reap_idle`, i.e. the SHORT finished rule: a finished session nobody has said they still want
|
||||
# is a whole copy of a film sitting on the download volume.
|
||||
monkeypatch.setattr(app_main.plex_stream, "reap_idle", lambda: reaped.append(1) or 0)
|
||||
|
||||
async def _cycle():
|
||||
async with app_main.lifespan(app):
|
||||
await asyncio.sleep(0.2)
|
||||
|
||||
asyncio.run(_cycle())
|
||||
assert len(reaped) >= 2, "no periodic reap ran while the app was up"
|
||||
|
||||
|
||||
class _Proc:
|
||||
def __init__(self, finished: bool):
|
||||
self.finished = finished
|
||||
self.killed = False
|
||||
|
||||
def poll(self):
|
||||
return 0 if self.finished else None
|
||||
|
||||
def terminate(self):
|
||||
self.killed = True
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return 0
|
||||
|
||||
|
||||
class _Session:
|
||||
def __init__(self, tmp_path, name: str, finished: bool, idle_s: float):
|
||||
self.dir = tmp_path / name
|
||||
self.dir.mkdir()
|
||||
self.proc = _Proc(finished)
|
||||
self.last_access = time.time() - idle_s
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sessions(monkeypatch):
|
||||
live: dict = {}
|
||||
monkeypatch.setattr(stream, "_sessions", live)
|
||||
return live
|
||||
|
||||
|
||||
def test_the_periodic_pass_reaps_a_finished_session_nobody_has_claimed(tmp_path, sessions):
|
||||
sessions["12345"] = _Session(tmp_path, "done", finished=True, idle_s=45)
|
||||
assert stream.reap_idle() == 1
|
||||
assert sessions == {}
|
||||
|
||||
|
||||
def test_the_periodic_pass_still_reaps_what_nobody_has_touched_for_the_idle_limit(
|
||||
tmp_path, sessions
|
||||
):
|
||||
sessions["1"] = _Session(
|
||||
tmp_path, "gone_done", finished=True, idle_s=stream._SESSION_IDLE_S + 5
|
||||
)
|
||||
sessions["2"] = _Session(
|
||||
tmp_path, "gone_live", finished=False, idle_s=stream._SESSION_IDLE_S + 5
|
||||
)
|
||||
assert stream.reap_idle() == 2
|
||||
assert sessions == {}
|
||||
|
||||
|
||||
def test_touching_a_session_keeps_it_through_the_short_rule(tmp_path, sessions):
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
sessions["12345"] = s
|
||||
stream.touch_session("12345")
|
||||
assert stream.reap_idle() == 0
|
||||
assert "12345" in sessions and s.dir.exists() and not s.proc.killed
|
||||
|
||||
|
||||
# --- the two call sites, driven through the real routes (DB lane) -------------------------------
|
||||
|
||||
|
||||
def _library_item(db, rating_key: str, *, playable: str = "remux"):
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
it = PlexItem(
|
||||
rating_key=rating_key,
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="A Movie",
|
||||
file_path="/media/a.mkv",
|
||||
playable=playable,
|
||||
)
|
||||
db.add(it)
|
||||
db.commit()
|
||||
return it
|
||||
|
||||
|
||||
def _client(db, user):
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
app.dependency_overrides[current_user] = lambda: user
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_a_paused_viewers_checkpoint_keeps_their_session_alive(db, monkeypatch, tmp_path, sessions):
|
||||
"""The whole reason the short rule is safe again: a paused player fetches nothing, so
|
||||
`last_access` never moved on its own — but it does keep checkpointing its position."""
|
||||
monkeypatch.setattr(state, "_configured_cache", True)
|
||||
it = _library_item(db, "paused1")
|
||||
user = User(email="paused@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
s = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
sessions["paused1"] = s
|
||||
try:
|
||||
resp = _client(db, user).post(
|
||||
f"/api/plex/item/{it.rating_key}/progress",
|
||||
json={"position_seconds": 600, "duration_seconds": 3000, "final": False},
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert stream.reap_idle() == 0, "the checkpoint did not say the viewer is still there"
|
||||
assert "paused1" in sessions and not s.proc.killed
|
||||
|
||||
|
||||
def test_starting_a_session_leaves_another_viewers_paused_one_alone(
|
||||
db, monkeypatch, tmp_path, sessions
|
||||
):
|
||||
"""The opportunistic sweep this route used to run was not scoped to the key being started: it
|
||||
walked the WHOLE registry with the 30 s finished rule, so pressing play on any item killed a
|
||||
paused viewer's session — the very case round 4 set out to protect (found in round 5)."""
|
||||
monkeypatch.setattr(state, "_configured_cache", True)
|
||||
monkeypatch.setattr(plex_paths, "local_media_path", lambda db_, fp: "/local/a.mkv")
|
||||
started = _library_item(db, "starting")
|
||||
user = User(email="starter@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
paused = _Session(tmp_path, "paused", finished=True, idle_s=90)
|
||||
sessions["someone-else"] = paused
|
||||
|
||||
def fake_start(db_, item, start, audio=None, aoff=0.0, multi=False, transcode=False):
|
||||
return SimpleNamespace(
|
||||
id="sid",
|
||||
media_start_s=start,
|
||||
entry="index.m3u8",
|
||||
transcode=transcode,
|
||||
hw=False,
|
||||
tonemap=False,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(stream, "start_session", fake_start)
|
||||
try:
|
||||
resp = _client(db, user).post(f"/api/plex/stream/{started.rating_key}/session")
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert "someone-else" in sessions and not paused.proc.killed
|
||||
@@ -0,0 +1,107 @@
|
||||
"""What a refused sign-in SAYS (E9.5 S4), DB-lane.
|
||||
|
||||
The states behind a 403 were true and useless: "Your account is awaiting admin approval." never
|
||||
revealed WHICH mailbox the account was on. On 2026-09-03 a registration as `npeter83.temu@gmail.com`
|
||||
met a Google claim of `npeter83temu@gmail.com`, and that missing address hid the mismatch for a day.
|
||||
So these pin the DETAIL, not just the status code — the address, and both gates, since E9.5 S1 made
|
||||
the two order-independent and the screen must report whichever is actually outstanding.
|
||||
|
||||
The two refusals that answer BEFORE the password is proven carry nothing, and that is the security
|
||||
half of the same contract: `invalid` must stay byte-identical for a real account and an unknown one.
|
||||
"""
|
||||
|
||||
from app.config import settings
|
||||
from app.models import Invite
|
||||
from app.routes.auth import _access_notice_url, upsert_pending_invite
|
||||
|
||||
_PW = "correct horse battery"
|
||||
|
||||
|
||||
def _login(client, email, password=_PW):
|
||||
return client.post("/auth/password-login", json={"email": email, "password": password})
|
||||
|
||||
|
||||
def test_pending_approval_names_the_address_the_account_is_on(client, make_user):
|
||||
# The typed spelling and the stored one are the same mailbox; the screen must show the STORED
|
||||
# one, because that is the mismatch a user cannot otherwise see.
|
||||
make_user(email="np.temu@gmail.com", password=_PW, is_active=False)
|
||||
|
||||
detail = _login(client, "nptemu@gmail.com").json()["detail"]
|
||||
|
||||
assert detail["reason"] == "pending_approval"
|
||||
assert detail["email"] == "np.temu@gmail.com"
|
||||
assert detail["email_verified"] is True and detail["approved"] is False
|
||||
|
||||
|
||||
def test_an_already_approved_account_still_reports_its_missing_verification(client, make_user):
|
||||
# S1 made the gates order-independent, so this really happens: approved, not yet verified. A
|
||||
# screen that assumed a sequence would tell this user to wait for an admin who already decided.
|
||||
make_user(
|
||||
email="approved.first@example.com", password=_PW, email_verified=False, is_active=True
|
||||
)
|
||||
|
||||
detail = _login(client, "approved.first@example.com").json()["detail"]
|
||||
|
||||
assert detail["reason"] == "email_unverified"
|
||||
assert detail["email_verified"] is False and detail["approved"] is True
|
||||
|
||||
|
||||
def test_a_suspended_account_is_told_who_to_contact(client, make_user, monkeypatch):
|
||||
monkeypatch.setattr(settings, "admin_emails", "ops@example.com")
|
||||
make_user(email="banned@example.com", password=_PW, is_suspended=True)
|
||||
|
||||
r = _login(client, "banned@example.com")
|
||||
|
||||
assert r.status_code == 403
|
||||
assert r.json()["detail"]["reason"] == "suspended"
|
||||
assert r.json()["detail"]["operator"] == "ops@example.com"
|
||||
|
||||
|
||||
def test_a_refusal_before_the_password_is_proven_carries_no_account_detail(client, make_user):
|
||||
make_user(email="real@example.com", password=_PW, is_active=False)
|
||||
|
||||
wrong_pw = _login(client, "real@example.com", "not the password").json()["detail"]
|
||||
unknown = _login(client, "nobody@example.com").json()["detail"]
|
||||
|
||||
# No address, no flags — the pending state above must not leak through the wrong-password door.
|
||||
assert wrong_pw == unknown == {"code": "login", "reason": "invalid"}
|
||||
|
||||
|
||||
# --- the access-request notice -------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_access_notice_names_the_address_and_rides_the_fragment():
|
||||
url = _access_notice_url("requested", "someone@example.com")
|
||||
|
||||
# A fragment never reaches a proxy access log or a Referer header — the same reasoning that put
|
||||
# the verify/reset tokens there (SB3), applied to an address.
|
||||
assert url == "/#access=requested&email=someone%40example.com"
|
||||
|
||||
|
||||
def test_a_plus_tagged_address_survives_the_round_trip():
|
||||
# URLSearchParams reads a bare `+` in a fragment as a SPACE, so the encoder must emit %2B — or
|
||||
# the very address family this epic exists to protect arrives mangled.
|
||||
url = _access_notice_url("requested", "me+siftlode@gmail.com")
|
||||
|
||||
assert "me%2Bsiftlode%40gmail.com" in url and "+siftlode" not in url
|
||||
|
||||
|
||||
def test_a_request_already_on_file_under_another_spelling_says_so(db):
|
||||
db.add(Invite(email="hartmann.peter@gmail.com", status="pending"))
|
||||
db.commit()
|
||||
|
||||
req = upsert_pending_invite(db, "hartmannpeter@gmail.com")
|
||||
url = _access_notice_url("requested", "hartmannpeter@gmail.com", req.invite)
|
||||
|
||||
assert req.is_new is False # one mailbox, one request — no second row for the admin queue
|
||||
assert "linked=hartmann.peter%40gmail.com" in url
|
||||
|
||||
|
||||
def test_the_same_spelling_twice_is_just_a_repeat_and_says_nothing_extra(db):
|
||||
db.add(Invite(email="same@example.com", status="pending"))
|
||||
db.commit()
|
||||
|
||||
req = upsert_pending_invite(db, "same@example.com")
|
||||
|
||||
# Nothing to explain: telling this user we "linked" their request to itself would be noise.
|
||||
assert "linked" not in _access_notice_url("requested", "same@example.com", req.invite)
|
||||
@@ -0,0 +1,289 @@
|
||||
"""E9.5 S7 — every outbound mail is written in the reader's language.
|
||||
|
||||
Two halves. The first is the catalog invariant: HU/EN parity with identical placeholders, enforced
|
||||
here because a mail is the one user-facing surface nobody can re-render by switching the UI language
|
||||
— it arrives once, in whatever language we picked. The second is the resolution chain: which
|
||||
language a given recipient actually gets, and where that signal was captured, since for an address
|
||||
with no account it exists only at the moment they asked us.
|
||||
"""
|
||||
|
||||
import re
|
||||
import string
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import mailtext
|
||||
from app.models import Invite, User
|
||||
from app.routes import auth as auth_routes
|
||||
|
||||
_PW = "correct horse battery"
|
||||
|
||||
|
||||
def _placeholders(template: str) -> set[str]:
|
||||
return {name for _, name, _, _ in string.Formatter().parse(template) if name}
|
||||
|
||||
|
||||
# --- the catalog itself ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key", sorted(mailtext.MAIL_TEXT))
|
||||
def test_every_mail_string_exists_in_both_languages(key):
|
||||
assert set(mailtext.MAIL_TEXT[key]) == set(mailtext.SUPPORTED)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key", sorted(mailtext.MAIL_TEXT))
|
||||
def test_every_translation_takes_the_same_placeholders(key):
|
||||
# The parity that actually bites: a translation that drops `{link}` renders a mail with no link
|
||||
# in it, and a translation that invents one raises KeyError inside a background task nobody is
|
||||
# watching. Both are silent in production and loud here.
|
||||
variants = mailtext.MAIL_TEXT[key]
|
||||
expected = _placeholders(variants[mailtext.DEFAULT])
|
||||
for lang, template in variants.items():
|
||||
assert _placeholders(template) == expected, f"{key} [{lang}]"
|
||||
|
||||
|
||||
def _app_sources_without_the_catalog() -> str:
|
||||
"""Every app source, with the catalog LITERAL cut out of mailtext.py.
|
||||
|
||||
Cutting it is the whole point: the dict spells each key as a quoted literal, so a scan that
|
||||
included it reported every key as used no matter what the catalog held. Cutting the whole FILE
|
||||
is too much, though — `link_expiry` legitimately asks for `duration.*`, and those keys would
|
||||
then look orphaned."""
|
||||
from pathlib import Path
|
||||
|
||||
app_dir = Path(__file__).resolve().parents[1] / "app"
|
||||
out = []
|
||||
for f in app_dir.rglob("*.py"):
|
||||
src = f.read_text(encoding="utf-8")
|
||||
if f.name == "mailtext.py":
|
||||
src = src[: src.index("MAIL_TEXT: dict")]
|
||||
out.append(src)
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def test_no_catalog_entry_is_orphaned():
|
||||
"""Every key is referenced from the app, so a mail that stopped being sent can't leave its text
|
||||
behind to be 'kept up to date' by the next person who greps for it."""
|
||||
sources = _app_sources_without_the_catalog()
|
||||
assert [k for k in mailtext.MAIL_TEXT if f'"{k}"' not in sources] == []
|
||||
|
||||
|
||||
def test_every_key_the_app_asks_for_exists():
|
||||
"""The other direction, and the one that raises in production: `text()` looks the key up in a
|
||||
dict, so a renamed or mistyped key is a KeyError inside a background task with no user in sight."""
|
||||
asked = set(re.findall(r'text\(\s*"([a-z_]+\.[a-z_]+)"', _app_sources_without_the_catalog()))
|
||||
assert asked, "the scan found no mailtext.text(...) call sites — the pattern has rotted"
|
||||
assert sorted(asked - set(mailtext.MAIL_TEXT)) == []
|
||||
|
||||
|
||||
def test_an_unknown_language_falls_back_to_english_instead_of_raising():
|
||||
# `preferences.language` is free-form JSON written by older builds and by locales we may stop
|
||||
# shipping. A send that raised on one would take out a background task with no user in sight.
|
||||
assert mailtext.text("test.subject", "de") == mailtext.text("test.subject", "en")
|
||||
assert mailtext.text("test.subject", None) == mailtext.text("test.subject", "en")
|
||||
|
||||
|
||||
# --- who gets which language ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_users_own_preference_decides_their_language(make_user):
|
||||
hu = make_user(email="hu@example.com", preferences={"language": "hu"})
|
||||
en = make_user(email="en@example.com", preferences={"language": "en"})
|
||||
none = make_user(email="none@example.com", preferences=None)
|
||||
|
||||
assert mailtext.lang_for_user(hu) == "hu"
|
||||
assert mailtext.lang_for_user(en) == "en"
|
||||
assert mailtext.lang_for_user(none) == "en"
|
||||
|
||||
|
||||
def test_an_address_with_no_account_falls_back_to_the_language_it_asked_in(db):
|
||||
db.add(Invite(email="asked@example.com", status="pending", lang="hu"))
|
||||
db.commit()
|
||||
|
||||
assert mailtext.lang_for_email(db, "asked@example.com") == "hu"
|
||||
|
||||
|
||||
def test_the_account_wins_over_the_invite_row(db, make_user):
|
||||
make_user(email="both@example.com", preferences={"language": "en"})
|
||||
db.add(Invite(email="both@example.com", status="approved", lang="hu"))
|
||||
db.commit()
|
||||
|
||||
# The invite records how they asked once; the account records what they have since chosen.
|
||||
assert mailtext.lang_for_email(db, "both@example.com") == "en"
|
||||
|
||||
|
||||
def test_a_row_that_never_recorded_a_language_is_walked_past(db, make_user):
|
||||
# Prod carries TWO accounts for one Gmail mailbox (the 2026-09-03 pair). Stopping at the first
|
||||
# row would answer English for a person whose other account says Hungarian.
|
||||
make_user(email="dup.one@gmail.com", preferences={})
|
||||
make_user(email="dupone@gmail.com", preferences={"language": "hu"})
|
||||
|
||||
assert mailtext.lang_for_email(db, "dup.one@gmail.com") == "hu"
|
||||
|
||||
|
||||
def test_the_lookup_is_canonical_like_every_other_identity_match(db, make_user):
|
||||
make_user(email="canon.match@gmail.com", preferences={"language": "hu"})
|
||||
|
||||
assert mailtext.lang_for_email(db, "canonmatch+tag@gmail.com") == "hu"
|
||||
|
||||
|
||||
# --- capturing the signal at the one moment it exists ---------------------------------------------
|
||||
|
||||
|
||||
def test_registration_records_the_language_of_the_sign_up_page(db, client):
|
||||
r = client.post(
|
||||
"/auth/register", json={"email": "signup@example.com", "password": _PW, "lang": "hu"}
|
||||
)
|
||||
|
||||
assert r.status_code == 200
|
||||
user = db.execute(select(User).where(User.email == "signup@example.com")).scalar_one()
|
||||
assert (user.preferences or {}).get("language") == "hu"
|
||||
|
||||
|
||||
def test_an_access_request_records_the_language_it_was_made_in(db, client):
|
||||
assert (
|
||||
client.post(
|
||||
"/auth/request-access", json={"email": "asking@example.com", "lang": "hu"}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
|
||||
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
|
||||
assert inv.lang == "hu"
|
||||
|
||||
|
||||
def test_asking_again_from_another_language_updates_the_row(db, client):
|
||||
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "en"})
|
||||
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
|
||||
|
||||
db.expire_all()
|
||||
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
|
||||
# The newest thing they told us wins — and there is still exactly one row (S3's canonical upsert).
|
||||
assert inv.lang == "hu"
|
||||
assert db.execute(select(Invite)).scalars().all() == [inv]
|
||||
|
||||
|
||||
def test_an_unsupported_language_is_ignored_rather_than_stored(db, client):
|
||||
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "../../etc"})
|
||||
|
||||
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
|
||||
assert inv.lang is None
|
||||
|
||||
|
||||
# --- the mails themselves -------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _bodies(monkeypatch):
|
||||
"""Capture what `_send` would have put on the wire, rather than the arguments a caller passed —
|
||||
the language only matters if it reaches the body."""
|
||||
from app import email as email_mod
|
||||
|
||||
out: list[tuple[list[str], str, str]] = []
|
||||
monkeypatch.setattr(
|
||||
email_mod,
|
||||
"_send",
|
||||
lambda to, subject, body, reply_to=None: out.append((to, subject, body)) is None,
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def test_the_confirmation_mail_is_written_in_the_accounts_language(db, client, monkeypatch):
|
||||
from app import email as email_mod
|
||||
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
|
||||
sent = _bodies(monkeypatch)
|
||||
|
||||
client.post("/auth/register", json={"email": "hu@example.com", "password": _PW, "lang": "hu"})
|
||||
|
||||
(_, subject, body) = sent[0]
|
||||
assert subject == mailtext.MAIL_TEXT["verify_email.subject"]["hu"]
|
||||
assert "#verify=" in body
|
||||
|
||||
|
||||
def test_the_approval_mail_answers_in_the_language_the_request_was_made_in(
|
||||
db, client, make_user, as_user, monkeypatch
|
||||
):
|
||||
from app import email as email_mod
|
||||
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
|
||||
sent = _bodies(monkeypatch)
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
|
||||
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
|
||||
sent.clear() # drop the admin notification; this test is about the reply to the requester
|
||||
|
||||
r = as_user(admin).post(f"/api/admin/invites/{inv.id}/approve")
|
||||
|
||||
assert r.status_code == 200
|
||||
(to, subject, _) = sent[0]
|
||||
assert to == ["asking@example.com"]
|
||||
assert subject == mailtext.MAIL_TEXT["access_approved.subject"]["hu"]
|
||||
|
||||
|
||||
def test_a_denial_answers_in_that_language_too(db, client, make_user, as_user, monkeypatch):
|
||||
from app import email as email_mod
|
||||
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
|
||||
sent = _bodies(monkeypatch)
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
|
||||
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
|
||||
sent.clear()
|
||||
|
||||
as_user(admin).post(f"/api/admin/invites/{inv.id}/deny")
|
||||
|
||||
assert sent[0][1] == mailtext.MAIL_TEXT["access_denied.subject"]["hu"]
|
||||
|
||||
|
||||
def test_the_admins_pick_decides_an_invitation_and_is_remembered(
|
||||
db, make_user, as_user, monkeypatch
|
||||
):
|
||||
from app import email as email_mod
|
||||
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
|
||||
sent = _bodies(monkeypatch)
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
# Hand-whitelisted: nobody has ever told us anything about this mailbox, which is exactly why
|
||||
# the send button asks.
|
||||
inv = Invite(email="stranger@example.com", status="approved")
|
||||
db.add(inv)
|
||||
db.commit()
|
||||
|
||||
r = as_user(admin).post(f"/api/admin/invites/{inv.id}/invite", json={"lang": "hu"})
|
||||
|
||||
assert r.status_code == 200
|
||||
assert sent[0][1] == mailtext.MAIL_TEXT["access_invitation.subject"]["hu"]
|
||||
db.expire_all()
|
||||
# Recorded, so a later mail to this address matches the one the admin already sent.
|
||||
assert db.get(Invite, inv.id).lang == "hu"
|
||||
|
||||
|
||||
def test_an_invitation_without_a_pick_still_goes_out(db, make_user, as_user, monkeypatch):
|
||||
from app import email as email_mod
|
||||
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
|
||||
sent = _bodies(monkeypatch)
|
||||
admin = make_user(email="admin@example.com", role="admin")
|
||||
inv = Invite(email="stranger@example.com", status="approved")
|
||||
db.add(inv)
|
||||
db.commit()
|
||||
|
||||
r = as_user(admin).post(f"/api/admin/invites/{inv.id}/invite")
|
||||
|
||||
assert r.status_code == 200
|
||||
assert sent[0][1] == mailtext.MAIL_TEXT["access_invitation.subject"]["en"]
|
||||
db.expire_all()
|
||||
# Nothing was picked, so nothing is remembered: the column means "what this person told us",
|
||||
# not "what we last defaulted to".
|
||||
assert db.get(Invite, inv.id).lang is None
|
||||
|
||||
|
||||
def test_admins_are_notified_in_their_own_languages(db, make_user):
|
||||
make_user(email="hu.admin@example.com", role="admin", preferences={"language": "hu"})
|
||||
make_user(email="en.admin@example.com", role="admin", preferences={"language": "en"})
|
||||
|
||||
groups = auth_routes.admin_notify_groups(db)
|
||||
|
||||
# One mail per language, not one blast that has to be wrong for somebody.
|
||||
assert groups == {"hu": ["hu.admin@example.com"], "en": ["en.admin@example.com"]}
|
||||
@@ -0,0 +1,58 @@
|
||||
"""/api/me carries whether this instance can actually send email.
|
||||
|
||||
It drives the operator warnings on the Configuration and Access-requests screens. Admin-only,
|
||||
because it is an operator concern and a regular user has nothing to do with it.
|
||||
"""
|
||||
|
||||
from app import email as email_mod
|
||||
|
||||
_PW = "correct horse battery"
|
||||
|
||||
|
||||
def _signed_in(client, make_user, **kw):
|
||||
"""/api/me resolves its user through optional_current_user, which the as_user override does not
|
||||
cover — so sign in for real and read the bootstrap payload the SPA actually gets."""
|
||||
user = make_user(password=_PW, **kw)
|
||||
assert (
|
||||
client.post("/auth/password-login", json={"email": user.email, "password": _PW}).status_code
|
||||
== 200
|
||||
)
|
||||
r = client.get("/api/me")
|
||||
assert r.status_code == 200 and r.json()["authenticated"] is True
|
||||
return r.json()
|
||||
|
||||
|
||||
def test_an_admin_is_told_when_email_is_not_configured(client, make_user, monkeypatch):
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: False)
|
||||
|
||||
me = _signed_in(client, make_user, email="admin@example.com", role="admin")
|
||||
|
||||
assert me["email_configured"] is False
|
||||
|
||||
|
||||
def test_an_admin_sees_nothing_wrong_when_email_works(client, make_user, monkeypatch):
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
|
||||
|
||||
me = _signed_in(client, make_user, email="admin@example.com", role="admin")
|
||||
|
||||
assert me["email_configured"] is True
|
||||
|
||||
|
||||
def test_a_regular_user_is_never_warned_about_the_mail_setup(client, make_user, monkeypatch):
|
||||
# Not their problem, and the flag must not become a way to probe the instance's configuration.
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: False)
|
||||
|
||||
me = _signed_in(client, make_user, email="plain@example.com")
|
||||
|
||||
assert me["email_configured"] is True
|
||||
|
||||
|
||||
def test_the_flag_reuses_the_request_session(client, make_user, monkeypatch):
|
||||
# /api/me is polled once a minute per open tab; reading the SMTP settings decrypts a Fernet
|
||||
# value, so it must not also check out a second pooled connection each time.
|
||||
seen: list[object] = []
|
||||
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: seen.append(db) or True)
|
||||
|
||||
_signed_in(client, make_user, email="admin@example.com", role="admin")
|
||||
|
||||
assert seen and seen[0] is not None
|
||||
@@ -0,0 +1,148 @@
|
||||
"""A row that is gone answers with a reason the bilingual UI can translate (deep-review round 3).
|
||||
|
||||
The 404 a user actually meets is the one for something deleted in another tab or on another device.
|
||||
`owned_or_404` / `get_or_404` sent it as a bare English string ("Unknown view", "Not found"), and the
|
||||
Hungarian UI toasted it verbatim. The structured-404 fix had gone into one ad-hoc helper instead of
|
||||
the shared one, so every other owner check kept the English. Needs the DB.
|
||||
"""
|
||||
|
||||
from app.auth import require_write_scope
|
||||
from app.models import DownloadJob, MediaAsset
|
||||
|
||||
|
||||
def test_a_gone_view_tag_or_pin_says_what_is_gone(db, make_user, as_user):
|
||||
client = as_user(make_user())
|
||||
for method, url, what in [
|
||||
("patch", "/api/saved-views/999999", "view"),
|
||||
("delete", "/api/saved-views/999999", "view"),
|
||||
("delete", "/api/tags/999999", "tag"),
|
||||
("delete", "/api/discover/pins/999999", "pin"),
|
||||
]:
|
||||
r = getattr(client, method)(url, **({"json": {"name": "x"}} if method == "patch" else {}))
|
||||
assert r.status_code == 404, (url, r.text)
|
||||
assert r.json()["detail"] == {"code": "not_found", "reason": what}, url
|
||||
|
||||
|
||||
def test_a_gone_channel_video_or_user_says_what_is_gone(db, make_user, as_user):
|
||||
"""The siblings that still hand-wrote `db.get(...) is None → 404 "Unknown …"` (deep-review
|
||||
round 4): the same gone row, but the Hungarian UI showed the English."""
|
||||
admin = make_user(role="admin")
|
||||
client = as_user(admin)
|
||||
for method, url, body, what in [
|
||||
("post", "/api/channels/UCgone000000000000000000/block", None, "channel"),
|
||||
("post", "/api/videos/gone0000000/state", {"status": "watched"}, "video"),
|
||||
(
|
||||
"post",
|
||||
"/api/videos/gone0000000/progress",
|
||||
{"position_seconds": 60, "duration_seconds": 600},
|
||||
"video",
|
||||
),
|
||||
("put", "/api/admin/downloads/quota/999999", {"max_bytes": 1}, "user"),
|
||||
]:
|
||||
r = getattr(client, method)(url, **({"json": body} if body is not None else {}))
|
||||
assert r.status_code == 404, (url, r.text)
|
||||
assert r.json()["detail"] == {"code": "not_found", "reason": what}, url
|
||||
|
||||
|
||||
def test_every_gone_channel_route_says_it_the_same_way(db, make_user, as_user, override_dep):
|
||||
"""The round-4 sweep converted ONE of the five `db.get(Channel, …) is None -> 404 "Unknown
|
||||
channel"` sites (block), and shipped the HU/EN string with it — so the Hungarian UI said it in
|
||||
Hungarian on the Block button and in English on the four beside it (deep-review round 5). The
|
||||
asymmetric fix is the smell this repo bans, so all five are pinned together.
|
||||
|
||||
Literally all five: round 6 found this loop issuing three requests under a docstring claiming
|
||||
five, with explore silently absent and subscribe excused as "covered by the tags route" — which
|
||||
reaches `_user_subscription`, a different site answering `reason: subscription`. Reverting
|
||||
either of those two to its bare English string left the whole lane green."""
|
||||
admin = make_user(role="admin")
|
||||
client = as_user(admin)
|
||||
# Subscribe's own gate (the optional YouTube write scope) sits in FRONT of the lookup, so a
|
||||
# plain account 403s before reaching it. The gate is not what is under test here.
|
||||
override_dep(require_write_scope, lambda: admin)
|
||||
gone = "UCgone000000000000000000"
|
||||
for method, url in [
|
||||
("get", f"/api/channels/{gone}"),
|
||||
("post", f"/api/channels/{gone}/explore"),
|
||||
("post", f"/api/channels/{gone}/reset-backfill"),
|
||||
("post", f"/api/channels/{gone}/subscribe"),
|
||||
("post", f"/api/channels/{gone}/block"),
|
||||
]:
|
||||
r = getattr(client, method)(url)
|
||||
assert r.status_code == 404, (url, r.text)
|
||||
assert r.json()["detail"] == {"code": "not_found", "reason": "channel"}, url
|
||||
|
||||
|
||||
def test_a_gone_subscription_recipient_or_job_says_what_is_gone(db, make_user, as_user):
|
||||
"""The three bare-English 404s the same sweep left behind, which `notifyActionError` printed
|
||||
verbatim into an otherwise Hungarian UI: "Not subscribed to this channel", "No user with that
|
||||
email.", "Unknown job".
|
||||
|
||||
All three, with a request each. Round 6 found this test naming three and issuing two: the
|
||||
recipient site had no test in the whole suite (nor did the `errors.notFound.recipient` keys
|
||||
added beside it), and `run_job_now` rode on its PATCH sibling's coverage."""
|
||||
admin = make_user(role="admin")
|
||||
client = as_user(admin)
|
||||
r = client.post("/api/channels/UCgone000000000000000000/tags", json={"tag_id": 1})
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.json()["detail"] == {"code": "not_found", "reason": "subscription"}
|
||||
for method, url in [
|
||||
("patch", "/api/admin/scheduler/jobs/no_such_job"),
|
||||
("post", "/api/admin/scheduler/jobs/no_such_job/run"),
|
||||
]:
|
||||
kw = {"json": {"interval_minutes": 30}} if method == "patch" else {}
|
||||
r = getattr(client, method)(url, **kw)
|
||||
assert r.status_code == 404, (url, r.text)
|
||||
assert r.json()["detail"] == {"code": "not_found", "reason": "job"}, url
|
||||
# Sharing needs a finished download of one's own to share; the recipient lookup is the next check.
|
||||
asset = MediaAsset(
|
||||
source_kind="url", source_ref="share-404", format_sig="sig", status="ready", ref_count=1
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
job = DownloadJob(
|
||||
user_id=admin.id,
|
||||
asset_id=asset.id,
|
||||
source_kind="url",
|
||||
source_ref="share-404",
|
||||
status="done",
|
||||
queue_pos=1,
|
||||
profile_snapshot={},
|
||||
)
|
||||
db.add(job)
|
||||
db.commit()
|
||||
r = client.post(f"/api/downloads/{job.id}/share", json={"email": "nobody@example.com"})
|
||||
assert r.status_code == 404, r.text
|
||||
assert r.json()["detail"] == {"code": "not_found", "reason": "recipient"}
|
||||
|
||||
|
||||
def test_a_nul_in_a_request_string_is_a_422_not_a_500(db, make_user, as_user):
|
||||
"""Postgres cannot hold NUL in text, so the driver refuses the query outright: a DataError, i.e.
|
||||
a 500 with a traceback, wherever a `%00` or an escaped NUL reached a comparison (deep-review
|
||||
round 4 — round 3 had guarded only the by-rating_key lookup). One app-wide handler now answers
|
||||
all of them."""
|
||||
client = as_user(make_user())
|
||||
r = client.get("/api/plex/playlists?contains=%00")
|
||||
assert r.status_code == 422, r.text
|
||||
assert r.json()["detail"] == {"code": "invalid_input", "reason": "nul"}
|
||||
assert client.get("/api/plex/playlists?contains_group=1,%00").status_code == 422
|
||||
|
||||
|
||||
def test_any_other_data_error_is_still_a_500():
|
||||
"""The other half of the app-wide handler, and the half a refactor can delete in silence: it is
|
||||
registered for EVERY DataError on every route and decides on a substring of the driver's message,
|
||||
so only a NUL may become a 422. Without this, simplifying it to an unconditional 422 — it reads
|
||||
like dead defensive code — would answer a numeric overflow or an over-long ingested title with
|
||||
"your input was bad", losing the traceback that names the real fault (deep-review round 5)."""
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.exc import DataError
|
||||
|
||||
from app.main import _nul_in_text
|
||||
|
||||
overflow = DataError("SELECT 1", {}, Exception("numeric field overflow"))
|
||||
with pytest.raises(DataError):
|
||||
asyncio.run(_nul_in_text(None, overflow)) # re-raised: it stays the 500 it always was
|
||||
|
||||
nul = DataError("SELECT 1", {}, Exception("invalid byte sequence: NUL (0x00)"))
|
||||
assert asyncio.run(_nul_in_text(None, nul)).status_code == 422
|
||||
@@ -0,0 +1,244 @@
|
||||
"""E10 S2 — the one-step "move this video to another list" endpoint, and the `updated_at` touch
|
||||
the rail's "recently changed" sort reads.
|
||||
|
||||
Two properties are load-bearing and neither is visible from the happy path alone:
|
||||
|
||||
1. **The move is one transaction.** The client-side shape it replaces (add to the target, then
|
||||
remove from the source) leaves the video in BOTH lists when the second call fails. So the
|
||||
tests assert the *pair* of lists after every call, never just the destination.
|
||||
2. **`updated_at` tracks item edits.** `UpdatedAtMixin.onupdate` only fires on an UPDATE of the
|
||||
playlist row, and add/remove/reorder/move write `playlist_items` — so the timestamp is bumped
|
||||
explicitly, and a regression there would silently degrade the sort to "renames only".
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from app.models import Channel, Playlist, PlaylistItem, Video
|
||||
|
||||
|
||||
def _seed_videos(db, ids):
|
||||
db.add(Channel(id="UCtest", title="Test"))
|
||||
now = datetime.now(timezone.utc)
|
||||
for v in ids:
|
||||
db.add(Video(id=v, channel_id="UCtest", published_at=now))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _playlist(db, user, name, video_ids, **kw):
|
||||
pl = Playlist(user_id=user.id, name=name, **kw)
|
||||
db.add(pl)
|
||||
db.flush()
|
||||
for i, vid in enumerate(video_ids):
|
||||
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=i))
|
||||
db.commit()
|
||||
return pl
|
||||
|
||||
|
||||
def _ids(db, pl_id):
|
||||
return [
|
||||
it.video_id
|
||||
for it in db.execute(
|
||||
PlaylistItem.__table__.select()
|
||||
.where(PlaylistItem.playlist_id == pl_id)
|
||||
.order_by(PlaylistItem.position)
|
||||
).all()
|
||||
]
|
||||
|
||||
|
||||
def _move(client, src, video_id, dst):
|
||||
return client.post(
|
||||
f"/api/playlists/{src}/items/{video_id}/move", json={"target_playlist_id": dst}
|
||||
)
|
||||
|
||||
|
||||
def test_move_leaves_the_video_in_exactly_one_list(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["v1", "v2"])
|
||||
src = _playlist(db, user, "Source", ["v1", "v2"])
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
|
||||
r = _move(as_user(user), src.id, "v1", dst.id)
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {
|
||||
"ok": True,
|
||||
"from_playlist_id": src.id,
|
||||
"to_playlist_id": dst.id,
|
||||
"video_id": "v1",
|
||||
}
|
||||
db.expire_all()
|
||||
assert _ids(db, src.id) == ["v2"]
|
||||
assert _ids(db, dst.id) == ["v1"]
|
||||
|
||||
|
||||
def test_move_appends_to_the_end_of_the_target(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["v1", "v2", "v3"])
|
||||
src = _playlist(db, user, "Source", ["v3"])
|
||||
dst = _playlist(db, user, "Target", ["v1", "v2"])
|
||||
|
||||
assert _move(as_user(user), src.id, "v3", dst.id).status_code == 200
|
||||
db.expire_all()
|
||||
assert _ids(db, dst.id) == ["v1", "v2", "v3"]
|
||||
|
||||
|
||||
def test_move_into_a_list_that_already_has_it_is_a_removal_not_a_duplicate(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["v1"])
|
||||
src = _playlist(db, user, "Source", ["v1"])
|
||||
dst = _playlist(db, user, "Target", ["v1"])
|
||||
|
||||
assert _move(as_user(user), src.id, "v1", dst.id).status_code == 200
|
||||
db.expire_all()
|
||||
assert _ids(db, src.id) == []
|
||||
assert _ids(db, dst.id) == ["v1"] # exactly one row, not two
|
||||
|
||||
|
||||
def test_move_to_the_same_list_is_refused(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["v1"])
|
||||
src = _playlist(db, user, "Source", ["v1"])
|
||||
|
||||
r = _move(as_user(user), src.id, "v1", src.id)
|
||||
assert r.status_code == 400
|
||||
assert r.json()["detail"] == {"code": "playlist_move", "reason": "same_playlist"}
|
||||
db.expire_all()
|
||||
assert _ids(db, src.id) == ["v1"]
|
||||
|
||||
|
||||
def test_move_of_a_video_that_left_the_source_reports_it(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_seed_videos(db, ["v1"])
|
||||
src = _playlist(db, user, "Source", [])
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
|
||||
r = _move(as_user(user), src.id, "v1", dst.id)
|
||||
assert r.status_code == 404
|
||||
assert r.json()["detail"] == {"code": "playlist_move", "reason": "not_in_playlist"}
|
||||
|
||||
|
||||
def test_move_into_someone_elses_list_404s_and_changes_nothing(db, make_user, as_user):
|
||||
user, other = make_user(), make_user()
|
||||
_seed_videos(db, ["v1"])
|
||||
src = _playlist(db, user, "Mine", ["v1"])
|
||||
theirs = _playlist(db, other, "Theirs", [])
|
||||
|
||||
assert _move(as_user(user), src.id, "v1", theirs.id).status_code == 404
|
||||
db.expire_all()
|
||||
assert _ids(db, src.id) == ["v1"]
|
||||
assert _ids(db, theirs.id) == []
|
||||
|
||||
|
||||
def test_a_list_that_is_not_yours_answers_with_a_reason_the_ui_can_translate(
|
||||
db, make_user, as_user
|
||||
):
|
||||
# The one 404 a user meets in practice (the list went away in another tab). It used to be a bare
|
||||
# English "Unknown playlist", which the Hungarian UI then showed verbatim.
|
||||
user, other = make_user(), make_user()
|
||||
_seed_videos(db, ["v1"])
|
||||
theirs = _playlist(db, other, "Theirs", ["v1"])
|
||||
|
||||
r = as_user(user).delete(f"/api/playlists/{theirs.id}/items/v1")
|
||||
assert r.status_code == 404
|
||||
assert r.json()["detail"] == {"code": "not_found", "reason": "playlist"}
|
||||
db.expire_all()
|
||||
assert _ids(db, theirs.id) == ["v1"]
|
||||
|
||||
|
||||
def test_a_move_into_a_list_that_is_gone_names_the_target_not_the_list_on_screen(
|
||||
db, make_user, as_user
|
||||
):
|
||||
# Both lists 404'd with the same reason, which the UI reads as "THIS playlist doesn't exist any
|
||||
# more" — said about the source the reader is looking at, which is still there.
|
||||
user = make_user()
|
||||
_seed_videos(db, ["v1"])
|
||||
src = _playlist(db, user, "Mine", ["v1"])
|
||||
gone = _playlist(db, user, "Gone", [])
|
||||
gone_id = gone.id
|
||||
db.delete(gone)
|
||||
db.commit()
|
||||
|
||||
r = _move(as_user(user), src.id, "v1", gone_id)
|
||||
assert r.status_code == 404
|
||||
assert r.json()["detail"] == {"code": "not_found", "reason": "move_target"}
|
||||
db.expire_all()
|
||||
assert _ids(db, src.id) == ["v1"]
|
||||
|
||||
|
||||
def test_a_refused_playlist_action_answers_with_a_reason_the_ui_can_translate(
|
||||
db, make_user, as_user
|
||||
):
|
||||
# The rest of the router's refusals were bare English too ("The Watch later list can't be
|
||||
# deleted.", "YouTube delete failed.", ...), shown verbatim under the Hungarian UI.
|
||||
user = make_user()
|
||||
client = as_user(user)
|
||||
wl = client.post("/api/playlists/watch-later", json={"video_id": "nope"})
|
||||
assert wl.status_code == 404
|
||||
assert wl.json()["detail"] == {"code": "not_found", "reason": "video"}
|
||||
_seed_videos(db, ["v1"])
|
||||
saved = client.post("/api/playlists/watch-later", json={"video_id": "v1"})
|
||||
wl_id = saved.json()["playlist_id"]
|
||||
r = client.delete(f"/api/playlists/{wl_id}")
|
||||
assert r.status_code == 400
|
||||
assert r.json()["detail"] == {"code": "playlist", "reason": "watch_later_undeletable"}
|
||||
|
||||
|
||||
def test_move_marks_both_linked_sides_dirty(db, make_user, as_user):
|
||||
"""A move edits two YouTube mirrors at once — the source lost an item, the target gained one.
|
||||
Marking only the source is the asymmetric fix this repo keeps re-learning."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["v1"])
|
||||
src = _playlist(db, user, "Src", ["v1"], yt_playlist_id="PLsrc", synced_video_ids='["v1"]')
|
||||
dst = _playlist(db, user, "Dst", [], yt_playlist_id="PLdst", synced_video_ids="[]")
|
||||
# Baseline: both are in sync with YouTube before the move.
|
||||
src.dirty = False
|
||||
dst.dirty = False
|
||||
db.commit()
|
||||
|
||||
assert _move(as_user(user), src.id, "v1", dst.id).status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, src.id).dirty is True
|
||||
assert db.get(Playlist, dst.id).dirty is True
|
||||
|
||||
|
||||
def test_item_edits_bump_updated_at(db, make_user, as_user):
|
||||
"""The rail's "recently changed first" sort reads this. Adding, moving and reordering all
|
||||
write `playlist_items` only, so nothing here is covered by the ORM's `onupdate`."""
|
||||
user = make_user()
|
||||
_seed_videos(db, ["v1", "v2"])
|
||||
src = _playlist(db, user, "Source", ["v1", "v2"])
|
||||
dst = _playlist(db, user, "Target", [])
|
||||
old = datetime.now(timezone.utc) - timedelta(days=3)
|
||||
src.updated_at = old
|
||||
dst.updated_at = old
|
||||
db.commit()
|
||||
client = as_user(user)
|
||||
|
||||
assert client.post(f"/api/playlists/{dst.id}/items", json={"video_id": "v1"}).status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, dst.id).updated_at > old
|
||||
|
||||
dst.updated_at = old
|
||||
db.commit()
|
||||
assert (
|
||||
client.put(f"/api/playlists/{src.id}/order", json={"video_ids": ["v2", "v1"]}).status_code
|
||||
== 200
|
||||
)
|
||||
db.expire_all()
|
||||
assert db.get(Playlist, src.id).updated_at > old
|
||||
|
||||
src.updated_at = old
|
||||
dst.updated_at = old
|
||||
db.commit()
|
||||
assert _move(client, src.id, "v2", dst.id).status_code == 200
|
||||
db.expire_all()
|
||||
# BOTH sides of a move are "recently changed" — the target is where the reader will look.
|
||||
assert db.get(Playlist, src.id).updated_at > old
|
||||
assert db.get(Playlist, dst.id).updated_at > old
|
||||
|
||||
|
||||
def test_the_list_endpoint_publishes_updated_at(db, make_user, as_user):
|
||||
user = make_user()
|
||||
_playlist(db, user, "Solo", [])
|
||||
rows = as_user(user).get("/api/playlists").json()
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["updated_at"] # the rail cannot sort by a field the API never sends
|
||||
@@ -0,0 +1,83 @@
|
||||
""" "Clear resume point" has to reach Plex — the case the status-only guard skipped.
|
||||
|
||||
`POST /api/plex/item/{rk}/state {"status": "new"}` backs the *Clear resume point* / *Mark unwatched*
|
||||
buttons. It deletes the local row and mirrors the un-mark to Plex — but it used to decide whether to
|
||||
push from the PREVIOUS STATUS alone (`prev_status != "new"`), and an in-progress item's row is
|
||||
`status="new"` carrying a position: that is how the importer maps a bare `viewOffset`
|
||||
(`_plex_watch_to_state`). So for the one case the button exists for, nothing was ever sent to Plex,
|
||||
Plex kept the offset, and the next pull imported it straight back — the user cleared *Protector*
|
||||
twice and it returned to Continue watching at 92s both times (2026-09-09).
|
||||
|
||||
One `unscrobble` clears the view count AND the offset (measured against the live server), so the
|
||||
push itself needed no change; what needed fixing is WHEN it is scheduled.
|
||||
|
||||
DB lane (`db` fixture) — skipped when no Postgres is reachable.
|
||||
"""
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import app.state as state
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.models import PlexItem, PlexLibrary, PlexState, User
|
||||
from app.plex import watch_sync as plex_watch
|
||||
|
||||
|
||||
def _seed(db, *, status: str, position: int):
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
it = PlexItem(
|
||||
rating_key="98614", library_id=lib.id, kind="movie", title="Protector", duration_s=5495
|
||||
)
|
||||
db.add(it)
|
||||
user = User(email="clear@example.com")
|
||||
db.add(user)
|
||||
db.flush()
|
||||
db.add(PlexState(user_id=user.id, item_id=it.id, status=status, position_seconds=position))
|
||||
db.commit()
|
||||
return it, user
|
||||
|
||||
|
||||
def _clear(db, user, it, monkeypatch) -> list[tuple]:
|
||||
"""Drive the endpoint with the Plex link satisfied and the background push recorded."""
|
||||
pushes: list[tuple] = []
|
||||
monkeypatch.setattr(state, "_configured_cache", True) # pass the instance-setup request gate
|
||||
monkeypatch.setattr(plex_watch, "link_for_push", lambda db_, uid: SimpleNamespace(id=1))
|
||||
monkeypatch.setattr(
|
||||
plex_watch,
|
||||
"push_state_to_plex",
|
||||
lambda user_id, item_id, rating_key, action, *rest: pushes.append((rating_key, action)),
|
||||
)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
app.dependency_overrides[current_user] = lambda: user
|
||||
try:
|
||||
resp = TestClient(app).post(f"/api/plex/item/{it.rating_key}/state", json={"status": "new"})
|
||||
assert resp.status_code == 200, resp.text
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
return pushes
|
||||
|
||||
|
||||
def test_clearing_an_in_progress_item_tells_plex(db, monkeypatch):
|
||||
# The regression: status "new" + a position IS the in-progress shape, and it must push.
|
||||
it, user = _seed(db, status="new", position=92)
|
||||
assert _clear(db, user, it, monkeypatch) == [("98614", "unwatched")]
|
||||
assert db.query(PlexState).count() == 0
|
||||
|
||||
|
||||
def test_clearing_a_watched_item_still_tells_plex(db, monkeypatch):
|
||||
# The case the old guard did cover — kept, so the fix cannot narrow into a new hole.
|
||||
it, user = _seed(db, status="watched", position=0)
|
||||
assert _clear(db, user, it, monkeypatch) == [("98614", "unwatched")]
|
||||
|
||||
|
||||
def test_unhiding_touches_plex_not_at_all(db, monkeypatch):
|
||||
# `hidden` is a Siftlode-only concept "with no Plex analogue, so it never touches Plex" — the
|
||||
# endpoint's own docstring. The old status-only guard pushed here anyway, un-marking an item on
|
||||
# Plex that the user had merely hidden.
|
||||
it, user = _seed(db, status="hidden", position=0)
|
||||
assert _clear(db, user, it, monkeypatch) == []
|
||||
@@ -0,0 +1,291 @@
|
||||
"""The Collection info page's member ordering (`GET /api/plex/collections/{rk}?sort=&sort_dir=`) and
|
||||
the `keys=` lookup that gives the sidebar token / filter chip a title instead of a raw rating_key.
|
||||
|
||||
Reported from daily use: "Collectionben nem működik a rendezés". The browse grid sorts fine — the
|
||||
collection page had no control at all and a hardcoded year-ascending order. Adding the control means
|
||||
the SAME vocabulary must actually work here, hence the pin against `PLEX_SORT_KEYS`.
|
||||
|
||||
Pure-logic tests run everywhere; the endpoint tests use the `db` fixture (DB lane).
|
||||
"""
|
||||
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import literal_column
|
||||
|
||||
import app.state as state
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.models import PlexCollection, PlexItem, PlexLibrary, PlexShow, PlexState, User
|
||||
from app.plex.query import PLEX_SORT_KEYS
|
||||
from app.routes.plex.browse import sort_column
|
||||
from app.routes.plex.collections import (
|
||||
DEFAULT_MEMBER_SORT,
|
||||
MEMBER_SORT_KEYS,
|
||||
Member,
|
||||
sort_members,
|
||||
)
|
||||
|
||||
NOW = datetime(2024, 6, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _m(title: str, **over) -> Member:
|
||||
base = dict(
|
||||
kind="movie",
|
||||
obj=None,
|
||||
title=title,
|
||||
year=None,
|
||||
rating=None,
|
||||
added=None,
|
||||
release=None,
|
||||
duration=None,
|
||||
last_watched=None,
|
||||
)
|
||||
base.update(over)
|
||||
return Member(**base) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _titles(members: list[Member]) -> list[str]:
|
||||
return [m.title for m in members]
|
||||
|
||||
|
||||
# --- the vocabulary is ONE vocabulary ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_every_sort_key_is_orderable_on_a_collection():
|
||||
"""The collection page shows the same Sort control as the browse grid, so a key it cannot order
|
||||
would be an option that silently does nothing."""
|
||||
assert set(MEMBER_SORT_KEYS) == set(PLEX_SORT_KEYS)
|
||||
|
||||
|
||||
def test_every_sort_key_is_orderable_on_the_browse_grid():
|
||||
"""The other half of the same pin: browse falls back to `added_at` for an unknown key, so a key
|
||||
missing from its map would order by date while the control claimed otherwise."""
|
||||
|
||||
class _Cols:
|
||||
title = literal_column("title")
|
||||
year = literal_column("year")
|
||||
rating = literal_column("rating")
|
||||
duration_s = literal_column("duration_s")
|
||||
rel = literal_column("rel")
|
||||
last_watched = literal_column("last_watched")
|
||||
added_at = literal_column("added_at")
|
||||
|
||||
class _U:
|
||||
c = _Cols()
|
||||
|
||||
u = _U()
|
||||
fallback = u.c.added_at
|
||||
unhandled = [k for k in PLEX_SORT_KEYS if k != "added" and sort_column(u, k) is fallback]
|
||||
assert unhandled == []
|
||||
# …and the fallback itself is still what an unknown key gets.
|
||||
assert sort_column(u, "no-such-key") is fallback
|
||||
|
||||
|
||||
# --- ordering behaviour -----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_orders_by_the_requested_key_in_both_directions():
|
||||
members = [_m("b", year=2006), _m("c", year=2017), _m("a", year=2003)]
|
||||
assert _titles(sort_members(members, "year", "asc")) == ["a", "b", "c"]
|
||||
assert _titles(sort_members(members, "year", "desc")) == ["c", "b", "a"]
|
||||
|
||||
|
||||
def test_members_without_a_value_sink_to_the_end_in_both_directions():
|
||||
"""A movie with no rating is not "the worst rated" — nullslast, like the browse grid."""
|
||||
members = [_m("rated-low", rating=6.5), _m("unrated"), _m("rated-high", rating=8.1)]
|
||||
assert _titles(sort_members(members, "rating", "desc")) == [
|
||||
"rated-high",
|
||||
"rated-low",
|
||||
"unrated",
|
||||
]
|
||||
assert _titles(sort_members(members, "rating", "asc")) == ["rated-low", "rated-high", "unrated"]
|
||||
|
||||
|
||||
def test_ties_stay_title_ascending_when_the_direction_flips():
|
||||
"""Flipping the direction must reverse the RANKING, not the alphabet inside a tie."""
|
||||
members = [_m("zulu", year=2003), _m("alpha", year=2003), _m("mike", year=2017)]
|
||||
assert _titles(sort_members(members, "year", "asc")) == ["alpha", "zulu", "mike"]
|
||||
assert _titles(sort_members(members, "year", "desc")) == ["mike", "alpha", "zulu"]
|
||||
|
||||
|
||||
def test_an_unknown_key_falls_back_to_the_pages_default():
|
||||
members = [_m("b", year=2006), _m("a", year=2017)]
|
||||
assert _titles(sort_members(members, "no-such-key", "asc")) == _titles(
|
||||
sort_members(members, DEFAULT_MEMBER_SORT, "asc")
|
||||
)
|
||||
|
||||
|
||||
def test_a_show_has_no_duration_and_therefore_sinks_on_that_key():
|
||||
members = [
|
||||
_m("short movie", duration=3600),
|
||||
_m("a show", kind="show", duration=None),
|
||||
_m("long movie", duration=9000),
|
||||
]
|
||||
assert _titles(sort_members(members, "duration", "desc")) == [
|
||||
"long movie",
|
||||
"short movie",
|
||||
"a show",
|
||||
]
|
||||
|
||||
|
||||
# --- the endpoint (DB lane) -------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed(db):
|
||||
lib_m = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
lib_s = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
|
||||
db.add_all([lib_m, lib_s])
|
||||
db.flush()
|
||||
|
||||
col = PlexCollection(rating_key="col1", library_id=lib_m.id, title="Franchise", child_count=3)
|
||||
other = PlexCollection(rating_key="col2", library_id=lib_m.id, title="Elsewhere", child_count=1)
|
||||
db.add_all([col, other])
|
||||
db.flush()
|
||||
|
||||
# Titles deliberately alphabetical-opposite to the years, so a year sort can't accidentally pass
|
||||
# by falling through to the title tiebreak.
|
||||
old = PlexItem(
|
||||
rating_key="mo_old",
|
||||
library_id=lib_m.id,
|
||||
kind="movie",
|
||||
title="Zulu",
|
||||
year=2003,
|
||||
rating=8.1,
|
||||
duration_s=8000,
|
||||
added_at=NOW - timedelta(days=1),
|
||||
originally_available_at=date(2003, 7, 9),
|
||||
collection_keys=["col1"],
|
||||
)
|
||||
mid = PlexItem(
|
||||
rating_key="mo_mid",
|
||||
library_id=lib_m.id,
|
||||
kind="movie",
|
||||
title="Mike",
|
||||
year=2006,
|
||||
rating=7.4,
|
||||
duration_s=9000,
|
||||
added_at=NOW - timedelta(days=3),
|
||||
originally_available_at=date(2006, 7, 7),
|
||||
collection_keys=["col1"],
|
||||
)
|
||||
new = PlexItem(
|
||||
rating_key="mo_new",
|
||||
library_id=lib_m.id,
|
||||
kind="movie",
|
||||
title="Alpha",
|
||||
year=2017,
|
||||
rating=6.5,
|
||||
duration_s=7000,
|
||||
added_at=NOW,
|
||||
originally_available_at=date(2017, 5, 26),
|
||||
collection_keys=["col1"],
|
||||
)
|
||||
# A show member with a resumed episode: `lastwatched` must work on this leg too, not just movies.
|
||||
show = PlexShow(
|
||||
rating_key="sh1",
|
||||
library_id=lib_s.id,
|
||||
title="Series",
|
||||
year=2010,
|
||||
child_count=1,
|
||||
collection_keys=["col1"],
|
||||
)
|
||||
db.add_all([old, mid, new, show])
|
||||
db.flush()
|
||||
ep = PlexItem(
|
||||
rating_key="sh1_e1",
|
||||
library_id=lib_s.id,
|
||||
kind="episode",
|
||||
title="E1",
|
||||
show_id=show.id,
|
||||
season_number=1,
|
||||
episode_number=1,
|
||||
)
|
||||
db.add(ep)
|
||||
db.flush()
|
||||
|
||||
user = User(email="colsort@example.com")
|
||||
db.add(user)
|
||||
db.flush()
|
||||
db.add_all(
|
||||
[
|
||||
# The show's episode is the most recently touched thing in the collection…
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=ep.id,
|
||||
status="new",
|
||||
position_seconds=60,
|
||||
progress_updated_at=NOW,
|
||||
),
|
||||
# …and one movie was watched earlier.
|
||||
PlexState(
|
||||
user_id=user.id,
|
||||
item_id=old.id,
|
||||
status="watched",
|
||||
watched_at=NOW - timedelta(days=2),
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
return user
|
||||
|
||||
|
||||
def _get(db, user, monkeypatch, path):
|
||||
monkeypatch.setattr(state, "_configured_cache", True)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
app.dependency_overrides[current_user] = lambda: user
|
||||
try:
|
||||
return TestClient(app).get(path)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
def _order(r) -> list[str]:
|
||||
return [i["title"] for i in r.json()["items"]]
|
||||
|
||||
|
||||
def test_collection_detail_defaults_to_release_year_ascending(db, monkeypatch):
|
||||
"""The order the page had before it grew a control — a franchise reads in release order."""
|
||||
user = _seed(db)
|
||||
r = _get(db, user, monkeypatch, "/api/plex/collections/col1")
|
||||
assert r.status_code == 200
|
||||
assert _order(r) == ["Zulu", "Mike", "Series", "Alpha"]
|
||||
|
||||
|
||||
def test_collection_detail_honours_sort_and_direction(db, monkeypatch):
|
||||
user = _seed(db)
|
||||
r = _get(db, user, monkeypatch, "/api/plex/collections/col1?sort=year&sort_dir=desc")
|
||||
assert _order(r) == ["Alpha", "Series", "Mike", "Zulu"]
|
||||
r = _get(db, user, monkeypatch, "/api/plex/collections/col1?sort=title&sort_dir=asc")
|
||||
assert _order(r) == ["Alpha", "Mike", "Series", "Zulu"]
|
||||
r = _get(db, user, monkeypatch, "/api/plex/collections/col1?sort=rating&sort_dir=desc")
|
||||
# The show has no rating, so it sinks — it is not "rated 0".
|
||||
assert _order(r) == ["Zulu", "Mike", "Alpha", "Series"]
|
||||
|
||||
|
||||
def test_lastwatched_ranks_a_shows_episodes_alongside_the_movies(db, monkeypatch):
|
||||
"""The show leg needs the rollup's `resumed_at`; with only a status map this key would order
|
||||
every show as never-watched."""
|
||||
user = _seed(db)
|
||||
r = _get(db, user, monkeypatch, "/api/plex/collections/col1?sort=lastwatched&sort_dir=desc")
|
||||
# Series (episode resumed NOW) > Zulu (watched 2 days ago) > the two never-touched, title-asc.
|
||||
assert _order(r) == ["Series", "Zulu", "Alpha", "Mike"]
|
||||
|
||||
|
||||
def test_collections_list_resolves_titles_by_key(db, monkeypatch):
|
||||
"""What a shared Plex link needs: the URL carries bare rating_keys, so the sidebar token and the
|
||||
filter chip have nothing to show but `17045` until these are resolved."""
|
||||
user = _seed(db)
|
||||
r = _get(db, user, monkeypatch, "/api/plex/collections?keys=col1")
|
||||
assert r.status_code == 200
|
||||
assert [c["title"] for c in r.json()["collections"]] == ["Franchise"]
|
||||
|
||||
|
||||
def test_an_empty_keys_param_asks_for_nothing(db, monkeypatch):
|
||||
"""`keys=` with nothing in it must not fall through to "every collection" — that would be a
|
||||
200 KB answer to a question about zero ids."""
|
||||
user = _seed(db)
|
||||
r = _get(db, user, monkeypatch, "/api/plex/collections?keys=")
|
||||
assert r.status_code == 200
|
||||
assert r.json()["collections"] == []
|
||||
@@ -155,6 +155,7 @@ def test_create_collection_rejects_duplicate_name(db, monkeypatch):
|
||||
json={"library": "1", "title": "marvel", "item_rating_key": "m1"},
|
||||
)
|
||||
assert r.status_code == 409, r.text
|
||||
assert r.json()["detail"] == {"code": "collection", "reason": "name_taken"}
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
"""An HLS artifact request names the session it belongs to, and gets nothing else.
|
||||
|
||||
Sessions are keyed by rating_key ALONE, while a session's directory is per (key, offset, tag). So a
|
||||
seek-restart — or a second viewer of the same item — replaces the registered session, and the old
|
||||
viewer's next request for `seg_12.ts`, resolved by rating_key, was answered 200 with the NEW
|
||||
session's twelfth segment: bytes produced from a different `-ss` offset under `-copyts`, i.e. a
|
||||
timeline the client is not on. Wrong content with a success status is the one failure a player cannot
|
||||
notice, and the reclaim-on-404 added in round 4 could never fire on it (deep-review round 5).
|
||||
|
||||
No `db` fixture: the route reads nothing from the database (its own `Depends(get_db)` went with the
|
||||
removed auto-start, and `current_user` is overridden wholesale), so these must not sit in the DB
|
||||
lane — there they SKIP on a machine with no Postgres while pytest prints green (round 6).
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import app.state as state
|
||||
from app.auth import current_user
|
||||
from app.main import app
|
||||
from app.models import User
|
||||
from app.plex import stream as plex_stream
|
||||
|
||||
|
||||
class _Proc:
|
||||
def __init__(self, running: bool = False):
|
||||
self.running = running
|
||||
|
||||
def poll(self):
|
||||
return None if self.running else 0
|
||||
|
||||
def terminate(self):
|
||||
pass
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return 0
|
||||
|
||||
|
||||
class _Session:
|
||||
def __init__(self, session_id: str, directory, running: bool = False, audio_langs=None):
|
||||
self.id = session_id
|
||||
self.dir = directory
|
||||
self.proc = _Proc(running)
|
||||
self.last_access = time.time()
|
||||
# The real session derives both of these from the argv it spawned; the guard reads
|
||||
# `variants` to tell a name this session can produce from one only the OTHER argv branch
|
||||
# writes, so a stand-in without it would test a shape that cannot exist.
|
||||
self.variants = plex_stream.session_variants(audio_langs)
|
||||
self.entry = "master.m3u8" if self.variants else "index.m3u8"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sessions(monkeypatch):
|
||||
live: dict = {}
|
||||
monkeypatch.setattr(plex_stream, "_sessions", live)
|
||||
return live
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(monkeypatch):
|
||||
monkeypatch.setattr(state, "_configured_cache", True)
|
||||
user = User(email="hls@example.com")
|
||||
app.dependency_overrides[current_user] = lambda: user
|
||||
try:
|
||||
yield TestClient(app)
|
||||
finally:
|
||||
app.dependency_overrides.pop(current_user, None)
|
||||
|
||||
|
||||
def test_a_segment_is_served_from_the_session_the_url_names(client, tmp_path, sessions):
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
(d / "seg_11.ts").write_bytes(b"") # its predecessor, so seg_12 is the next one
|
||||
(d / "seg_12.ts").write_bytes(b"mine")
|
||||
sessions["m1"] = _Session("sid-a", d)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.content == b"mine"
|
||||
|
||||
|
||||
def test_a_segment_of_a_replaced_session_is_a_404_not_the_new_sessions_bytes(
|
||||
client, tmp_path, sessions
|
||||
):
|
||||
"""Viewer A is at 0:00; viewer B seeks to 25:00 and their session takes the registry slot. A's
|
||||
next segment must fail honestly — the player reclaims at the playhead — not arrive as B's."""
|
||||
theirs = tmp_path / "m1_1500_0_+0.00"
|
||||
theirs.mkdir()
|
||||
(theirs / "seg_11.ts").write_bytes(b"")
|
||||
(theirs / "seg_12.ts").write_bytes(b"theirs")
|
||||
sessions["m1"] = _Session("sid-b", theirs)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
|
||||
assert r.status_code == 404, r.text
|
||||
assert b"theirs" not in r.content
|
||||
|
||||
|
||||
def test_a_playlist_with_no_live_session_is_a_404_not_a_silent_restart_at_zero(
|
||||
client, monkeypatch, sessions
|
||||
):
|
||||
"""The entry playlist used to auto-start a session when none was registered — always at 0.0,
|
||||
silently throwing away the viewer's position. A 404 sends the player through its own reclaim,
|
||||
which restarts at the playhead."""
|
||||
started: list[float] = []
|
||||
monkeypatch.setattr(
|
||||
plex_stream,
|
||||
"start_session",
|
||||
lambda *a, **k: started.append(a[2] if len(a) > 2 else 0.0),
|
||||
)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-gone/index.m3u8")
|
||||
assert r.status_code == 404, r.text
|
||||
assert started == []
|
||||
|
||||
|
||||
def test_a_segment_that_will_never_arrive_fails_at_once(client, tmp_path, sessions):
|
||||
"""`stream_hls_file` is a SYNC route: it runs on the ~40-slot threadpool and holds a pooled DB
|
||||
connection while it waits. It used to poll out the whole 20 s for a name that could never appear
|
||||
— an ordinary seek past the produced region was enough to park a worker and a connection, and
|
||||
forty hopeless names parked the API (deep-review round 5). ffmpeg writes segments in order, so a
|
||||
segment whose predecessor is absent is not the next one, and that is knowable at once — even
|
||||
while ffmpeg is still running, which is the case `proc` alone cannot shorten."""
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
(d / "seg_0.ts").write_bytes(b"x") # the session is producing; seg_99999 is simply not next
|
||||
sessions["m1"] = _Session("sid-a", d, running=True)
|
||||
started = time.monotonic()
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_99999.ts")
|
||||
elapsed = time.monotonic() - started
|
||||
assert r.status_code == 404, r.text
|
||||
# The wait is 20 s; anything in that neighbourhood means the producer's exit was not consulted.
|
||||
assert elapsed < 2.0, f"waited {elapsed:.1f}s for a segment that can never arrive"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"name",
|
||||
[
|
||||
"seg_0_9000.ts", # the VIDEO rendition of a multi-audio session: variant 0, segment 9000
|
||||
"seg_a0_9000.ts", # the first audio rendition: `name:a0` from -var_stream_map
|
||||
],
|
||||
)
|
||||
def test_a_multi_audio_segment_beyond_the_region_fails_at_once_too(
|
||||
client, tmp_path, sessions, name
|
||||
):
|
||||
"""The round-5 guard read `seg_<seq>_<variant>.ts`, but ffmpeg substitutes `%v` FIRST — so the
|
||||
real names are `seg_<variant>_<seq>.ts` and every multi-audio request fell through to the full
|
||||
20 s wait: the video rendition because its variant "0" parsed as segment 0, the audio renditions
|
||||
(`a0`, `a1`) because they did not parse at all. Dual-audio is the common shape in this library,
|
||||
so the guard was inert exactly where it was needed (round 6)."""
|
||||
d = tmp_path / "m1_0_multi_+0.00"
|
||||
d.mkdir()
|
||||
(d / "seg_0_0.ts").write_bytes(b"x")
|
||||
(d / "seg_a0_0.ts").write_bytes(b"x")
|
||||
sessions["m1"] = _Session("sid-a", d, running=True, audio_langs=["hu", "en"])
|
||||
started = time.monotonic()
|
||||
r = client.get(f"/api/plex/stream/m1/hls/sid-a/{name}")
|
||||
elapsed = time.monotonic() - started
|
||||
assert r.status_code == 404, r.text
|
||||
assert elapsed < 2.0, f"waited {elapsed:.1f}s for {name}"
|
||||
|
||||
|
||||
def test_a_multi_audio_segment_whose_predecessor_exists_is_served(client, tmp_path, sessions):
|
||||
"""The other half of the same rule: reading the names correctly must not start refusing real
|
||||
ones. `seg_a0_1.ts` follows `seg_a0_0.ts` — the previous SEGMENT of that rendition, not the
|
||||
previous variant."""
|
||||
d = tmp_path / "m1_0_multi_+0.00"
|
||||
d.mkdir()
|
||||
(d / "seg_a0_0.ts").write_bytes(b"x")
|
||||
(d / "seg_a0_1.ts").write_bytes(b"audio")
|
||||
sessions["m1"] = _Session("sid-a", d, running=True, audio_langs=["hu", "en"])
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_a0_1.ts")
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.content == b"audio"
|
||||
|
||||
|
||||
def test_a_name_the_session_cannot_produce_fails_at_once(client, tmp_path, sessions):
|
||||
"""The guard was a blocklist: anything that was not a single-audio segment name was waved
|
||||
through to the timeout, so `GET …/anything.m3u8` still parked a worker for the full 30 s — and
|
||||
the route comment claimed a flood of hopeless names was what it prevented. A session produces
|
||||
its playlists and its segments and nothing else, so any other name is answerable now."""
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
sessions["m1"] = _Session("sid-a", d, running=True)
|
||||
started = time.monotonic()
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/nope.m3u8")
|
||||
elapsed = time.monotonic() - started
|
||||
assert r.status_code == 404, r.text
|
||||
assert elapsed < 2.0, f"waited {elapsed:.1f}s for a playlist this session never writes"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("digits", [100, 5000])
|
||||
def test_an_absurd_segment_index_is_a_404_not_a_500(client, tmp_path, sessions, digits):
|
||||
"""`int()` on an unbounded digit run raises ValueError past 4300 digits, and only OSError was
|
||||
caught — so a long enough name reached the client as a 500 with a traceback. Same bug class the
|
||||
subtitle routes were bounded for; the answer is the same 404 any other unproducible name gets."""
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
sessions["m1"] = _Session("sid-a", d, running=True)
|
||||
r = client.get(f"/api/plex/stream/m1/hls/sid-a/seg_{'9' * digits}.ts")
|
||||
assert r.status_code == 404, r.text
|
||||
|
||||
|
||||
def test_a_session_torn_down_during_the_read_is_a_404_not_a_500(
|
||||
client, monkeypatch, tmp_path, sessions
|
||||
):
|
||||
"""`_kill` rmtrees the whole directory and nothing holds `_lock` across the wait and the read,
|
||||
so the file can vanish in between — on every seek-restart, since `loadSession` awaits the new
|
||||
session before destroying the old hls instance. The player's recovery is keyed on the 404
|
||||
STATUS, so a 500 here skipped the reclaim and ended playback with "couldn't play"."""
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
(d / "seg_11.ts").write_bytes(b"")
|
||||
sessions["m1"] = _Session("sid-a", d)
|
||||
# The file existed when the wait checked, and is gone by the time the route reads it.
|
||||
monkeypatch.setattr(plex_stream, "wait_for", lambda *a, **k: True)
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
|
||||
assert r.status_code == 404, r.text
|
||||
|
||||
|
||||
def test_a_dead_producer_is_not_waited_out(client, tmp_path, sessions):
|
||||
"""The route passes `proc` to `wait_for`. Without it a session whose ffmpeg has already exited
|
||||
— a finished remux, a VAAPI init refusal — polls the whole 20 s for a segment that will never
|
||||
be written, holding a threadpool worker and its pooled DB connection. The predecessor IS on
|
||||
disk here, so `artifact_may_arrive` says yes and only `proc` can end the wait."""
|
||||
d = tmp_path / "m1_0_0_+0.00"
|
||||
d.mkdir()
|
||||
(d / "seg_5.ts").write_bytes(b"x")
|
||||
sessions["m1"] = _Session("sid-a", d, running=False)
|
||||
started = time.monotonic()
|
||||
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_6.ts")
|
||||
elapsed = time.monotonic() - started
|
||||
assert r.status_code == 404, r.text
|
||||
assert elapsed < 2.0, f"waited {elapsed:.1f}s on a producer that had already exited"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("audio_langs", [None, ["hu", "en"]])
|
||||
def test_the_name_ffmpeg_is_asked_for_is_a_name_the_guard_understands(tmp_path, audio_langs):
|
||||
"""One vocabulary, enforced. `segment_name` is what the warm-up and the PTS probe wait for, what
|
||||
the argv's own `-hls_segment_filename` pattern is built from, and what `artifact_may_arrive`
|
||||
parses back — they were three separate statements of the layout and two of them were wrong. Pin
|
||||
them to each other instead."""
|
||||
sess = _Session("sid", tmp_path, audio_langs=audio_langs)
|
||||
variant = sess.variants[0] if sess.variants else None
|
||||
name = plex_stream.segment_name(3, variant)
|
||||
prev = plex_stream.segment_name(2, variant)
|
||||
assert not plex_stream.artifact_may_arrive(sess, name)
|
||||
(tmp_path / prev).write_bytes(b"x")
|
||||
assert plex_stream.artifact_may_arrive(sess, name)
|
||||
|
||||
|
||||
def test_the_argv_asks_for_exactly_the_names_the_guard_accepts(tmp_path):
|
||||
"""The producer and the parser, pinned to each other through the argv itself.
|
||||
|
||||
Round 6's HIGH was a guard that read `seg_%v_%d.ts` sequence-first, so it was inert on every
|
||||
multi-audio item, and a warm-up that waited 8 s for a name ffmpeg never writes. The fix was to
|
||||
give the layout one spelling — but a comment saying so is what had just failed, so assert that
|
||||
the `-hls_segment_filename` pattern in the real argv is literally what `segment_name` produces,
|
||||
and that every playlist the argv names is one `playlist_names` accepts."""
|
||||
for audio_langs in (None, ["hu", "en", "de"]):
|
||||
args = plex_stream._ffmpeg_cmd(
|
||||
tmp_path / "in.mkv", 0.0, tmp_path, 0, 0.0, audio_langs, loudnorm=False
|
||||
)
|
||||
variants = plex_stream.session_variants(audio_langs)
|
||||
sess = _Session("sid", tmp_path, audio_langs=audio_langs)
|
||||
# The two patterns in the argv are the vocabulary's own spelling, `%d`/`%v` and all.
|
||||
seg_pattern = args[args.index("-hls_segment_filename") + 1]
|
||||
pl_pattern = args[-1].rsplit("/", 1)[-1].rsplit("\\", 1)[-1]
|
||||
if variants:
|
||||
assert seg_pattern.endswith(plex_stream.segment_name("%d", "%v")), seg_pattern
|
||||
assert pl_pattern == plex_stream.variant_playlist_name("%v"), pl_pattern
|
||||
assert args[args.index("-master_pl_name") + 1] == plex_stream._MASTER_PL
|
||||
else:
|
||||
assert seg_pattern.endswith(plex_stream.segment_name("%d")), seg_pattern
|
||||
assert pl_pattern == plex_stream._INDEX_PL, pl_pattern
|
||||
assert "-master_pl_name" not in args
|
||||
|
||||
# And every name those patterns can actually expand to is one the guard lets through — the
|
||||
# entry playlist hls.js is handed included, since a session whose own manifest is refused
|
||||
# cannot start at all.
|
||||
for v in variants or (None,):
|
||||
assert plex_stream.artifact_may_arrive(sess, plex_stream.segment_name(0, v))
|
||||
if v is not None:
|
||||
assert plex_stream.artifact_may_arrive(sess, plex_stream.variant_playlist_name(v))
|
||||
assert plex_stream.artifact_may_arrive(sess, sess.entry)
|
||||
|
||||
|
||||
def test_a_name_only_the_other_argv_branch_writes_is_refused_at_once(tmp_path, client, sessions):
|
||||
"""The guard is keyed to THIS session's vocabulary, not to HLS in general.
|
||||
|
||||
The two argv branches are disjoint: a single-audio session is given `seg_%d.ts` + `index.m3u8`
|
||||
and no `%v` at all, so `master.m3u8`, `stream_*.m3u8` and every `seg_<variant>_<n>.ts` are names
|
||||
it can never write — and vice versa. Accepting the union read them as "an HLS name, so wait and
|
||||
see", and each one then held a threadpool worker and its pooled DB connection for the full
|
||||
timeout: measured through this route against a live single-audio session, `master.m3u8` took
|
||||
30.0 s and `seg_a0_0.ts` 20.1 s to answer 404 (deep-review round 7)."""
|
||||
single = tmp_path / "m1_0_0_+0.00"
|
||||
single.mkdir()
|
||||
sessions["m1"] = _Session("sid-a", single, running=True)
|
||||
multi = tmp_path / "m2_0_multi_+0.00"
|
||||
multi.mkdir()
|
||||
sessions["m2"] = _Session("sid-b", multi, running=True, audio_langs=["hu", "en"])
|
||||
|
||||
# Each of these belongs to the other shape (or to no shape at all) and must be refused without
|
||||
# a wait — the assertion is the ELAPSED time as much as the status.
|
||||
for key, sid, names in (
|
||||
("m1", "sid-a", ["master.m3u8", "stream_0.m3u8", "seg_0_0.ts", "seg_zzzz_0.ts"]),
|
||||
("m2", "sid-b", ["index.m3u8", "seg_0.ts", "seg_a9_0.ts", "stream_zz.m3u8"]),
|
||||
):
|
||||
for name in names:
|
||||
started = time.monotonic()
|
||||
r = client.get(f"/api/plex/stream/{key}/hls/{sid}/{name}")
|
||||
elapsed = time.monotonic() - started
|
||||
assert r.status_code == 404, f"{name}: {r.text}"
|
||||
assert elapsed < 2.0, (
|
||||
f"{name}: waited {elapsed:.1f}s for a name this session cannot write"
|
||||
)
|
||||
|
||||
# And the control: the names each session CAN write are still waited for, not refused.
|
||||
assert plex_stream.artifact_may_arrive(sessions["m1"], "index.m3u8")
|
||||
assert plex_stream.artifact_may_arrive(sessions["m1"], "seg_0.ts")
|
||||
assert plex_stream.artifact_may_arrive(sessions["m2"], "master.m3u8")
|
||||
assert plex_stream.artifact_may_arrive(sessions["m2"], "stream_a1.m3u8")
|
||||
assert plex_stream.artifact_may_arrive(sessions["m2"], "seg_a1_0.ts")
|
||||
@@ -12,7 +12,7 @@ import app.plex.sync as syncmod
|
||||
from app import sysconfig
|
||||
from app.models import PlexItem, PlexLibrary
|
||||
from app.plex.sync import _apply_item, _canonical_guid, _sync_overlay
|
||||
from app.routes.plex._shared import _content_libs
|
||||
from app.plex.query import content_libs
|
||||
|
||||
|
||||
class _StubPlex:
|
||||
@@ -155,7 +155,7 @@ def test_sync_designates_overlay_and_writes_hu(db, monkeypatch):
|
||||
assert m.title_hu == "Öldöklés"
|
||||
assert m.genres_hu == ["Akció"]
|
||||
# The HU section is not browsable and mirrored no rows of its own.
|
||||
movie_ids, _ = _content_libs(db)
|
||||
movie_ids, _ = content_libs(db)
|
||||
assert en.id in movie_ids and hu.id not in movie_ids
|
||||
assert db.scalar(select(func.count(PlexItem.id)).where(PlexItem.library_id == hu.id)) == 0
|
||||
assert stats["overlaid"] == 1
|
||||
@@ -240,7 +240,7 @@ def test_content_libs_excludes_overlay_libraries(db):
|
||||
db.add(overlay)
|
||||
db.commit()
|
||||
|
||||
movie_ids, show_ids = _content_libs(db)
|
||||
movie_ids, show_ids = content_libs(db)
|
||||
assert en_lib.id in movie_ids
|
||||
assert overlay.id not in movie_ids # the HU overlay is never browsable
|
||||
assert show_ids == [en_shows.id]
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""DB-backed regression (E5 S4): GET /api/plex/person/{name} returns a cast/crew member's
|
||||
filmography — movies + shows in the ENABLED libraries whose cast OR directors include the name,
|
||||
each carrying the caller's watch-state, plus the cached headshot. Also covers `_content_libs`
|
||||
each carrying the caller's watch-state, plus the cached headshot. Also covers `content_libs`
|
||||
scoping (disabled libraries excluded) and the kind filter (episodes are not movies).
|
||||
|
||||
Uses the DB lane (`db` fixture) — skipped when no Postgres is reachable.
|
||||
@@ -20,7 +20,7 @@ from app.models import (
|
||||
PlexState,
|
||||
User,
|
||||
)
|
||||
from app.routes.plex._shared import _content_libs
|
||||
from app.plex.query import content_libs
|
||||
|
||||
NAME = "Robert De Niro"
|
||||
|
||||
@@ -151,5 +151,5 @@ def test_content_libs_partitions_only_enabled(db):
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
movie_ids, show_ids = _content_libs(db)
|
||||
movie_ids, show_ids = content_libs(db)
|
||||
assert len(movie_ids) == 1 and len(show_ids) == 1 # the disabled movie lib is excluded
|
||||
|
||||
@@ -0,0 +1,495 @@
|
||||
"""The playback-class filter (E7): filter the Plex feed by Direct Play / Direct Stream / Transcode·HW /
|
||||
Transcode·SW. Covers the shared classifier + Stream parsing (unit), the sync derivation + codec-change
|
||||
reset, the bit-depth backfill, and the /library filter + /facets counts (DB lane).
|
||||
|
||||
The HW/SW split is derived from the stored `transcode_hw` facet (codec at sync, refined by the bit-depth
|
||||
backfill). These tests pin that a 10-bit/HDR HEVC lands in SW even though its codec is HW-decodable —
|
||||
the exact case the runtime VAAPI path (plex.stream) also special-cases.
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import app.state as state
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.models import PlexItem, PlexLibrary, User
|
||||
from app.plex import sync
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
class TestTranscodeHwCapable:
|
||||
"""The single classifier both the catalog facet and (by codec) the runtime encoder choice key off."""
|
||||
|
||||
def test_non_transcode_items_are_none(self):
|
||||
assert sync.transcode_hw_capable("direct", "h264") is None
|
||||
assert sync.transcode_hw_capable("remux", "h264") is None
|
||||
assert sync.transcode_hw_capable(None, "hevc") is None
|
||||
|
||||
def test_hw_decodable_8bit_sdr_is_hw(self):
|
||||
assert sync.transcode_hw_capable("transcode", "hevc") is True
|
||||
assert sync.transcode_hw_capable("transcode", "vp9", bit_depth=8) is True
|
||||
|
||||
def test_non_decodable_codec_is_sw(self):
|
||||
# AV1 / MPEG-4 ASP (XviD) can't be HW-decoded on Gen 9.5 → software regardless of bit depth.
|
||||
assert sync.transcode_hw_capable("transcode", "av1") is False
|
||||
assert sync.transcode_hw_capable("transcode", "mpeg4", bit_depth=8) is False
|
||||
|
||||
def test_10bit_or_hdr_hevc_is_sw(self):
|
||||
# A HW-decodable codec that is 10-bit or HDR still breaks the VAAPI nv12 pipeline → software.
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=10) is False
|
||||
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=8, hdr=True) is False
|
||||
|
||||
|
||||
class TestVideoStreamProps:
|
||||
def test_reads_bit_depth_and_hdr_from_the_video_stream(self):
|
||||
meta = {
|
||||
"Media": [
|
||||
{
|
||||
"Part": [
|
||||
{
|
||||
"Stream": [
|
||||
{"streamType": 2, "codec": "eac3"}, # audio — skipped
|
||||
{"streamType": 1, "bitDepth": 10, "colorTrc": "smpte2084"},
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
assert sync._video_stream_props(meta) == (10, True)
|
||||
|
||||
def test_8bit_sdr_stream(self):
|
||||
meta = {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]}
|
||||
assert sync._video_stream_props(meta) == (8, False)
|
||||
|
||||
def test_absent_stream_is_none_false(self):
|
||||
assert sync._video_stream_props({}) == (None, False)
|
||||
assert sync._video_stream_props({"Media": [{"Part": [{}]}]}) == (None, False)
|
||||
|
||||
|
||||
class TestSyncDerivation:
|
||||
"""`_apply_item` sets transcode_hw codec-only at sync (bit depth unknown from the listing), and a
|
||||
changed video codec resets the probe watermark so a stale bit-depth can't linger."""
|
||||
|
||||
def _meta(self, vcodec, *, key="/parts/1/file.mkv", file="/m.mkv"):
|
||||
return {
|
||||
"title": "X",
|
||||
"Media": [{"videoCodec": vcodec, "Part": [{"file": file, "key": key}]}],
|
||||
}
|
||||
|
||||
def test_codec_only_derivation_at_sync(self):
|
||||
row = PlexItem(rating_key="x", kind="movie")
|
||||
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc"))
|
||||
assert row.playable == "transcode" and row.transcode_hw is True # decodable, 8-bit assumed
|
||||
row2 = PlexItem(rating_key="y", kind="movie")
|
||||
sync._apply_item(row2, lib_id=1, kind="movie", meta=self._meta("av1"))
|
||||
assert row2.transcode_hw is False # av1 not HW-decodable
|
||||
|
||||
def test_probed_value_survives_resync_when_codec_unchanged(self):
|
||||
row = PlexItem(rating_key="x", kind="movie")
|
||||
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc"))
|
||||
# Simulate the backfill downgrading this 10-bit HEVC to software.
|
||||
row.transcode_hw = False
|
||||
row.video_bit_depth = 10
|
||||
row.media_probed_at = sync.datetime.now(sync.timezone.utc)
|
||||
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc"))
|
||||
assert row.transcode_hw is False # NOT re-derived back to True
|
||||
assert row.video_bit_depth == 10 and row.media_probed_at is not None
|
||||
|
||||
def test_codec_change_resets_the_probe(self):
|
||||
row = PlexItem(rating_key="x", kind="movie")
|
||||
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc"))
|
||||
row.transcode_hw = False
|
||||
row.video_bit_depth = 10
|
||||
row.media_probed_at = sync.datetime.now(sync.timezone.utc)
|
||||
# The file was replaced with an h264 remux — the stale 10-bit probe must be cleared.
|
||||
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("h264"))
|
||||
assert row.video_bit_depth is None and row.media_probed_at is None
|
||||
|
||||
def test_same_codec_reencode_with_a_new_part_resets_the_probe(self):
|
||||
# A same-codec bit-depth re-encode (e.g. 8-bit→10-bit HEVC) keeps codec 'hevc' but replaces the
|
||||
# physical file, so its Plex Part key changes. Codec-only reset would miss this and keep the stale
|
||||
# 8-bit/HW verdict forever (the backfill only re-probes media_probed_at IS NULL rows) — the
|
||||
# part-key move must reset the watermark so the new bit depth is re-probed.
|
||||
row = PlexItem(rating_key="x", kind="movie")
|
||||
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc", key="/parts/1/f.mkv"))
|
||||
row.transcode_hw = True
|
||||
row.video_bit_depth = 8
|
||||
row.media_probed_at = sync.datetime.now(sync.timezone.utc)
|
||||
# Re-encoded in place: same codec 'hevc', but a new Part key (new file).
|
||||
sync._apply_item(row, lib_id=1, kind="movie", meta=self._meta("hevc", key="/parts/2/f.mkv"))
|
||||
assert row.video_bit_depth is None and row.media_probed_at is None # re-probe scheduled
|
||||
|
||||
|
||||
def test_migration_codec_set_matches_hw_decodable_codecs():
|
||||
# Migration 0077 hardcodes the HW-decodable codec set in raw SQL and its comment mandates it stay
|
||||
# identical to plex.sync.HW_DECODABLE_CODECS. A comment isn't enforcement (house rule) — pin the
|
||||
# equality so adding/removing a codec from the frozenset without editing the migration fails here
|
||||
# (else already-migrated prod rows keep a wrong transcode_hw the runtime derivation wouldn't catch).
|
||||
mig = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "alembic"
|
||||
/ "versions"
|
||||
/ "0077_plex_playback_class.py"
|
||||
)
|
||||
m = re.search(r"lower\(codec_video\)\s+IN\s+\(([^)]*)\)", mig.read_text())
|
||||
assert m, "could not find the codec IN(...) list in migration 0077"
|
||||
codecs = {c.strip().strip("'\"").lower() for c in m.group(1).split(",")}
|
||||
assert codecs == {c.lower() for c in sync.HW_DECODABLE_CODECS}
|
||||
|
||||
|
||||
class _FakePlex:
|
||||
"""Stands in for PlexClient.metadata — returns a canned full-metadata dict per rating_key."""
|
||||
|
||||
def __init__(self, by_key):
|
||||
self.by_key = by_key
|
||||
|
||||
def metadata(self, rating_key, markers=False):
|
||||
return self.by_key.get(rating_key)
|
||||
|
||||
|
||||
def test_probe_media_batch_refines_hw_to_sw_for_10bit(db):
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
hevc8 = PlexItem(
|
||||
rating_key="h8",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="8bit",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
)
|
||||
hevc10 = PlexItem(
|
||||
rating_key="h10",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="10bit",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
)
|
||||
xvid = PlexItem(
|
||||
rating_key="xv",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="xvid",
|
||||
playable="transcode",
|
||||
codec_video="mpeg4",
|
||||
transcode_hw=False,
|
||||
)
|
||||
db.add_all([hevc8, hevc10, xvid])
|
||||
db.commit()
|
||||
plex = _FakePlex(
|
||||
{
|
||||
"h8": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]},
|
||||
"h10": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 10}]}]}]},
|
||||
}
|
||||
)
|
||||
probed = sync._probe_media_batch(db, plex)
|
||||
db.refresh(hevc8)
|
||||
db.refresh(hevc10)
|
||||
db.refresh(xvid)
|
||||
# Only the two HW-decodable HEVC items are probed; the XviD is SW-by-codec and never fetched.
|
||||
assert probed == 2
|
||||
assert hevc8.video_bit_depth == 8 and hevc8.transcode_hw is True and hevc8.media_probed_at
|
||||
assert hevc10.video_bit_depth == 10 and hevc10.transcode_hw is False # downgraded to software
|
||||
assert xvid.media_probed_at is None # not in the work-list
|
||||
# A second run has nothing left to do (watermarks set).
|
||||
assert sync._probe_media_batch(db, plex) == 0
|
||||
|
||||
|
||||
def test_probe_stamps_even_when_plex_has_no_stream(db):
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
item = PlexItem(
|
||||
rating_key="ns",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="nostream",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
)
|
||||
db.add(item)
|
||||
db.commit()
|
||||
plex = _FakePlex({"ns": {"Media": [{"Part": [{}]}]}}) # no Stream array
|
||||
assert sync._probe_media_batch(db, plex) == 1
|
||||
db.refresh(item)
|
||||
# Probed (watermark set so it isn't re-fetched forever); bit depth unknown → stays codec-HW.
|
||||
assert item.media_probed_at is not None and item.video_bit_depth is None
|
||||
assert item.transcode_hw is True
|
||||
|
||||
|
||||
def test_probe_media_batch_is_bounded_and_oldest_first(db):
|
||||
# The per-run bound (limit) caps Plex load; oldest-id-first means successive runs cover the whole
|
||||
# backlog without starving any item. Seed 3 eligible items, probe with limit=2.
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
items = [
|
||||
PlexItem(
|
||||
rating_key=f"h{i}",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title=f"m{i}",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
)
|
||||
for i in range(3)
|
||||
]
|
||||
db.add_all(items) # inserted in order → ascending ids
|
||||
db.commit()
|
||||
plex = _FakePlex(
|
||||
{
|
||||
f"h{i}": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]}
|
||||
for i in range(3)
|
||||
}
|
||||
)
|
||||
assert sync._probe_media_batch(db, plex, limit=2) == 2 # exactly the bound
|
||||
for it in items:
|
||||
db.refresh(it)
|
||||
# The two LOWEST-id rows were probed; the third waits for the next run.
|
||||
assert items[0].media_probed_at is not None and items[1].media_probed_at is not None
|
||||
assert items[2].media_probed_at is None
|
||||
# The next run continues with the remaining item.
|
||||
assert sync._probe_media_batch(db, plex, limit=2) == 1
|
||||
db.refresh(items[2])
|
||||
assert items[2].media_probed_at is not None
|
||||
|
||||
|
||||
class _RaisingPlex:
|
||||
"""metadata() raises for `bad_key`, returns a canned dict otherwise — to exercise the transient
|
||||
fetch-failure branch (leave the watermark NULL so the item retries next run)."""
|
||||
|
||||
def __init__(self, by_key, bad_key):
|
||||
self.by_key = by_key
|
||||
self.bad_key = bad_key
|
||||
|
||||
def metadata(self, rating_key, markers=False):
|
||||
if rating_key == self.bad_key:
|
||||
raise RuntimeError("plex timeout")
|
||||
return self.by_key.get(rating_key)
|
||||
|
||||
|
||||
def test_probe_media_batch_leaves_a_transient_failure_unstamped_to_retry(db):
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
good = PlexItem(
|
||||
rating_key="g",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="good",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
)
|
||||
bad = PlexItem(
|
||||
rating_key="b",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="bad",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
)
|
||||
db.add_all([good, bad])
|
||||
db.commit()
|
||||
plex = _RaisingPlex(
|
||||
{"g": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 10}]}]}]}}, bad_key="b"
|
||||
)
|
||||
# The raise on 'b' must not abort the batch: 'g' is still probed and counted; 'b' stays NULL.
|
||||
assert sync._probe_media_batch(db, plex) == 1
|
||||
db.refresh(good)
|
||||
db.refresh(bad)
|
||||
assert good.media_probed_at is not None and good.transcode_hw is False # 10-bit → SW
|
||||
assert bad.media_probed_at is None # transient failure → retried next run, not stamped
|
||||
|
||||
|
||||
def _seed_classes(db):
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
db.add_all(
|
||||
[
|
||||
PlexItem(
|
||||
rating_key="d1", library_id=lib.id, kind="movie", title="Direct", playable="direct"
|
||||
),
|
||||
PlexItem(
|
||||
rating_key="r1", library_id=lib.id, kind="movie", title="Remux", playable="remux"
|
||||
),
|
||||
PlexItem(
|
||||
rating_key="thw",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="HW",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=True,
|
||||
),
|
||||
PlexItem(
|
||||
rating_key="tsw",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="SW",
|
||||
playable="transcode",
|
||||
codec_video="mpeg4",
|
||||
transcode_hw=False,
|
||||
),
|
||||
]
|
||||
)
|
||||
return lib
|
||||
|
||||
|
||||
def _client(db, user, monkeypatch):
|
||||
monkeypatch.setattr(state, "_configured_cache", True)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
app.dependency_overrides[current_user] = lambda: user
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_library_filters_by_playback_class(db, monkeypatch):
|
||||
_seed_classes(db)
|
||||
user = User(email="pclass@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
client = _client(db, user, monkeypatch)
|
||||
try:
|
||||
# Software-only transcodes.
|
||||
sw = client.get("/api/plex/library?scope=movie&playable=transcode_sw")
|
||||
assert sw.status_code == 200, sw.text
|
||||
assert {i["id"] for i in sw.json()["items"]} == {"tsw"}
|
||||
# HW transcode + Direct Play, OR-combined.
|
||||
both = client.get("/api/plex/library?scope=movie&playable=transcode_hw,direct")
|
||||
assert {i["id"] for i in both.json()["items"]} == {"thw", "d1"}
|
||||
# The HW card carries the transcode_hw flag for the 4-way badge.
|
||||
card = next(i for i in both.json()["items"] if i["id"] == "thw")
|
||||
assert card["transcode_hw"] is True
|
||||
# No filter → everything.
|
||||
allm = client.get("/api/plex/library?scope=movie")
|
||||
assert {i["id"] for i in allm.json()["items"]} == {"d1", "r1", "thw", "tsw"}
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
def test_facets_count_playback_classes_in_severity_order(db, monkeypatch):
|
||||
_seed_classes(db)
|
||||
user = User(email="pclassfacet@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
client = _client(db, user, monkeypatch)
|
||||
try:
|
||||
resp = client.get("/api/plex/facets?scope=movie")
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
assert resp.status_code == 200, resp.text
|
||||
playable = resp.json()["playable"]
|
||||
# Fixed good→heavy order (not by count), one per seeded class.
|
||||
assert [p["value"] for p in playable] == ["direct", "remux", "transcode_hw", "transcode_sw"]
|
||||
assert {p["value"]: p["count"] for p in playable} == {
|
||||
"direct": 1,
|
||||
"remux": 1,
|
||||
"transcode_hw": 1,
|
||||
"transcode_sw": 1,
|
||||
}
|
||||
|
||||
|
||||
def test_unprobed_transcode_folds_into_sw(db, monkeypatch):
|
||||
# transcode_hw NULL is the COMMON initial state (before the bit-depth backfill runs). The query's
|
||||
# CASE has no explicit NULL arm — it relies on the `playable=='transcode'` else folding NULL in with
|
||||
# software. Pin that fold for BOTH the filter and the facet, so a reorder can't silently drop the
|
||||
# item from the feed (an honest-state violation).
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
db.add(
|
||||
PlexItem(
|
||||
rating_key="tnull",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="Unprobed",
|
||||
playable="transcode",
|
||||
codec_video="hevc",
|
||||
transcode_hw=None,
|
||||
)
|
||||
)
|
||||
user = User(email="pnull@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
client = _client(db, user, monkeypatch)
|
||||
try:
|
||||
sw = client.get("/api/plex/library?scope=movie&playable=transcode_sw")
|
||||
assert {i["id"] for i in sw.json()["items"]} == {"tnull"} # NULL → SW bucket
|
||||
hw = client.get("/api/plex/library?scope=movie&playable=transcode_hw")
|
||||
assert hw.json()["items"] == [] # never HW
|
||||
facets = client.get("/api/plex/facets?scope=movie")
|
||||
counts = {p["value"]: p["count"] for p in facets.json()["playable"]}
|
||||
assert counts.get("transcode_sw") == 1 and counts.get("transcode_hw", 0) == 0
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
def test_facet_playable_self_excludes_the_active_selection(db, monkeypatch):
|
||||
# A selected playable class must NOT narrow the playable facet's OWN counts (facet self-exclusion),
|
||||
# otherwise picking one class would collapse the list and you couldn't OR-add a second. Selecting
|
||||
# 'direct' must still return all four classes at their full, unnarrowed counts.
|
||||
_seed_classes(db)
|
||||
user = User(email="pself@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
client = _client(db, user, monkeypatch)
|
||||
try:
|
||||
resp = client.get("/api/plex/facets?scope=movie&playable=direct")
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
assert resp.status_code == 200, resp.text
|
||||
counts = {p["value"]: p["count"] for p in resp.json()["playable"]}
|
||||
assert counts == {"direct": 1, "remux": 1, "transcode_hw": 1, "transcode_sw": 1}
|
||||
|
||||
|
||||
def test_active_playable_narrows_the_other_facets(db, monkeypatch):
|
||||
# The flip side of self-exclusion: an active playable class DOES narrow the OTHER facet groups (so
|
||||
# e.g. the year range reflects only the selected class). Seed a Direct movie (2000) and a
|
||||
# software-transcode movie (2020); selecting playable=direct must collapse the year facet to 2000.
|
||||
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
|
||||
db.add(lib)
|
||||
db.flush()
|
||||
db.add_all(
|
||||
[
|
||||
PlexItem(
|
||||
rating_key="d2000",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="Direct2000",
|
||||
playable="direct",
|
||||
year=2000,
|
||||
),
|
||||
PlexItem(
|
||||
rating_key="s2020",
|
||||
library_id=lib.id,
|
||||
kind="movie",
|
||||
title="SW2020",
|
||||
playable="transcode",
|
||||
codec_video="mpeg4",
|
||||
transcode_hw=False,
|
||||
year=2020,
|
||||
),
|
||||
]
|
||||
)
|
||||
user = User(email="pnarrow@example.com")
|
||||
db.add(user)
|
||||
db.commit()
|
||||
client = _client(db, user, monkeypatch)
|
||||
try:
|
||||
wide = client.get("/api/plex/facets?scope=movie").json()
|
||||
narrow = client.get("/api/plex/facets?scope=movie&playable=direct").json()
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
assert (wide["year_min"], wide["year_max"]) == (2000, 2020) # both classes span the range
|
||||
assert (narrow["year_min"], narrow["year_max"]) == (2000, 2000) # narrowed to the Direct subset
|
||||
@@ -58,7 +58,12 @@ def _post(db, user, monkeypatch, rating_key, *, captured):
|
||||
def fake_start(db_, it, start, audio, aoff, multi, transcode=False):
|
||||
captured["transcode"] = transcode
|
||||
return SimpleNamespace(
|
||||
media_start_s=start, entry="index.m3u8", transcode=transcode, hw=False
|
||||
id="sid",
|
||||
media_start_s=start,
|
||||
entry="index.m3u8",
|
||||
transcode=transcode,
|
||||
hw=False,
|
||||
tonemap=False,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(plex_stream, "start_session", fake_start)
|
||||
|
||||
@@ -267,50 +267,206 @@ class TestTranscodeCommand:
|
||||
)
|
||||
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # not a software scale
|
||||
|
||||
def test_hdr_tonemap_uses_the_opencl_graph(self):
|
||||
# The HDR/10-bit HW path: VAAPI decode → OpenCL tonemap → VAAPI encode. Gen 9.5 can't tonemap in
|
||||
# the VAAPI VPP, so this is the only HW route for a 10-bit/HDR source.
|
||||
cmd = stream._ffmpeg_cmd(
|
||||
self.SRC, 0.0, self.OUT, None, transcode=True, hw=True, tonemap=True
|
||||
)
|
||||
assert _venc(cmd) == "h264_vaapi"
|
||||
assert "libx264" not in cmd
|
||||
# Named devices + an OpenCL device DERIVED from the VAAPI one, all before -i.
|
||||
assert cmd[cmd.index("-init_hw_device") + 1] == f"vaapi=va:{stream._VAAPI_DEVICE}"
|
||||
assert "opencl=ocl@va" in cmd
|
||||
assert cmd[cmd.index("-hwaccel_device") + 1] == "va"
|
||||
assert cmd.index("-init_hw_device") < cmd.index("-i")
|
||||
# The filter graph runs on the derived OpenCL device and tone-maps HDR→SDR to 8-bit nv12.
|
||||
assert cmd[cmd.index("-filter_hw_device") + 1] == "ocl"
|
||||
vf = cmd[cmd.index("-vf") + 1]
|
||||
assert "hwmap=derive_device=opencl" in vf
|
||||
assert "tonemap_opencl=" in vf and "format=nv12" in vf
|
||||
assert "hwmap=derive_device=vaapi:reverse=1" in vf
|
||||
assert "scale_vaapi" not in vf # the plain 8-bit path's filter must NOT be used here
|
||||
assert (
|
||||
cmd[cmd.index("-force_key_frames") + 1] == f"expr:gte(t,n_forced*{stream._SEG_SECONDS})"
|
||||
)
|
||||
|
||||
|
||||
class TestWantHw:
|
||||
"""VAAPI is used only where the render node was passed in AND (in auto) the source is HW-decodable,
|
||||
with an admin `plex_transcode_hw` policy (auto/on/off) on top."""
|
||||
with an admin `plex_transcode_hw` policy (auto/on/off) on top. A HW session on a 10-bit/HDR source
|
||||
tone-maps on the GPU via OpenCL (tonemap=True) when the OpenCL runtime is present, else drops to
|
||||
software — see the `_select_encode` cases below."""
|
||||
|
||||
SRC = Path("/media/movie.mkv")
|
||||
|
||||
@staticmethod
|
||||
def _policy(monkeypatch, mode):
|
||||
monkeypatch.setattr(stream.sysconfig, "get_str", lambda db, k: mode)
|
||||
|
||||
@staticmethod
|
||||
def _kind(monkeypatch, kind):
|
||||
# Pin the source classifier so the tests don't shell out to ffprobe. `kind` is one of
|
||||
# stream._VK_HWSAFE / _VK_HDR / _VK_SDR10.
|
||||
monkeypatch.setattr(stream, "_probe_video_kind", lambda src: kind)
|
||||
|
||||
@staticmethod
|
||||
def _hwsafe(monkeypatch, safe):
|
||||
# Back-compat convenience for the hw-gate tests that only care about 8-bit-SDR vs not: True →
|
||||
# _VK_HWSAFE, False → _VK_SDR10 (a non-HDR non-hwsafe class, so the hw half still resolves).
|
||||
monkeypatch.setattr(
|
||||
stream, "_probe_video_kind", lambda src: stream._VK_HWSAFE if safe else stream._VK_SDR10
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _opencl(monkeypatch, present):
|
||||
# Pin the OpenCL-runtime probe so the routing doesn't depend on the CI box's /etc/OpenCL.
|
||||
monkeypatch.setattr(stream, "_opencl_present", lambda: present)
|
||||
|
||||
def test_no_hw_when_not_transcoding(self, monkeypatch):
|
||||
self._policy(monkeypatch, "auto")
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=False) is False
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=False, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
|
||||
def test_no_hw_without_a_render_node(self, monkeypatch):
|
||||
self._policy(monkeypatch, "auto")
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: False)
|
||||
assert stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True) is False
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
|
||||
def test_auto_uses_hw_for_a_decodable_source_with_the_render_node(self, monkeypatch):
|
||||
def test_auto_uses_hw_for_a_decodable_8bit_source_with_the_render_node(self, monkeypatch):
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._hwsafe(monkeypatch, True)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True) is True
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC)
|
||||
is True
|
||||
)
|
||||
|
||||
def test_auto_falls_back_to_software_for_a_non_decodable_source(self, monkeypatch):
|
||||
# AV1 / MPEG-4 ASP (XviD) aren't decodable on Gen 9.5 → the GPU can't feed its own encoder.
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._hwsafe(monkeypatch, True) # decodability gate must fire BEFORE the probe
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True) is False
|
||||
assert stream._want_hw(None, _FakeItem(codec_video="mpeg4"), transcode=True) is False
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="mpeg4"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
|
||||
def test_auto_hdr_source_falls_back_to_software_without_opencl(self, monkeypatch):
|
||||
# A decodable HEVC that is true-HDR (PQ/HLG) can't use scale_vaapi=format=nv12 (Gen 9.5), and with
|
||||
# no OpenCL runtime to tonemap on the GPU it drops to software libx264 (BUG 1's behaviour).
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._kind(monkeypatch, stream._VK_HDR)
|
||||
self._opencl(monkeypatch, False)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (False, False)
|
||||
|
||||
def test_auto_hdr_source_uses_hw_tonemap_with_opencl(self, monkeypatch):
|
||||
# With the OpenCL runtime present, a true-HDR HEVC tone-maps on the GPU (VAAPI decode → OpenCL
|
||||
# tonemap → VAAPI encode) instead of dropping to CPU — the HW HDR path.
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._kind(monkeypatch, stream._VK_HDR)
|
||||
self._opencl(monkeypatch, True)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, True)
|
||||
|
||||
def test_auto_10bit_SDR_source_uses_software_not_tonemap(self, monkeypatch):
|
||||
# F1 regression: a 10-bit *SDR* (bt709) source is NOT HDR, so it must NOT get the hable HDR tone
|
||||
# curve even when the OpenCL runtime is present — that would visibly alter an SDR picture. It takes
|
||||
# software libx264 (an exact 10→8-bit downconvert), never (True, True).
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._kind(monkeypatch, stream._VK_SDR10)
|
||||
self._opencl(monkeypatch, True)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (False, False)
|
||||
|
||||
def test_auto_8bit_source_uses_plain_hw_not_tonemap(self, monkeypatch):
|
||||
# An 8-bit SDR source takes the plain full-GPU scale path — never the tonemap graph — even where
|
||||
# OpenCL is available.
|
||||
self._policy(monkeypatch, "auto")
|
||||
self._kind(monkeypatch, stream._VK_HWSAFE)
|
||||
self._opencl(monkeypatch, True)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
|
||||
def test_off_forces_software_even_for_a_decodable_source(self, monkeypatch):
|
||||
self._policy(monkeypatch, "off")
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True) is False
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
|
||||
def test_on_forces_hw_even_for_a_non_decodable_source(self, monkeypatch):
|
||||
# The admin override trusts the GPU to decode it; skips the decodability gate (render node still
|
||||
# required — HW is impossible without it).
|
||||
def test_on_forces_hw_and_skips_both_source_gates(self, monkeypatch):
|
||||
# The admin override trusts the GPU; skips the decodability AND the bit-depth gate (the runtime
|
||||
# HW→SW fallback still backstops it). Render node still required — HW is impossible without it.
|
||||
self._policy(monkeypatch, "on")
|
||||
self._hwsafe(monkeypatch, False) # would veto in auto; "on" ignores it
|
||||
self._opencl(
|
||||
monkeypatch, False
|
||||
) # pin: tonemap-half is irrelevant here, keep it deterministic
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True) is True
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True, src=self.SRC)
|
||||
is True
|
||||
)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: False)
|
||||
assert stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True) is False
|
||||
assert (
|
||||
stream._want_hw(None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC)
|
||||
is False
|
||||
)
|
||||
|
||||
def test_on_still_picks_the_tonemap_graph_for_an_hdr_source(self, monkeypatch):
|
||||
# "on" trusts the GPU but must still route a true-HDR source through the tonemap graph (plain
|
||||
# scale_vaapi would hit the VPP wall) when OpenCL is present.
|
||||
self._policy(monkeypatch, "on")
|
||||
self._kind(monkeypatch, stream._VK_HDR)
|
||||
self._opencl(monkeypatch, True)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, True)
|
||||
|
||||
def test_on_10bit_SDR_uses_plain_hw_not_tonemap(self, monkeypatch):
|
||||
# F1 regression under "on": a 10-bit SDR source is NOT HDR, so even with OpenCL present "on" must
|
||||
# pick plain HW (True, False) — the HDR tone curve must never touch an SDR picture.
|
||||
self._policy(monkeypatch, "on")
|
||||
self._kind(monkeypatch, stream._VK_SDR10)
|
||||
self._opencl(monkeypatch, True)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
|
||||
def test_on_without_opencl_is_plain_hw(self, monkeypatch):
|
||||
# "on" with no OpenCL runtime → plain HW, tonemap=False, regardless of source class (the runtime
|
||||
# HW→SW fallback backstops a source the plain path can't actually handle). Pins the tonemap half
|
||||
# of the "on" tuple, which test_on_forces_hw_and_skips_both_source_gates only checks via _want_hw.
|
||||
self._policy(monkeypatch, "on")
|
||||
self._kind(monkeypatch, stream._VK_HDR) # even HDR: no runtime → can't tonemap on the GPU
|
||||
self._opencl(monkeypatch, False)
|
||||
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
|
||||
assert stream._select_encode(
|
||||
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
|
||||
) == (True, False)
|
||||
|
||||
|
||||
class TestColdStartWarmup:
|
||||
@@ -327,13 +483,14 @@ class TestColdStartWarmup:
|
||||
src.write_bytes(b"x")
|
||||
monkeypatch.setattr(stream.paths, "local_media_path", lambda db, fp: src)
|
||||
monkeypatch.setattr(stream.sysconfig, "get_int", lambda db, k: 2)
|
||||
monkeypatch.setattr(stream.sysconfig, "get_bool", lambda db, k: False)
|
||||
monkeypatch.setattr(stream.subprocess, "Popen", lambda *a, **k: _FakeProc())
|
||||
return tmp_path
|
||||
|
||||
def test_warmup_waits_for_the_first_few_segments(self, fake_env, monkeypatch):
|
||||
waited = []
|
||||
|
||||
def spy(path, timeout=20.0):
|
||||
def spy(path, timeout=20.0, proc=None):
|
||||
waited.append(path.name)
|
||||
return True
|
||||
|
||||
@@ -346,7 +503,7 @@ class TestColdStartWarmup:
|
||||
def test_warmup_stops_at_the_first_segment_that_isnt_ready(self, fake_env, monkeypatch):
|
||||
waited = []
|
||||
|
||||
def spy(path, timeout=20.0):
|
||||
def spy(path, timeout=20.0, proc=None):
|
||||
waited.append(path.name)
|
||||
return path.name != "seg_1.ts" # seg_1 never appears → warm-up gives up there
|
||||
|
||||
@@ -360,7 +517,7 @@ class TestColdStartWarmup:
|
||||
# still returned so the client's own segment requests can drive it, as before.
|
||||
waited = []
|
||||
|
||||
def spy(path, timeout=20.0):
|
||||
def spy(path, timeout=20.0, proc=None):
|
||||
waited.append(path.name)
|
||||
return False
|
||||
|
||||
@@ -398,7 +555,13 @@ class TestTranscodeStartSession:
|
||||
"get_str",
|
||||
lambda db, k: {"plex_transcode_preset": "slow"}.get(k, "auto"),
|
||||
)
|
||||
monkeypatch.setattr(stream, "wait_for", lambda path, timeout=20.0: True)
|
||||
# Loudness normalisation (item 14) is off unless a test turns it on via `stream._loudnorm_on`.
|
||||
monkeypatch.setattr(
|
||||
stream.sysconfig,
|
||||
"get_bool",
|
||||
lambda db, k: {"plex_audio_loudnorm": getattr(stream, "_loudnorm_on", False)}[k],
|
||||
)
|
||||
monkeypatch.setattr(stream, "wait_for", lambda path, timeout=20.0, proc=None: True)
|
||||
_CapturingProc.calls = []
|
||||
monkeypatch.setattr(stream.subprocess, "Popen", _CapturingProc)
|
||||
return tmp_path
|
||||
@@ -406,7 +569,9 @@ class TestTranscodeStartSession:
|
||||
def test_software_transcode_threads_the_knobs_into_the_command(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr(stream, "_want_hw", lambda db, item, transcode: False)
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
|
||||
)
|
||||
s = stream.start_session(
|
||||
db=None, item=_FakeItem(codec_video="mpeg4"), start_s=0.0, transcode=True
|
||||
)
|
||||
@@ -421,25 +586,362 @@ class TestTranscodeStartSession:
|
||||
assert kwargs.get("preexec_fn") is stream._sw_child_preexec
|
||||
|
||||
def test_hardware_transcode_uses_the_gpu_and_never_nices(self, capturing_env, monkeypatch):
|
||||
monkeypatch.setattr(stream, "_want_hw", lambda db, item, transcode: True)
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (True, False)
|
||||
)
|
||||
s = stream.start_session(
|
||||
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
|
||||
)
|
||||
assert s is not None and s.transcode is True and s.hw is True
|
||||
assert s is not None and s.transcode is True and s.hw is True and s.tonemap is False
|
||||
assert len(_CapturingProc.calls) == 1 # first segment arrived — no HW→SW fallback respawn
|
||||
(cmd, *_), kwargs = _CapturingProc.calls[0]
|
||||
assert _venc(cmd) == "h264_vaapi"
|
||||
assert "libx264" not in cmd
|
||||
assert (
|
||||
"-vf" not in cmd or cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12"
|
||||
) # no SW cap
|
||||
# Plain-HW path unconditionally emits the nv12 downconvert (a disjunction on "-vf not in cmd"
|
||||
# would silently pass if the filter were dropped — assert it is actually present).
|
||||
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # no SW cap
|
||||
assert kwargs.get("preexec_fn") is None # GPU encode is cheap — never niced
|
||||
|
||||
def test_hdr_tonemap_transcode_threads_the_opencl_graph_and_flags_the_session(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
# A HW tonemap session (10-bit/HDR + OpenCL) must build the OpenCL graph in the argv and carry
|
||||
# tonemap=True on the session, still on the GPU (hw=True) and never niced.
|
||||
monkeypatch.setattr(stream, "_select_encode", lambda db, item, transcode, src: (True, True))
|
||||
s = stream.start_session(
|
||||
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
|
||||
)
|
||||
assert s is not None and s.transcode is True and s.hw is True and s.tonemap is True
|
||||
(cmd, *_), kwargs = _CapturingProc.calls[0]
|
||||
assert _venc(cmd) == "h264_vaapi"
|
||||
assert "opencl=ocl@va" in cmd and cmd[cmd.index("-filter_hw_device") + 1] == "ocl"
|
||||
assert "tonemap_opencl=" in cmd[cmd.index("-vf") + 1]
|
||||
assert kwargs.get("preexec_fn") is None
|
||||
|
||||
def test_a_hw_session_that_produces_nothing_falls_back_to_software(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
# BUG 1 backstop: _want_hw picked HW but the VAAPI session emits no first segment (a VAProfile
|
||||
# refusal the gates didn't catch). start_session must respawn on libx264 and report hw=False so
|
||||
# the item still plays and the route/player honestly show CPU.
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (True, False)
|
||||
)
|
||||
seen = {"n": 0}
|
||||
|
||||
def flaky_wait(path, timeout=20.0, proc=None):
|
||||
seen["n"] += 1
|
||||
return seen["n"] > 1 # the first seg-0 wait (HW) fails; every wait after succeeds (SW)
|
||||
|
||||
monkeypatch.setattr(stream, "wait_for", flaky_wait)
|
||||
s = stream.start_session(
|
||||
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
|
||||
)
|
||||
assert (
|
||||
s is not None and s.transcode is True and s.hw is False
|
||||
) # honest CPU after the fallback
|
||||
assert len(_CapturingProc.calls) == 2 # the HW attempt, then the SW respawn
|
||||
(hw_cmd, *_), _ = _CapturingProc.calls[0]
|
||||
(sw_cmd, *_), sw_kwargs = _CapturingProc.calls[1]
|
||||
assert _venc(hw_cmd) == "h264_vaapi"
|
||||
assert _venc(sw_cmd) == "libx264"
|
||||
if os.name == "posix":
|
||||
assert sw_kwargs.get("preexec_fn") is stream._sw_child_preexec # the SW child is niced
|
||||
|
||||
def test_a_hw_TONEMAP_session_that_produces_nothing_falls_back_to_software(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
# The fallback gate is `not ready and s.hw`, shared by plain-HW and tonemap sessions — but only
|
||||
# the plain-HW variant was covered. Pin the tonemap variant too: a (True, True) session that
|
||||
# emits nothing (an OpenCL/tonemap init refusal) must respawn on libx264 and end hw=False,
|
||||
# tonemap=False, so an HDR item still plays on CPU.
|
||||
monkeypatch.setattr(stream, "_select_encode", lambda db, item, transcode, src: (True, True))
|
||||
seen = {"n": 0}
|
||||
|
||||
def flaky_wait(path, timeout=20.0, proc=None):
|
||||
seen["n"] += 1
|
||||
return (
|
||||
seen["n"] > 1
|
||||
) # the first seg-0 wait (HW tonemap) fails; every wait after succeeds
|
||||
|
||||
monkeypatch.setattr(stream, "wait_for", flaky_wait)
|
||||
s = stream.start_session(
|
||||
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
|
||||
)
|
||||
assert s is not None and s.transcode is True and s.hw is False and s.tonemap is False
|
||||
assert len(_CapturingProc.calls) == 2 # the HW tonemap attempt, then the SW respawn
|
||||
(hw_cmd, *_), _ = _CapturingProc.calls[0]
|
||||
(sw_cmd, *_), _ = _CapturingProc.calls[1]
|
||||
assert (
|
||||
"tonemap_opencl=" in hw_cmd[hw_cmd.index("-vf") + 1]
|
||||
) # first attempt built the OpenCL graph
|
||||
assert _venc(sw_cmd) == "libx264" # respawn is plain software, no tonemap
|
||||
|
||||
def test_a_superseded_hw_session_does_not_respawn_over_the_newer_one(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
# F2/race: start_session waits on seg_0 OUTSIDE _lock. If a concurrent same-key start (a rapid
|
||||
# seek-restart) or _enforce_cap replaces _sessions[key] while we wait, our proc dying must NOT
|
||||
# trigger the SW fallback — that would tear down the NEWER session and breach the transcode cap.
|
||||
# The identity guard must detect it's been superseded and yield the currently-registered session.
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (True, False)
|
||||
)
|
||||
|
||||
class _Newer: # stands in for the session a concurrent _spawn registered
|
||||
hw = True
|
||||
tonemap = False
|
||||
transcode = True
|
||||
last_access = 0.0
|
||||
|
||||
newer = _Newer()
|
||||
|
||||
def wait_then_supersede(path, timeout=20.0, proc=None):
|
||||
# Simulate the concurrent replacement happening during our out-of-lock wait, then report the
|
||||
# (now-killed) HW proc produced nothing.
|
||||
stream._sessions["12345"] = newer
|
||||
return False
|
||||
|
||||
monkeypatch.setattr(stream, "wait_for", wait_then_supersede)
|
||||
s = stream.start_session(
|
||||
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
|
||||
)
|
||||
# No SW respawn — only the initial HW _spawn ran; the newer session was left intact.
|
||||
assert len(_CapturingProc.calls) == 1
|
||||
assert s is newer # yielded the currently-registered session, did not clobber it
|
||||
assert stream._sessions["12345"] is newer
|
||||
|
||||
def test_a_cap_culled_hw_session_does_not_respawn_and_is_not_reported_missing(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
# F2/race, the cull sub-case: while we wait on seg_0 out of lock, cross-key cap pressure lets
|
||||
# _enforce_cap evict THIS session's key entirely (no replacement). The fallback must still not
|
||||
# respawn (that would re-breach the cap the cull just enforced), and must NOT return None — a None
|
||||
# is turned by the route into a dishonest "file not found" 404 for a file that exists. It yields
|
||||
# our own (now-dead) session instead, so the client's segment requests degrade to the retryable
|
||||
# "not available yet" path.
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (True, False)
|
||||
)
|
||||
captured = {}
|
||||
|
||||
def wait_then_cull(path, timeout=20.0, proc=None):
|
||||
captured["s"] = stream._sessions[
|
||||
"12345"
|
||||
] # the HW session the initial _spawn just registered
|
||||
del stream._sessions[
|
||||
"12345"
|
||||
] # simulate _enforce_cap dropping the LRU under cap pressure
|
||||
return False
|
||||
|
||||
monkeypatch.setattr(stream, "wait_for", wait_then_cull)
|
||||
s = stream.start_session(
|
||||
db=None, item=_FakeItem(codec_video="hevc"), start_s=0.0, transcode=True
|
||||
)
|
||||
assert (
|
||||
len(_CapturingProc.calls) == 1
|
||||
) # no SW respawn — the cull is respected, cap not re-breached
|
||||
assert (
|
||||
s is captured["s"]
|
||||
) # our own session yielded, NOT None (which the route would 404 as missing)
|
||||
assert "12345" not in stream._sessions # the culled key stays gone
|
||||
|
||||
def test_remux_reads_no_transcode_knobs_and_stream_copies(self, capturing_env, monkeypatch):
|
||||
# A remux start must stream-copy the video and spawn no niced child (the transcode knobs and
|
||||
# the preexec nice are gated behind `transcode=True`).
|
||||
monkeypatch.setattr(stream, "_want_hw", lambda db, item, transcode: False)
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
|
||||
)
|
||||
s = stream.start_session(db=None, item=_FakeItem(), start_s=0.0) # transcode defaults False
|
||||
assert s is not None and s.transcode is False
|
||||
(cmd, *_), kwargs = _CapturingProc.calls[0]
|
||||
assert _venc(cmd) == "copy"
|
||||
assert kwargs.get("preexec_fn") is None
|
||||
|
||||
|
||||
class TestProbeVideoKind:
|
||||
"""The ffprobe-based source classifier that routes the transcode pipeline: 8-bit SDR → plain VAAPI;
|
||||
true-HDR (PQ/HLG) → the OpenCL tonemap graph; 10-bit *SDR* → software (an exact downconvert, NEVER
|
||||
the HDR tone curve — the F1 distinction). Exercises the REAL function body (not a monkeypatch)."""
|
||||
|
||||
@staticmethod
|
||||
def _probe(monkeypatch, stdout):
|
||||
class _R:
|
||||
def __init__(self):
|
||||
self.stdout = stdout
|
||||
|
||||
monkeypatch.setattr(stream.subprocess, "run", lambda *a, **k: _R())
|
||||
|
||||
def test_8bit_sdr_is_hwsafe(self, monkeypatch):
|
||||
self._probe(monkeypatch, "pix_fmt=yuv420p\nbits_per_raw_sample=8\ncolor_transfer=bt709\n")
|
||||
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
|
||||
|
||||
def test_10bit_sdr_pix_fmt_is_sdr10(self, monkeypatch):
|
||||
# yuv420p10le with a non-HDR transfer (bt2020-10 is NOT PQ/HLG) = 10-bit SDR → software, not
|
||||
# tonemap. Signalled by the pix_fmt suffix even when bits_per_raw_sample is absent.
|
||||
self._probe(monkeypatch, "pix_fmt=yuv420p10le\ncolor_transfer=bt2020-10\n")
|
||||
assert stream._probe_video_kind(Path("/x")) == stream._VK_SDR10
|
||||
|
||||
def test_bits_per_raw_sample_over_8_sdr_is_sdr10(self, monkeypatch):
|
||||
self._probe(monkeypatch, "pix_fmt=yuv420p\nbits_per_raw_sample=10\ncolor_transfer=bt709\n")
|
||||
assert stream._probe_video_kind(Path("/x")) == stream._VK_SDR10
|
||||
|
||||
def test_pq_transfer_is_hdr(self, monkeypatch):
|
||||
# PQ (smpte2084) is true HDR — needs a real tonemap, even if only 8-bit-tagged.
|
||||
self._probe(
|
||||
monkeypatch, "pix_fmt=yuv420p10le\nbits_per_raw_sample=10\ncolor_transfer=smpte2084\n"
|
||||
)
|
||||
assert stream._probe_video_kind(Path("/x")) == stream._VK_HDR
|
||||
|
||||
def test_hlg_transfer_is_hdr(self, monkeypatch):
|
||||
# HLG (arib-std-b67) is the other HDR transfer — previously untested; it must classify as HDR.
|
||||
self._probe(
|
||||
monkeypatch,
|
||||
"pix_fmt=yuv420p10le\nbits_per_raw_sample=10\ncolor_transfer=arib-std-b67\n",
|
||||
)
|
||||
assert stream._probe_video_kind(Path("/x")) == stream._VK_HDR
|
||||
|
||||
def test_empty_probe_output_is_hwsafe(self, monkeypatch):
|
||||
# A probe that returns nothing (no streams / stripped metadata) → assume hwsafe; the runtime
|
||||
# HW→SW fallback backstops a wrong guess.
|
||||
self._probe(monkeypatch, "")
|
||||
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
|
||||
|
||||
def test_probe_failure_assumes_hwsafe(self, monkeypatch):
|
||||
# ffprobe missing/erroring → assume hwsafe; the runtime HW→SW fallback is the real backstop, so a
|
||||
# probe miss costs at most one wasted HW attempt, never a stranded item.
|
||||
def boom(*a, **k):
|
||||
raise OSError("no ffprobe")
|
||||
|
||||
monkeypatch.setattr(stream.subprocess, "run", boom)
|
||||
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
|
||||
|
||||
|
||||
class TestOpenclPresent:
|
||||
"""The OpenCL-runtime probe (_opencl_present) — the precondition for the HW HDR tonemap graph. Drives
|
||||
the REAL glob against a temp ICD dir instead of monkeypatching the function, so a regression to the
|
||||
ICD path / glob / OSError guard would be caught (this routes every HDR source on prod)."""
|
||||
|
||||
def test_true_when_an_icd_file_is_registered(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(stream, "_OPENCL_ICD_DIR", tmp_path)
|
||||
(tmp_path / "intel.icd").write_text("libintelocl.so\n")
|
||||
assert stream._opencl_present() is True
|
||||
|
||||
def test_false_when_the_dir_is_empty(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(stream, "_OPENCL_ICD_DIR", tmp_path)
|
||||
assert stream._opencl_present() is False
|
||||
|
||||
def test_false_and_no_raise_when_the_dir_is_missing(self, tmp_path, monkeypatch):
|
||||
# A missing dir: Path.glob yields nothing (no raise) → False.
|
||||
monkeypatch.setattr(stream, "_OPENCL_ICD_DIR", tmp_path / "nope")
|
||||
assert stream._opencl_present() is False
|
||||
|
||||
def test_false_when_glob_raises_oserror(self, monkeypatch):
|
||||
# The `except OSError` guard proper: an unreadable ICD dir (permission denied) makes glob RAISE,
|
||||
# not return empty. That must be swallowed to False, never propagated up through _select_encode
|
||||
# into start_session (which would crash HDR routing).
|
||||
class _BoomDir:
|
||||
def glob(self, pattern):
|
||||
raise OSError("permission denied")
|
||||
|
||||
monkeypatch.setattr(stream, "_OPENCL_ICD_DIR", _BoomDir())
|
||||
assert stream._opencl_present() is False
|
||||
|
||||
|
||||
class TestWaitForProcFastFail:
|
||||
"""wait_for(proc=…) must stop as soon as the ffmpeg proc exits (a VAAPI init refusal dies in ~1s
|
||||
producing nothing) instead of blocking the whole timeout — that fast-fail is what keeps the HW→SW
|
||||
fallback from turning into a 25s stall. Exercises the REAL wait_for (every other test monkeypatches
|
||||
it wholesale, so this branch would otherwise never run)."""
|
||||
|
||||
class _DeadProc:
|
||||
def poll(self):
|
||||
return 1 # already exited non-zero
|
||||
|
||||
def test_returns_true_when_the_segment_exists_despite_a_dead_proc(self, tmp_path):
|
||||
seg = tmp_path / "seg_0.ts"
|
||||
seg.write_bytes(b"x")
|
||||
# The proc exited but DID write a first segment → ready. (The final existence re-check.)
|
||||
assert stream.wait_for(seg, timeout=5.0, proc=self._DeadProc()) is True
|
||||
|
||||
def test_returns_false_fast_when_a_dead_proc_wrote_nothing(self, tmp_path):
|
||||
# Missing segment + dead proc → give up immediately, not after `timeout`. Assert it's fast by
|
||||
# using a long timeout the test would visibly hang on if the fast-fail branch regressed.
|
||||
seg = tmp_path / "seg_0.ts"
|
||||
assert stream.wait_for(seg, timeout=30.0, proc=self._DeadProc()) is False
|
||||
|
||||
def test_empty_segment_from_a_dead_proc_is_not_ready(self, tmp_path):
|
||||
# A zero-byte segment is not a real segment (size > 0 required).
|
||||
seg = tmp_path / "seg_0.ts"
|
||||
seg.write_bytes(b"")
|
||||
assert stream.wait_for(seg, timeout=30.0, proc=self._DeadProc()) is False
|
||||
|
||||
|
||||
class TestAudioLoudnorm:
|
||||
"""`plex_audio_loudnorm` (bug hunt 2026-09, item 14): the HLS path always downmixes 5.1 to stereo
|
||||
AAC and never normalised it, so a cinema mix arrives with whispered dialogue and deafening action.
|
||||
The knob is OFF by default — it makes a film ~11 dB louder and narrows its dynamic range, which is
|
||||
a taste call. Measured on a real 5.1 AC3 film: I -27.6 -> -16.6 LUFS, LRA 13.3 -> 9.8 LU.
|
||||
|
||||
Reuses TestTranscodeStartSession's capturing fixture: what matters is the argv ffmpeg is actually
|
||||
spawned with, not `_ffmpeg_cmd` called with literals."""
|
||||
|
||||
capturing_env = TestTranscodeStartSession.capturing_env
|
||||
|
||||
@staticmethod
|
||||
def _spawn(monkeypatch, *, on: bool, multi: bool = False):
|
||||
monkeypatch.setattr(stream, "_loudnorm_on", on, raising=False)
|
||||
monkeypatch.setattr(
|
||||
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
|
||||
)
|
||||
if multi:
|
||||
monkeypatch.setattr(stream, "_probe_audio_langs", lambda src: ["hun", "eng"])
|
||||
s = stream.start_session(db=None, item=_FakeItem(), start_s=0.0, multi=multi)
|
||||
assert s is not None
|
||||
(cmd, *_), _ = _CapturingProc.calls[0]
|
||||
return s, cmd
|
||||
|
||||
def test_the_knob_off_leaves_the_audio_exactly_as_it_was(self, capturing_env, monkeypatch):
|
||||
_, cmd = self._spawn(monkeypatch, on=False)
|
||||
assert "-af" not in cmd
|
||||
assert cmd[cmd.index("-c:a") + 1] == "aac" and cmd[cmd.index("-ac") + 1] == "2"
|
||||
|
||||
def test_the_knob_on_normalises_AFTER_the_downmix(self, capturing_env, monkeypatch):
|
||||
# Order is the whole point: measuring 5.1 and folding it down afterwards would normalise
|
||||
# something other than what the viewer hears.
|
||||
_, cmd = self._spawn(monkeypatch, on=True)
|
||||
af = cmd[cmd.index("-af") + 1]
|
||||
stages = af.split(",")
|
||||
assert stages[0] == "aformat=channel_layouts=stereo"
|
||||
assert stages[1].startswith("loudnorm=")
|
||||
# loudnorm runs its own 192 kHz resampler internally; the AAC encoder needs 48 kHz back.
|
||||
assert stages[-1] == "aresample=48000"
|
||||
|
||||
def test_every_audio_rendition_of_a_multi_audio_file_is_normalised(
|
||||
self, capturing_env, monkeypatch
|
||||
):
|
||||
# A multi-audio session emits one AAC output per language. `-af` without a stream index
|
||||
# covers all of them — a per-stream filter would have normalised only the first.
|
||||
_, cmd = self._spawn(monkeypatch, on=True, multi=True)
|
||||
assert cmd.count("-map") == 3 # video + two audio tracks
|
||||
assert cmd.count("-af") == 1
|
||||
assert "loudnorm=" in cmd[cmd.index("-af") + 1]
|
||||
|
||||
def test_the_session_directory_records_the_setting(self, capturing_env, monkeypatch):
|
||||
# The knob changes the AUDIO THE SEGMENTS CONTAIN, so it belongs in the directory name with
|
||||
# the other parameters that do. Otherwise a session left from before the flip is reused.
|
||||
off, _ = self._spawn(monkeypatch, on=False)
|
||||
_CapturingProc.calls = []
|
||||
on, _ = self._spawn(monkeypatch, on=True)
|
||||
assert off.dir.name != on.dir.name
|
||||
assert on.dir.name.endswith("-ln_+0.00")
|
||||
|
||||
@pytest.mark.parametrize("on", [False, True])
|
||||
def test_the_orphan_sweep_still_recognises_its_own_directory(
|
||||
self, capturing_env, monkeypatch, on
|
||||
):
|
||||
# The tag is appended with a DASH, not an underscore, precisely so `_SESSION_DIR_RE` (the only
|
||||
# thing keeping the sweep off a shared scratch path) still matches. A session directory the
|
||||
# sweep no longer recognises is one that never gets cleaned up.
|
||||
s, _ = self._spawn(monkeypatch, on=on)
|
||||
assert _SESSION_DIR_RE.match(s.dir.name)
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""The subtitle routes must refuse a session the rest of the app refuses.
|
||||
|
||||
They used to sit behind `_require_session`, the cookie-only gate the poster proxies use — a signed
|
||||
cookie was enough, with none of `current_user`'s checks (is_active, is_suspended, the session epoch
|
||||
that "log out everywhere" bumps). Fine for a poster; not for a route that starts reads of the media
|
||||
library. Found by the 2026-09 bug-hunt deep-review; the gate itself pre-dates that round.
|
||||
"""
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import app.auth as auth
|
||||
import app.state as app_state
|
||||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.plex import subtitles
|
||||
from app.routes.plex import playback
|
||||
|
||||
|
||||
class _NoPool:
|
||||
def __init__(self) -> None:
|
||||
self.jobs: list = []
|
||||
|
||||
def submit(self, fn, *args) -> None:
|
||||
self.jobs.append((fn, args))
|
||||
|
||||
|
||||
def test_a_suspended_account_cannot_drive_the_subtitle_extractor(
|
||||
db, make_user, override_dep, monkeypatch, tmp_path
|
||||
):
|
||||
user = make_user(is_suspended=True)
|
||||
override_dep(get_db, lambda: db)
|
||||
monkeypatch.setattr(app_state, "_configured_cache", True)
|
||||
# A validly SIGNED cookie for that account — which is all the old gate looked at.
|
||||
for mod in (auth, playback):
|
||||
monkeypatch.setattr(mod, "resolved_user_id", lambda request: (user.id, True))
|
||||
monkeypatch.setattr(subtitles, "CACHE_DIR", tmp_path)
|
||||
pool = _NoPool()
|
||||
monkeypatch.setattr(subtitles, "_WIN_POOL", pool)
|
||||
monkeypatch.setattr(subtitles, "_FULL_POOL", pool)
|
||||
subtitles.reset_state()
|
||||
|
||||
client = TestClient(app)
|
||||
assert client.get("/api/plex/subtitle/98614/0/status?at=600").status_code == 401
|
||||
assert client.get("/api/plex/subtitle/98614/0").status_code == 401
|
||||
assert pool.jobs == [], "a refused session still scheduled an extraction"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,63 @@
|
||||
"""A progress checkpoint takes a FINITE position.
|
||||
|
||||
The JSON parser under FastAPI accepts `Infinity` and `NaN`, and both progress handlers truncate the
|
||||
position with `int()` — so one hand-written checkpoint came back as a 500 with a traceback. Found in
|
||||
the 2026-09 bug hunt's second review round, as the sibling of the subtitle routes' non-finite
|
||||
`at`/`offset`. Pure: the body is refused before the handler touches the database.
|
||||
"""
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import app.state as app_state
|
||||
from app.auth import current_user
|
||||
from app.main import app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(monkeypatch, override_dep) -> TestClient:
|
||||
override_dep(current_user, lambda: SimpleNamespace(id=1))
|
||||
monkeypatch.setattr(app_state, "_configured_cache", True) # pass the instance-setup gate, no DB
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["/api/videos/v1/progress", "/api/plex/item/98614/progress"])
|
||||
@pytest.mark.parametrize("raw", ["Infinity", "-Infinity", "NaN"])
|
||||
def test_a_non_finite_position_is_refused(client, path, raw):
|
||||
r = client.post(
|
||||
path,
|
||||
content=f'{{"position_seconds": {raw}, "duration_seconds": 3000}}',
|
||||
headers={"content-type": "application/json"},
|
||||
)
|
||||
assert r.status_code == 422, r.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["/api/videos/v1/progress", "/api/plex/item/98614/progress"])
|
||||
@pytest.mark.parametrize("field", ["position_seconds", "duration_seconds"])
|
||||
def test_a_position_past_the_int4_column_is_refused_not_a_500(client, path, field):
|
||||
"""Finite but past the int4 column: the flush was a DataError, i.e. a 500 with a traceback."""
|
||||
body = {"position_seconds": 10, "duration_seconds": 3000, field: 2**31}
|
||||
assert client.post(path, json=body).status_code == 422
|
||||
|
||||
|
||||
def test_the_refusal_keeps_the_shape_the_ui_reads(client):
|
||||
"""The handler replaces FastAPI's own for every validation error in the app, and the UI turns
|
||||
exactly this shape — a LIST of errors — into its localized "check your input" sentence. The
|
||||
refused value is echoed by its name, because JSON cannot carry NaN."""
|
||||
r = client.post(
|
||||
"/api/videos/v1/progress",
|
||||
content='{"position_seconds": NaN, "duration_seconds": 3000}',
|
||||
headers={"content-type": "application/json"},
|
||||
)
|
||||
detail = r.json()["detail"]
|
||||
assert isinstance(detail, list) and detail[0]["loc"] == ["body", "position_seconds"]
|
||||
assert detail[0]["input"] == "nan"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("q", ["start=inf", "start=nan", "aoff=inf", "aoff=-inf", "aoff=nan"])
|
||||
def test_a_session_at_an_impossible_offset_is_refused(client, q):
|
||||
"""`start=inf` reached `int()` in the session builder (a 500); `aoff=inf` spawned an ffmpeg
|
||||
with a non-finite offset."""
|
||||
assert client.post(f"/api/plex/stream/98614/session?{q}").status_code == 422
|
||||
@@ -0,0 +1,65 @@
|
||||
"""The sliding-window limiter behind /auth/register, password-login, password-reset and demo.
|
||||
|
||||
Pure logic, no DB. `reset()` in particular is only ever called by a test fixture (conftest clears
|
||||
the module-level limiters between tests), so without a test here a refactor could turn it into a
|
||||
no-op and nothing would notice until an unrelated test started failing with a 429.
|
||||
"""
|
||||
|
||||
from app import ratelimit
|
||||
from app.ratelimit import RateLimiter
|
||||
|
||||
|
||||
def test_allow_trips_at_the_cap_and_stays_tripped():
|
||||
lim = RateLimiter(max_events=2, window_seconds=300)
|
||||
|
||||
assert lim.allow("ip") is True
|
||||
assert lim.allow("ip") is True
|
||||
assert lim.allow("ip") is False
|
||||
assert lim.allow("ip") is False
|
||||
|
||||
|
||||
def test_hammering_while_blocked_does_not_push_the_window_forward(monkeypatch):
|
||||
# A blocked attempt must not be recorded, otherwise someone who keeps retrying holds their own
|
||||
# window open forever — an indefinite lockout for anyone sharing that IP. Only observable across
|
||||
# the window boundary, so drive the clock: two attempts fill the cap, ten blocked retries land
|
||||
# just before it expires, and the caller must still be let back in on the far side.
|
||||
clock = {"t": 1000.0}
|
||||
monkeypatch.setattr(ratelimit.time, "monotonic", lambda: clock["t"])
|
||||
lim = RateLimiter(max_events=2, window_seconds=60)
|
||||
|
||||
assert lim.allow("ip") is True
|
||||
assert lim.allow("ip") is True
|
||||
for _ in range(10):
|
||||
clock["t"] += 5 # still inside the window; every one of these is refused
|
||||
assert lim.allow("ip") is False
|
||||
|
||||
clock["t"] = 1000.0 + 61 # past the window of the two attempts that actually counted
|
||||
assert lim.allow("ip") is True
|
||||
|
||||
|
||||
def test_keys_are_independent():
|
||||
lim = RateLimiter(max_events=1, window_seconds=300)
|
||||
|
||||
assert lim.allow("a") is True
|
||||
assert lim.allow("a") is False
|
||||
assert lim.allow("b") is True
|
||||
|
||||
|
||||
def test_reset_clears_every_key():
|
||||
lim = RateLimiter(max_events=1, window_seconds=300)
|
||||
lim.allow("a")
|
||||
lim.allow("b")
|
||||
assert lim.allow("a") is False
|
||||
|
||||
lim.reset()
|
||||
|
||||
assert lim.allow("a") is True
|
||||
assert lim.allow("b") is True
|
||||
|
||||
|
||||
def test_expired_attempts_fall_out_of_the_window():
|
||||
# A zero-length window means every previous attempt is already outside it.
|
||||
lim = RateLimiter(max_events=1, window_seconds=0)
|
||||
|
||||
assert lim.allow("ip") is True
|
||||
assert lim.allow("ip") is True
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user