Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c85ae42b31 | ||
|
|
cb9a89682a | ||
|
|
b842e4e871 | ||
|
|
5082636f15 | ||
|
|
13b7577d9f | ||
|
|
fad656988a | ||
|
|
3021be5166 | ||
|
|
c288ba147b | ||
|
|
fe6308d844 | ||
|
|
24448e7c6f | ||
|
|
d13483a593 | ||
|
|
3bab2e7dc7 | ||
|
|
1b1e3cdcf2 | ||
|
|
27c2fcbd2e | ||
|
|
911527a548 | ||
|
|
8fd91b7b15 | ||
|
|
45c0ee7d58 | ||
|
|
4b1f6d7ec2 | ||
|
|
51bf780100 | ||
|
|
3ada34073b | ||
|
|
d74f2a9ca1 | ||
|
|
a8dddc8c24 | ||
|
|
460bdec406 | ||
|
|
66181d1394 | ||
|
|
0a1fef28ac | ||
|
|
fe572d9207 | ||
|
|
c8247c0dde | ||
|
|
722c37a6d0 | ||
|
|
5ba5679239 | ||
|
|
926e5e9899 | ||
|
|
9251918e08 | ||
|
|
2b6d968833 | ||
|
|
95e6b2c3d9 | ||
|
|
80c55fbf1b | ||
|
|
011f269152 | ||
|
|
dcb244dd04 | ||
|
|
08a9f4e255 | ||
|
|
1673134c36 | ||
|
|
27427eef32 | ||
|
|
a248ca6c13 | ||
|
|
620f84dba4 | ||
|
|
2efa7d5a75 | ||
|
|
827df7494c | ||
|
|
86712b6b0e | ||
|
|
90567718b2 | ||
|
|
c31d7f9d65 | ||
|
|
db56f25387 | ||
|
|
4f58d15d3f | ||
|
|
2c66ce1cf4 | ||
|
|
ce45445d17 | ||
|
|
383d394e32 | ||
|
|
06e8291896 | ||
|
|
db6b60029f | ||
|
|
02134799cc | ||
|
|
c8bc843517 | ||
|
|
759a7536b9 | ||
|
|
b2d34e6e6c |
+5
-3
@@ -68,9 +68,11 @@ SMTP_FROM=
|
||||
|
||||
# --- Scheduler ---
|
||||
# The background scheduler (subscription sync, backfill, enrichment) runs inside the app.
|
||||
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. If you ever run
|
||||
# more than one instance against the same database, keep SCHEDULER_ENABLED=true on exactly one
|
||||
# of them and false on the rest, to avoid double quota use and write races.
|
||||
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. Run ONE app instance
|
||||
# per database: a starting instance closes every run still open as interrupted (it assumes the
|
||||
# previous process is gone), and "is this job already running" is held in the process. A second
|
||||
# instance would cut the first one's runs short and could run the same job twice.
|
||||
# SCHEDULER_ENABLED=false keeps the recurring jobs off; "run now" still works.
|
||||
SCHEDULER_ENABLED=true
|
||||
|
||||
# --- Files library & downloads ---
|
||||
|
||||
@@ -7,6 +7,64 @@ edit the release notes and re-run the generator.
|
||||
|
||||
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
|
||||
|
||||
## 0.108.0 — 2026-10-18
|
||||
|
||||
### New
|
||||
|
||||
- Activity: one admin page for background runs, admin actions, the server log and quota, with one time range, account and search. A run opens to its outcome and its log lines, an admin action to its request's lines; old audit and log links open their tab.
|
||||
- Scheduler: each job shows its last three runs (a run cut short by a restart is marked) and a button opens all of them; turning on a channel's full history is now a run of its own, listed under Ad-hoc runs.
|
||||
- Configuration: how long the run history is kept (Activity › Run history, 180 days by default). Stats keeps your own numbers; the system quota view moved to Activity › Quota.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Configuration: saving a setting with the value already in force now keeps it, instead of dropping the change.
|
||||
- Configuration: the unused SponsorBlock default setting is gone.
|
||||
|
||||
## 0.107.3 — 2026-10-17
|
||||
|
||||
### New
|
||||
|
||||
- Development: the end-to-end suite runs the WCAG A and AA accessibility checks over every page and the player, and every shell script is linted in the gate.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Readability: muted text in the light themes is a shade darker, and the item count, the labels and the selected Files folder sit on a solid tint, so text on the frosted-glass bars stays readable (4.5:1) in every scheme.
|
||||
|
||||
## 0.107.2 — 2026-10-17
|
||||
|
||||
### New
|
||||
|
||||
- Development: the gate script can list its checks (scripts/check.sh --list) and run just one (--only), and the backend tests run on every CPU core, each worker with its own database.
|
||||
|
||||
## 0.107.1 — 2026-10-16
|
||||
|
||||
### Fixed
|
||||
|
||||
- Downloads: a failed download's explanation now says to press Retry, instead of pointing to a download list that no longer exists.
|
||||
|
||||
## 0.107.0 — 2026-10-15
|
||||
|
||||
### New
|
||||
|
||||
- API tokens: Settings › API tokens gives one machine of yours a key to the Files folders you pick — or, in one click, to a Hub folder for swapping files with it. Signing out your other sessions or changing your password revokes every token.
|
||||
- Share by link: a link can now open a whole folder of yours — its page lists what is in it and plays a video above the list.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Uploads: a finished upload is announced once, and shows in Files at once.
|
||||
|
||||
## 0.106.2 — 2026-10-14
|
||||
|
||||
### Fixed
|
||||
|
||||
- Pictures: a thumbnail, poster or channel picture that fails to load is retried, and meanwhile the card shows its empty frame instead of a broken-image mark.
|
||||
|
||||
## 0.106.1 — 2026-10-14
|
||||
|
||||
### Fixed
|
||||
|
||||
- Plex player: clicking outside the Settings panel closes the panel again instead of closing the player.
|
||||
|
||||
## 0.106.0 — 2026-10-13
|
||||
|
||||
### New
|
||||
|
||||
@@ -12,7 +12,7 @@ router-free navigation provider.
|
||||
|
||||
## The gate — never merge or ship red
|
||||
|
||||
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
|
||||
The local gate is **ruff (`check` + `format --check`) + shellcheck + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
|
||||
eslint + prettier --check + knip + vitest + vite build + pytest + pip-audit + npm audit**. It runs
|
||||
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
|
||||
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
|
||||
@@ -55,14 +55,15 @@ how it gets built. Concretely:
|
||||
- **Challenge unsound requests.** If a request is technically ill-advised and you can back it with real
|
||||
arguments, push back or propose an alternative — don't comply blindly.
|
||||
- **Verify UI in the browser before calling it done.** tsc/unit/DOM tests miss visual and interaction
|
||||
bugs; do a real visual check of any UI change, and run DESIGN.md's four-step Conformance check.
|
||||
bugs; do a real visual check of any UI change, and run DESIGN.md's Conformance check (a copy-only
|
||||
change runs its step 4 alone: every state the new text renders in, in both languages).
|
||||
- **Be terse everywhere else.** Outside those decision points, lead with the answer or decision. During
|
||||
implementation and debugging especially: no restating the same point reworded, no filler.
|
||||
|
||||
## Architecture shape (don't fight it)
|
||||
|
||||
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
|
||||
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
|
||||
`downloads/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
|
||||
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
|
||||
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
|
||||
Frontend: `main.tsx` provider tree → `App.tsx` shell (a shared `/watch` link boots from its own
|
||||
@@ -91,8 +92,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
|
||||
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
|
||||
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
|
||||
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
|
||||
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
|
||||
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
|
||||
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` and `jobruns` own their session) |
|
||||
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
|
||||
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
|
||||
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
|
||||
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
|
||||
@@ -101,7 +102,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
|
||||
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
|
||||
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
|
||||
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
|
||||
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
|
||||
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect | a bare `catch {}` or `notify()` with no error attached |
|
||||
| FE floating or full-screen layer | `ui/Overlay` (portals to `<body>`) or the popover engine (`AnchoredPopover` portals itself); stacking by the z-index scale (`z-menu` … `z-popover`, docs/DESIGN.md "Layers") | one rendered in place inside a virtualized row or a masked/transformed scroller — clipped or painted over; a magic `z-[n]` |
|
||||
| FE player on a native media element | `lib/mediaSession` (`useMediaSession`) + `lib/mediaKeys` + `lib/resume` (`useResume`), the element held as STATE | a private `navigator.mediaSession` / `keydown` / position-saving effect per player |
|
||||
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
|
||||
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
|
||||
@@ -114,7 +116,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
|
||||
Not repeated here, because a lint rule refuses the wrong one and its message names the right one:
|
||||
buttons (`ui/Button`, `IconButton`, `Pressable`, `Chip`, `DragHandle`), text fields and native
|
||||
controls (`ui/Input`, `Checkbox`, `Slider`, …), glass surfaces (`SURFACE.*`), a native `<select>`
|
||||
(`SegmentedToggle` / `FilterSelect`), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
|
||||
(`SegmentedToggle` / `FilterSelect`), a picture (`ui/RemoteImage`, never a bare `<img>`), a box or font size (the spacing scale or rem,
|
||||
never a px-frozen `w-[Npx]` / `text-[Npx]`: the text-size slider scales the root font), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
|
||||
|
||||
## House rules that bite (learned from real bugs)
|
||||
|
||||
|
||||
+7
-1
@@ -55,8 +55,14 @@ RUN adduser --disabled-password --gecos "" appuser \
|
||||
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
|
||||
# `apt-get upgrade`: the base image is a snapshot, and the Debian security fixes released since it
|
||||
# was cut are only in the archive — the first Trivy scan found one (pcre2) fixed upstream but not in it.
|
||||
# APT_REFRESH: the upgrade only runs when this layer is rebuilt, and a build cache keeps it for as long
|
||||
# as nothing above it changes — a publish shipped an image whose curl and librsvg were a release behind
|
||||
# fixes already in the archive. The release build passes the day here, so the layer is rebuilt at
|
||||
# least daily; empty (the default, a self-hosted build) leaves the cache alone.
|
||||
ARG DENO_VERSION=v2.9.1
|
||||
RUN apt-get update \
|
||||
ARG APT_REFRESH=
|
||||
RUN echo "apt refresh: ${APT_REFRESH:-cached}" \
|
||||
&& apt-get update \
|
||||
&& apt-get upgrade -y --no-install-recommends \
|
||||
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
|
||||
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
|
||||
|
||||
@@ -630,41 +630,7 @@
|
||||
}
|
||||
}
|
||||
],
|
||||
"./app/routes/admin_logs.py": [
|
||||
{
|
||||
"code": "reportReturnType",
|
||||
"range": {
|
||||
"startColumn": 15,
|
||||
"endColumn": 47,
|
||||
"lineCount": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"./app/routes/audit.py": [
|
||||
{
|
||||
"code": "reportCallIssue",
|
||||
"range": {
|
||||
"startColumn": 17,
|
||||
"endColumn": 98,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportAssignmentType",
|
||||
"range": {
|
||||
"startColumn": 17,
|
||||
"endColumn": 98,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportArgumentType",
|
||||
"range": {
|
||||
"startColumn": 22,
|
||||
"endColumn": 97,
|
||||
"lineCount": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"code": "reportAttributeAccessIssue",
|
||||
"range": {
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
"""Machine access to the Files library: API tokens bound to folders, and the Hub folder
|
||||
|
||||
Revision ID: 0106_api_tokens
|
||||
Revises: 0105_job_source_key
|
||||
Create Date: 2026-10-14
|
||||
|
||||
`api_tokens` is a user's token for a machine (a script, the hub): only its SHA-256 is kept, plus the
|
||||
first characters to recognise it by, and whether its user was told it is about to run out. `api_token_grants` says what it may do in which folder — list,
|
||||
read, write, delete — so a token reaches nothing it was not given, folder by folder.
|
||||
|
||||
A third folder kind, `hub`: the one folder a user exchanges files with their hub machine through.
|
||||
Like a source folder its name is the client's to render; like an own folder a file may be moved in.
|
||||
One per user.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0106_api_tokens"
|
||||
down_revision: str | None = "0105_job_source_key"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
_KIND_OLD = (
|
||||
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
|
||||
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
|
||||
)
|
||||
_KIND_NEW = _KIND_OLD + " OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
|
||||
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_NEW)
|
||||
op.create_index(
|
||||
"uq_file_folder_hub",
|
||||
"file_folders",
|
||||
["user_id"],
|
||||
unique=True,
|
||||
postgresql_where=sa.text("kind = 'hub'"),
|
||||
)
|
||||
op.create_table(
|
||||
"api_tokens",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column(
|
||||
"user_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("users.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
),
|
||||
sa.Column("name", sa.String(80), nullable=False),
|
||||
sa.Column("prefix", sa.String(16), nullable=False),
|
||||
sa.Column("token_hash", sa.String(64), nullable=False, unique=True),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True)),
|
||||
sa.Column("last_used_at", sa.DateTime(timezone=True)),
|
||||
sa.Column("expiry_warned", sa.Boolean(), nullable=False, server_default=sa.text("false")),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
op.create_table(
|
||||
"api_token_grants",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column(
|
||||
"token_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("api_tokens.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"folder_id",
|
||||
sa.Integer(),
|
||||
sa.ForeignKey("file_folders.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
),
|
||||
sa.Column("can_list", sa.Boolean(), nullable=False, server_default=sa.text("false")),
|
||||
sa.Column("can_read", sa.Boolean(), nullable=False, server_default=sa.text("false")),
|
||||
sa.Column("can_write", sa.Boolean(), nullable=False, server_default=sa.text("false")),
|
||||
sa.Column("can_delete", sa.Boolean(), nullable=False, server_default=sa.text("false")),
|
||||
sa.UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
|
||||
sa.CheckConstraint(
|
||||
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("api_token_grants")
|
||||
op.drop_table("api_tokens")
|
||||
op.drop_index("uq_file_folder_hub", table_name="file_folders")
|
||||
# A Hub folder may hold files (their foreign key RESTRICTs a delete): it becomes an own folder,
|
||||
# named "Hub <id>" — or "Hub <id> (2)", … when its user already has a folder of that name,
|
||||
# which `uq_file_folder_own_name` (per user, case-blind) would refuse.
|
||||
conn = op.get_bind()
|
||||
hubs = conn.execute(sa.text("SELECT id, user_id FROM file_folders WHERE kind = 'hub'")).all()
|
||||
for folder_id, user_id in hubs:
|
||||
name, n = f"Hub {folder_id}", 1
|
||||
while conn.execute(
|
||||
sa.text("SELECT 1 FROM file_folders WHERE user_id = :u AND lower(name) = lower(:n)"),
|
||||
{"u": user_id, "n": name},
|
||||
).first():
|
||||
n += 1
|
||||
name = f"Hub {folder_id} ({n})"
|
||||
conn.execute(
|
||||
sa.text("UPDATE file_folders SET kind = 'own', name = :n WHERE id = :i"),
|
||||
{"n": name, "i": folder_id},
|
||||
)
|
||||
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
|
||||
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_OLD)
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Share-link tokens no longer stored in the clear
|
||||
|
||||
Revision ID: 0107_link_token_at_rest
|
||||
Revises: 0106_api_tokens
|
||||
Create Date: 2026-10-15
|
||||
|
||||
A `/watch` link's token is the credential itself, and it sat in `download_links.token` as written,
|
||||
so a copy of the database was every shared file. Now a link is found by the token's SHA-256
|
||||
(`token_hash`), and the token is kept only Fernet-encrypted (`token_enc`, TOKEN_ENCRYPTION_KEY) so its
|
||||
owner can copy the same link again. Without a key the copy is not kept: the link still opens, its
|
||||
address just cannot be shown again (`app/downloads/links.py`).
|
||||
|
||||
Every existing link is converted here, so the addresses already sent keep working. The downgrade
|
||||
writes the tokens back from their encrypted copies; a link with none left (no key, or a key since
|
||||
changed) gets a fresh random token, which is the same as revoking it.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import secrets
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
from app import security
|
||||
|
||||
revision: str = "0107_link_token_at_rest"
|
||||
down_revision: str | None = "0106_api_tokens"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("download_links", sa.Column("token_hash", sa.String(64)))
|
||||
op.add_column("download_links", sa.Column("token_enc", sa.Text()))
|
||||
conn = op.get_bind()
|
||||
sealing = security.can_encrypt()
|
||||
for link_id, token in conn.execute(sa.text("SELECT id, token FROM download_links")).fetchall():
|
||||
conn.execute(
|
||||
sa.text("UPDATE download_links SET token_hash = :h, token_enc = :e WHERE id = :id"),
|
||||
{
|
||||
"id": link_id,
|
||||
"h": hashlib.sha256(token.encode()).hexdigest(),
|
||||
"e": security.encrypt(token) if sealing else None,
|
||||
},
|
||||
)
|
||||
op.alter_column("download_links", "token_hash", nullable=False)
|
||||
op.create_unique_constraint("uq_download_link_token_hash", "download_links", ["token_hash"])
|
||||
op.drop_constraint("uq_download_link_token", "download_links", type_="unique")
|
||||
op.drop_column("download_links", "token")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.add_column("download_links", sa.Column("token", sa.String(64)))
|
||||
conn = op.get_bind()
|
||||
rows = conn.execute(sa.text("SELECT id, token_enc FROM download_links")).fetchall()
|
||||
for link_id, enc in rows:
|
||||
token = security.decrypt_optional(enc) or secrets.token_urlsafe(24)
|
||||
conn.execute(
|
||||
sa.text("UPDATE download_links SET token = :t WHERE id = :id"),
|
||||
{"id": link_id, "t": token},
|
||||
)
|
||||
op.alter_column("download_links", "token", nullable=False)
|
||||
op.create_unique_constraint("uq_download_link_token", "download_links", ["token"])
|
||||
op.drop_constraint("uq_download_link_token_hash", "download_links", type_="unique")
|
||||
op.drop_column("download_links", "token_enc")
|
||||
op.drop_column("download_links", "token_hash")
|
||||
@@ -0,0 +1,40 @@
|
||||
"""A share link may open a folder
|
||||
|
||||
Revision ID: 0108_folder_links
|
||||
Revises: 0107_link_token_at_rest
|
||||
Create Date: 2026-10-15
|
||||
|
||||
A `/watch` link opened one file. Now it may open one of its owner's own folders instead: the link's
|
||||
page lists what is in the folder when it is opened. `folder_id` sits beside `job_id`, exactly one of
|
||||
the two set; a folder's links go with the folder.
|
||||
|
||||
The downgrade has nowhere to keep a folder link, so it deletes them: they stop opening.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0108_folder_links"
|
||||
down_revision: str | None = "0107_link_token_at_rest"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"download_links",
|
||||
sa.Column("folder_id", sa.Integer(), sa.ForeignKey("file_folders.id", ondelete="CASCADE")),
|
||||
)
|
||||
op.create_index("ix_download_links_folder_id", "download_links", ["folder_id"])
|
||||
op.alter_column("download_links", "job_id", nullable=True)
|
||||
op.create_check_constraint(
|
||||
"ck_download_link_target", "download_links", "(job_id IS NULL) <> (folder_id IS NULL)"
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("DELETE FROM download_links WHERE folder_id IS NOT NULL")
|
||||
op.drop_constraint("ck_download_link_target", "download_links", type_="check")
|
||||
op.alter_column("download_links", "job_id", nullable=False)
|
||||
op.drop_index("ix_download_links_folder_id", table_name="download_links")
|
||||
op.drop_column("download_links", "folder_id")
|
||||
@@ -0,0 +1,160 @@
|
||||
"""Background runs get a durable history
|
||||
|
||||
Revision ID: 0109_job_runs
|
||||
Revises: 0108_folder_links
|
||||
Create Date: 2026-10-17
|
||||
|
||||
What a scheduler job last did lived in the API process's memory, gone at every restart; the audit
|
||||
log kept a row for a run only when it was manual, failed, or changed something. Now every run is a
|
||||
`job_runs` row, written when it starts and when it ends. A run's quota spend points at it
|
||||
(`quota_events.job_run_id`), and so do the lines it logged (`log_events.run_id`, no foreign key: the
|
||||
log writer runs behind the event).
|
||||
|
||||
Runs are not admin actions, so the audit log stops carrying them: the existing `job_run` audit rows
|
||||
MOVE into `job_runs` (start = end = when the row was written, the only time it kept), and leave the
|
||||
audit log. The downgrade moves every run back into the audit log as such a row (one still running as
|
||||
interrupted, what the audit row has no column for in its `after`), then drops the table.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision: str = "0109_job_runs"
|
||||
down_revision: str | None = "0108_folder_links"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"job_runs",
|
||||
sa.Column("id", sa.BigInteger(), primary_key=True),
|
||||
sa.Column("job", sa.String(40), nullable=False),
|
||||
sa.Column("trigger", sa.String(16), nullable=False),
|
||||
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="SET NULL")),
|
||||
sa.Column("target", sa.String(128)),
|
||||
sa.Column("status", sa.String(16), nullable=False),
|
||||
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("finished_at", sa.DateTime(timezone=True)),
|
||||
sa.Column("summary", sa.Text()),
|
||||
sa.Column("result", postgresql.JSONB()),
|
||||
sa.Column("error", sa.Text()),
|
||||
sa.Column("failure", postgresql.JSONB()),
|
||||
sa.Column("quota_units", sa.Integer(), nullable=False, server_default="0"),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('running', 'ok', 'error', 'skipped', 'interrupted')",
|
||||
name="ck_job_runs_status",
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"trigger IN ('scheduled', 'manual', 'ad_hoc')", name="ck_job_runs_trigger"
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"(status = 'running') = (finished_at IS NULL)", name="ck_job_runs_finished"
|
||||
),
|
||||
)
|
||||
op.create_index("ix_job_runs_job_started", "job_runs", ["job", "started_at"])
|
||||
op.create_index("ix_job_runs_actor_id", "job_runs", ["actor_id"])
|
||||
op.create_index("ix_job_runs_started_at", "job_runs", ["started_at"])
|
||||
|
||||
op.add_column(
|
||||
"quota_events",
|
||||
sa.Column("job_run_id", sa.BigInteger(), sa.ForeignKey("job_runs.id", ondelete="SET NULL")),
|
||||
)
|
||||
op.create_index("ix_quota_events_job_run_id", "quota_events", ["job_run_id"])
|
||||
|
||||
op.add_column("log_events", sa.Column("run_id", sa.BigInteger()))
|
||||
op.create_index(
|
||||
"ix_log_events_run",
|
||||
"log_events",
|
||||
["run_id"],
|
||||
postgresql_where=sa.text("run_id IS NOT NULL"),
|
||||
)
|
||||
|
||||
# The audit trail's runs move over: the three statuses the scheduler wrote, and `interrupted`,
|
||||
# which a downgrade writes back (as it does a run's trigger, target, spend and start, which the
|
||||
# audit row has no column for: its actor alone cannot say "manual" once that admin is deleted),
|
||||
# so a rollback and a redeploy keep every run as it was. A status the run table does
|
||||
# not know would fail the CHECK.
|
||||
op.execute(
|
||||
"""
|
||||
INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at, finished_at,
|
||||
summary, result, error, failure, quota_units)
|
||||
SELECT target_id,
|
||||
coalesce(after::jsonb ->> 'trigger',
|
||||
CASE WHEN actor_id IS NULL THEN 'scheduled' ELSE 'manual' END),
|
||||
actor_id,
|
||||
after::jsonb ->> 'target',
|
||||
after::jsonb ->> 'status',
|
||||
coalesce((after::jsonb ->> 'started_at')::timestamptz, created_at),
|
||||
created_at,
|
||||
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb ->> 'summary' END,
|
||||
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb -> 'result' END,
|
||||
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb ->> 'summary' END,
|
||||
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb -> 'result' END,
|
||||
coalesce((after::jsonb ->> 'quota_units')::int, 0)
|
||||
FROM audit_log
|
||||
WHERE action = 'job_run'
|
||||
AND target_id IS NOT NULL
|
||||
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
|
||||
ORDER BY id
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM audit_log
|
||||
WHERE action = 'job_run'
|
||||
AND target_id IS NOT NULL
|
||||
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# A run still `running` is one the rollback stopped: it goes back as interrupted. What the audit
|
||||
# row's own columns cannot say rides in `after` (its trigger, target, spend and start), so the
|
||||
# upgrade restores them.
|
||||
op.execute(
|
||||
"""
|
||||
INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after, created_at)
|
||||
SELECT actor_id,
|
||||
'job_run',
|
||||
'job',
|
||||
job,
|
||||
left(job || ': ' || st
|
||||
|| coalesce(' — ' || CASE WHEN st = 'error' THEN error ELSE summary END, ''),
|
||||
255),
|
||||
(jsonb_build_object(
|
||||
'status', st,
|
||||
'summary', CASE WHEN st = 'error' THEN error ELSE summary END)
|
||||
|| CASE
|
||||
WHEN st = 'error' AND failure IS NOT NULL
|
||||
THEN jsonb_build_object('result', failure)
|
||||
WHEN st <> 'error' AND result IS NOT NULL
|
||||
THEN jsonb_build_object('result', result)
|
||||
ELSE '{}'::jsonb
|
||||
END
|
||||
|| jsonb_build_object('trigger', trigger, 'quota_units', units,
|
||||
'started_at', started_at)
|
||||
|| CASE WHEN target IS NOT NULL
|
||||
THEN jsonb_build_object('target', target) ELSE '{}'::jsonb END)::json,
|
||||
coalesce(finished_at, started_at)
|
||||
FROM (SELECT *, CASE WHEN status = 'running' THEN 'interrupted' ELSE status END AS st,
|
||||
-- A run stopped mid-way has not summed its spend yet: sum it as the startup
|
||||
-- sweep would, before the link to it is dropped.
|
||||
CASE WHEN status = 'running'
|
||||
THEN (SELECT coalesce(sum(q.units), 0) FROM quota_events q
|
||||
WHERE q.job_run_id = job_runs.id)
|
||||
ELSE quota_units END AS units
|
||||
FROM job_runs) AS r
|
||||
ORDER BY id
|
||||
"""
|
||||
)
|
||||
op.drop_index("ix_log_events_run", table_name="log_events")
|
||||
op.drop_column("log_events", "run_id")
|
||||
op.drop_index("ix_quota_events_job_run_id", table_name="quota_events")
|
||||
op.drop_column("quota_events", "job_run_id")
|
||||
op.drop_index("ix_job_runs_started_at", table_name="job_runs")
|
||||
op.drop_index("ix_job_runs_actor_id", table_name="job_runs")
|
||||
op.drop_index("ix_job_runs_job_started", table_name="job_runs")
|
||||
op.drop_table("job_runs")
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Job intervals move into sysconfig; the unread SponsorBlock default goes
|
||||
|
||||
Revision ID: 0110_intervals_in_sysconfig
|
||||
Revises: 0109_job_runs
|
||||
Create Date: 2026-10-18
|
||||
|
||||
An admin's interval override lived in its own table, `scheduler_settings`, beside the one store every
|
||||
other admin knob uses. Each override becomes the `job_interval_<job>` sysconfig key and the table
|
||||
goes. Only the jobs that still exist are copied: a row for any other id never did anything.
|
||||
|
||||
The `sponsorblock_default` setting was offered on the Configuration page but nothing ever read it, so
|
||||
its key is gone and its stored value with it — the downgrade cannot bring that value back.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0110_intervals_in_sysconfig"
|
||||
down_revision: str | None = "0109_job_runs"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
# The interval jobs as of this revision (a migration does not read the live code's list).
|
||||
_JOBS = (
|
||||
"rss_poll",
|
||||
"enrich",
|
||||
"backfill",
|
||||
"autotag",
|
||||
"shorts",
|
||||
"subscriptions",
|
||||
"playlist_sync",
|
||||
"maintenance",
|
||||
"explore_cleanup",
|
||||
"download_gc",
|
||||
"plex_sync",
|
||||
"plex_watch_sync",
|
||||
"plex_watch_reconcile",
|
||||
"tmdb_enrich",
|
||||
"people_enrich",
|
||||
"audit_gc",
|
||||
"log_gc",
|
||||
)
|
||||
_PREFIX = "job_interval_"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
sa.text(
|
||||
"INSERT INTO system_config (key, value, encrypted)"
|
||||
" SELECT :prefix || job_id, interval_minutes::text, false FROM scheduler_settings"
|
||||
" WHERE interval_minutes IS NOT NULL AND job_id IN :jobs"
|
||||
" ON CONFLICT (key) DO NOTHING"
|
||||
).bindparams(sa.bindparam("jobs", expanding=True), prefix=_PREFIX, jobs=list(_JOBS))
|
||||
)
|
||||
op.drop_table("scheduler_settings")
|
||||
op.execute("DELETE FROM system_config WHERE key = 'sponsorblock_default'")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.create_table(
|
||||
"scheduler_settings",
|
||||
sa.Column("job_id", sa.String(40), primary_key=True),
|
||||
sa.Column("interval_minutes", sa.Integer(), nullable=False),
|
||||
)
|
||||
op.execute(
|
||||
sa.text(
|
||||
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
|
||||
" SELECT substr(key, :start), value::int FROM system_config WHERE key IN :keys"
|
||||
).bindparams(
|
||||
sa.bindparam("keys", expanding=True),
|
||||
start=len(_PREFIX) + 1,
|
||||
keys=[_PREFIX + j for j in _JOBS],
|
||||
)
|
||||
)
|
||||
op.execute(
|
||||
sa.text("DELETE FROM system_config WHERE key IN :keys").bindparams(
|
||||
sa.bindparam("keys", expanding=True), keys=[_PREFIX + j for j in _JOBS]
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,28 @@
|
||||
"""An audit row keeps the request that made it
|
||||
|
||||
Revision ID: 0111_audit_request_id
|
||||
Revises: 0110_intervals_in_sysconfig
|
||||
Create Date: 2026-10-18
|
||||
|
||||
The activity page opens an admin action's log lines from its row: `audit_log.request_id` is the
|
||||
request the change was made in, the same id `log_events.request_id` carries. Older rows have none —
|
||||
the request that made them was never recorded.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0111_audit_request_id"
|
||||
down_revision: str | None = "0110_intervals_in_sysconfig"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("audit_log", sa.Column("request_id", sa.String(64)))
|
||||
op.create_index("ix_audit_log_request_id", "audit_log", ["request_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_audit_log_request_id", table_name="audit_log")
|
||||
op.drop_column("audit_log", "request_id")
|
||||
@@ -20,7 +20,9 @@ Three layers, by *who* runs the code — not by feature:
|
||||
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline;
|
||||
`issues/bundle.py` is the one definition of a bug report's files, for the zip export and the tracker.
|
||||
|
||||
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
|
||||
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth` (with `apitokens`, a machine's
|
||||
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`, `jobruns` (every
|
||||
background run's durable row, its words in `runstatus`),
|
||||
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
|
||||
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
|
||||
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
|
||||
@@ -35,7 +37,7 @@ pre-abstract for a single caller; don't leave shared logic duplicated once there
|
||||
## Transaction ownership
|
||||
|
||||
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
|
||||
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
|
||||
(only `quota` and `jobruns` own their session)"** — so read it that way, not as an exception to it. The commit belongs
|
||||
to the **operation owner**; anything called *inside* another transaction only stages its change
|
||||
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
|
||||
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
|
||||
@@ -44,7 +46,7 @@ mutation it records, so an audit entry can never exist without its change (or vi
|
||||
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
|
||||
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
|
||||
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
|
||||
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
|
||||
`jobruns`' run rows — the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
|
||||
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
|
||||
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
|
||||
attribute — see `service.decr_ref_count`.
|
||||
@@ -66,8 +68,9 @@ mutation it records, so an audit entry can never exist without its change (or vi
|
||||
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
|
||||
startup, not user-tunable at runtime.
|
||||
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
|
||||
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
|
||||
slated to fold into `sysconfig` — do not add more like it.)
|
||||
goes in `SPECS`; don't invent a fourth store. Each spec states when a change takes effect
|
||||
(`applies`, required): `live` keys are read where they are used, never at import (test-held). The
|
||||
scheduler's job intervals are `SPECS` keys too, edited on the Scheduler page, not the Configuration page.
|
||||
|
||||
## The gate
|
||||
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
"""Machine access to the Files library: API tokens, and what each one may do where.
|
||||
|
||||
A user makes a token for a machine of theirs — their hub, a script — and it reaches their library as
|
||||
`Authorization: Bearer sfl_…`. Only on `/api/files`: the routes there take `files_caller` instead of
|
||||
`current_user`, and nothing else does (a test holds the route table to that), so a token can never
|
||||
manage the account, the tokens, or anything outside the library. And only in the folders its grants
|
||||
name, with the rights they give: list, read, write, delete. Every route asks the one question,
|
||||
`FilesCaller.may(folder_id, right)`; a signed-in user may everything in their own folders.
|
||||
|
||||
The token is `sfl_` + 32 random bytes in base62 + a CRC32 of them: a mistyped or cut-off token is
|
||||
refused without a database read, and a secret scanner can tell one by its shape. Only its SHA-256 is
|
||||
kept (`security.hash_token`) — the user sees it once, when it is made — with its first characters to
|
||||
tell tokens apart by. It is looked up by that hash on every request: no signed claims, so a deleted
|
||||
token stops working at once. A token about to run out warns its user once, a week before
|
||||
(`warn_expiring`), so a machine does not lose its key unannounced.
|
||||
"""
|
||||
|
||||
import secrets
|
||||
import zlib
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from enum import StrEnum
|
||||
|
||||
from fastapi import Depends, Request
|
||||
from sqlalchemy import delete, or_, select, update
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import audit, debugsession, logs
|
||||
from app.audit import AuditAction
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.models import ApiToken, ApiTokenGrant, User
|
||||
from app.notifications import create_notification
|
||||
from app.ratelimit import RateLimiter, client_key
|
||||
from app.refusal import refused
|
||||
from app.security import hash_token
|
||||
from app.utils import iso
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
PREFIX = "sfl_"
|
||||
_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
|
||||
_BODY_BYTES = 32
|
||||
_BODY_LEN = 43 # 62**43 > 2**256: every 32-byte value fits
|
||||
_CHECK_LEN = 6 # 62**6 > 2**32: every CRC32 fits
|
||||
TOKEN_LEN = len(PREFIX) + _BODY_LEN + _CHECK_LEN
|
||||
# What a token is shown by once it is made: enough to tell a user's tokens apart, far too little to
|
||||
# narrow a guess (8 of its 43 random characters).
|
||||
SHOWN_LEN = len(PREFIX) + 8
|
||||
|
||||
# A machine that runs away (a loop in a script) is held back per token; a caller trying tokens is
|
||||
# held back per client, counting only the refusals.
|
||||
_calls = RateLimiter(600, 60)
|
||||
_bad = RateLimiter(20, 300)
|
||||
# How stale `last_used_at` may get before a request writes it again: a busy script costs one write a
|
||||
# minute, not one per request.
|
||||
_TOUCH_EVERY = timedelta(minutes=1)
|
||||
# How long before a token runs out its user is told (`warn_expiring`); the token list marks it
|
||||
# "expires soon" for the same span.
|
||||
EXPIRY_WARNING = timedelta(days=7)
|
||||
|
||||
|
||||
class Right(StrEnum):
|
||||
LIST = "list"
|
||||
READ = "read"
|
||||
WRITE = "write"
|
||||
DELETE = "delete"
|
||||
|
||||
|
||||
# Each right's column on a grant. A dict over the enum, so a new right without a column fails here.
|
||||
GRANT_COLUMNS = {
|
||||
Right.LIST: "can_list",
|
||||
Right.READ: "can_read",
|
||||
Right.WRITE: "can_write",
|
||||
Right.DELETE: "can_delete",
|
||||
}
|
||||
|
||||
|
||||
def _b62(value: int, width: int) -> str:
|
||||
out = []
|
||||
for _ in range(width):
|
||||
value, digit = divmod(value, 62)
|
||||
out.append(_ALPHABET[digit])
|
||||
return "".join(reversed(out))
|
||||
|
||||
|
||||
def _check(body: str) -> str:
|
||||
return _b62(zlib.crc32(body.encode()), _CHECK_LEN)
|
||||
|
||||
|
||||
def mint() -> str:
|
||||
"""A new token, in full. The caller stores its hash and shows it to the user, once."""
|
||||
body = _b62(int.from_bytes(secrets.token_bytes(_BODY_BYTES)), _BODY_LEN)
|
||||
return PREFIX + body + _check(body)
|
||||
|
||||
|
||||
def well_formed(raw: str) -> bool:
|
||||
"""Whether `raw` has a token's shape and its checksum adds up."""
|
||||
if len(raw) != TOKEN_LEN or not raw.startswith(PREFIX):
|
||||
return False
|
||||
body, check = raw[len(PREFIX) : -_CHECK_LEN], raw[-_CHECK_LEN:]
|
||||
# The check too: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it.
|
||||
if any(c not in _ALPHABET for c in body + check):
|
||||
return False
|
||||
return secrets.compare_digest(_check(body), check)
|
||||
|
||||
|
||||
def grant_rights(grant: ApiTokenGrant) -> frozenset[Right]:
|
||||
return frozenset(r for r, col in GRANT_COLUMNS.items() if getattr(grant, col))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FilesCaller:
|
||||
"""Who is asking the Files library, and where they may do what. `grants` is None for a signed-in
|
||||
user (everything in their own folders); for a token, folder id → its rights there."""
|
||||
|
||||
user: User
|
||||
token_id: int | None = None
|
||||
token_prefix: str | None = None
|
||||
grants: Mapping[int, frozenset[Right]] | None = None
|
||||
|
||||
def may(self, folder_id: int | None, right: Right) -> bool:
|
||||
if self.grants is None:
|
||||
return True
|
||||
return folder_id is not None and right in self.grants.get(folder_id, frozenset())
|
||||
|
||||
def folders_with(self, right: Right) -> set[int] | None:
|
||||
"""The folders `right` is granted in; None = any of the user's own."""
|
||||
if self.grants is None:
|
||||
return None
|
||||
return {f for f, rights in self.grants.items() if right in rights}
|
||||
|
||||
@property
|
||||
def via(self) -> str | None:
|
||||
"""How an audit row says a token did it, None when the user did."""
|
||||
return None if self.token_prefix is None else f"api token {self.token_prefix}…"
|
||||
|
||||
|
||||
def bearer_token(request: Request) -> str | None:
|
||||
"""The token of an `Authorization: Bearer` header; None when the request carries none (another
|
||||
scheme counts as none — it is not ours to judge)."""
|
||||
scheme, _, value = request.headers.get("authorization", "").partition(" ")
|
||||
if scheme.lower() != "bearer":
|
||||
return None
|
||||
return value.strip()
|
||||
|
||||
|
||||
def session_caller(request: Request, db: Session = Depends(get_db)) -> User | None:
|
||||
"""The signed-in user, or None when the request brings a token instead: a token wins over a
|
||||
cookie, so a machine's request is judged by its grants alone, whatever else it carries."""
|
||||
if bearer_token(request) is not None:
|
||||
return None
|
||||
return current_user(request, db)
|
||||
|
||||
|
||||
def files_caller(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
user: User | None = Depends(session_caller),
|
||||
) -> FilesCaller:
|
||||
"""The `/api/files` caller: a signed-in user, or a valid token's user with its grants."""
|
||||
if user is not None:
|
||||
return FilesCaller(user=user)
|
||||
return _token_caller(request, db, bearer_token(request) or "")
|
||||
|
||||
|
||||
def _refuse_token(key: str, reason: str) -> Exception:
|
||||
_bad.allow(key)
|
||||
log.info(
|
||||
"a request brought an API token that does not hold",
|
||||
extra=logs.event("apitoken.refused", reason=reason),
|
||||
)
|
||||
return refused(401, "not_authenticated")
|
||||
|
||||
|
||||
def _token_caller(request: Request, db: Session, raw: str) -> FilesCaller:
|
||||
key = client_key(request)
|
||||
if _bad.exhausted(key):
|
||||
raise refused(429, "api_token_rate_limited")
|
||||
if not well_formed(raw):
|
||||
raise _refuse_token(key, "malformed")
|
||||
token = db.execute(
|
||||
select(ApiToken).where(ApiToken.token_hash == hash_token(raw))
|
||||
).scalar_one_or_none()
|
||||
if token is None:
|
||||
raise _refuse_token(key, "unknown")
|
||||
now = datetime.now(timezone.utc)
|
||||
if token.expires_at is not None and token.expires_at <= now:
|
||||
raise _refuse_token(key, "expired")
|
||||
user = db.get(User, token.user_id)
|
||||
if user is None or not user.is_active or user.is_suspended:
|
||||
raise _refuse_token(key, "account")
|
||||
if not _calls.allow(str(token.id)):
|
||||
raise refused(429, "api_token_rate_limited")
|
||||
grants = {
|
||||
g.folder_id: grant_rights(g)
|
||||
for g in db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == token.id))
|
||||
.scalars()
|
||||
.all()
|
||||
}
|
||||
caller = FilesCaller(user=user, token_id=token.id, token_prefix=token.prefix, grants=grants)
|
||||
db.execute(
|
||||
update(ApiToken)
|
||||
.where(
|
||||
ApiToken.id == token.id,
|
||||
or_(ApiToken.last_used_at.is_(None), ApiToken.last_used_at < now - _TOUCH_EVERY),
|
||||
)
|
||||
.values(last_used_at=now)
|
||||
.execution_options(synchronize_session=False)
|
||||
)
|
||||
# Committed here, ahead of the route: a read-only route never commits, and a write route must
|
||||
# not carry this along into a rollback of its own.
|
||||
db.commit()
|
||||
logs.set_user(user.id, *debugsession.level_for(user, None))
|
||||
request.state.user_id = user.id
|
||||
return caller
|
||||
|
||||
|
||||
def revoke_all(db: Session, user_id: int) -> int:
|
||||
"""Revoke every token of `user_id`, each with its audit row (NOT committed). A token is a
|
||||
credential like a session, so whatever ends every session of an account — a password reset or
|
||||
change, "log out everywhere", a revoked untrusted password — ends its tokens too: one minted
|
||||
from a stolen session must not outlive the owner's answer to that theft. Returns how many."""
|
||||
gone = (
|
||||
db.execute(delete(ApiToken).where(ApiToken.user_id == user_id).returning(ApiToken.id))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
for token_id in gone:
|
||||
audit.record(
|
||||
db,
|
||||
user_id,
|
||||
AuditAction.API_TOKEN_REVOKE,
|
||||
target_type="api_token",
|
||||
target_id=token_id,
|
||||
summary="revoked with every session of the account",
|
||||
)
|
||||
return len(gone)
|
||||
|
||||
|
||||
def expiring(token: ApiToken, now: datetime) -> bool:
|
||||
"""Whether `token` runs out within the warning span (and has not yet)."""
|
||||
return token.expires_at is not None and now < token.expires_at <= now + EXPIRY_WARNING
|
||||
|
||||
|
||||
def warn_expiring(db: Session) -> int:
|
||||
"""Tell each user whose token runs out within `EXPIRY_WARNING` — once per token: the mark and the
|
||||
notice commit together, and a token made again (a new row) warns afresh. Returns how many."""
|
||||
now = datetime.now(timezone.utc)
|
||||
hits = db.execute(
|
||||
update(ApiToken)
|
||||
.where(
|
||||
ApiToken.expiry_warned.is_(False),
|
||||
ApiToken.expires_at.is_not(None),
|
||||
ApiToken.expires_at > now,
|
||||
ApiToken.expires_at <= now + EXPIRY_WARNING,
|
||||
)
|
||||
.values(expiry_warned=True)
|
||||
.returning(ApiToken.id, ApiToken.user_id, ApiToken.name, ApiToken.expires_at)
|
||||
.execution_options(synchronize_session=False)
|
||||
).all()
|
||||
for token_id, user_id, name, expires_at in hits:
|
||||
create_notification(
|
||||
db,
|
||||
user_id,
|
||||
"api_token_expiring",
|
||||
title=f"API token {name} expires soon", # English fallback; the client names it from data
|
||||
data={"token_id": token_id, "name": name, "expires_at": iso(expires_at)},
|
||||
commit=False,
|
||||
)
|
||||
db.commit()
|
||||
return len(hits)
|
||||
+17
-7
@@ -1,9 +1,9 @@
|
||||
"""Admin/scheduler audit trail: an append-only record of who changed what and when.
|
||||
"""Admin audit trail: an append-only record of who changed what and when.
|
||||
|
||||
Design (locked): log at ACTION / job-run granularity, NEVER per-row — a scheduler run that
|
||||
touches thousands of videos is ONE audit row summarising the outcome. Automatic job runs are
|
||||
only logged when they actually changed something (or errored); all admin actions and all manual
|
||||
runs are always logged. Secrets are never stored (config changes log the key name only).
|
||||
Design (locked): log at ACTION granularity, NEVER per-row — an action that touches thousands of
|
||||
rows is ONE audit row summarising the outcome. Background runs are not actions: they have their
|
||||
own history (app/jobruns.py). Secrets are never stored (config changes log the key name only).
|
||||
Each row keeps the request that made it, so the request's log lines are one click away.
|
||||
|
||||
`record()` only db.add()s the row — the caller commits it in the SAME transaction as the
|
||||
mutation it describes, so the trail can't drift from reality (and a rolled-back mutation drops
|
||||
@@ -15,6 +15,7 @@ from datetime import datetime, timedelta, timezone
|
||||
from sqlalchemy import delete
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import logs
|
||||
from app.models import AuditLog
|
||||
|
||||
|
||||
@@ -48,8 +49,6 @@ class AuditAction:
|
||||
MAINTENANCE_TUNE = "maintenance_tune"
|
||||
SYNC_PAUSE = "sync_pause"
|
||||
SYNC_RESUME = "sync_resume"
|
||||
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
|
||||
JOB_RUN = "job_run"
|
||||
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
|
||||
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
|
||||
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
|
||||
@@ -69,6 +68,16 @@ class AuditAction:
|
||||
FILE_FOLDER_RENAME = "file_folder_rename"
|
||||
FILE_FOLDER_RETENTION = "file_folder_retention"
|
||||
FILE_FOLDER_DELETE = "file_folder_delete"
|
||||
# A machine's token to the library (`routes.tokens`): made (its folders and rights) or revoked.
|
||||
# What a token then did is the file rows above, their summary naming it.
|
||||
API_TOKEN_CREATE = "api_token_create"
|
||||
API_TOKEN_REVOKE = "api_token_revoke"
|
||||
# A public `/watch` link to a file or a folder (`routes.downloads`): made, its settings changed,
|
||||
# revoked.
|
||||
# Never its token or password — only whether it has one.
|
||||
DOWNLOAD_LINK_CREATE = "download_link_create"
|
||||
DOWNLOAD_LINK_UPDATE = "download_link_update"
|
||||
DOWNLOAD_LINK_REVOKE = "download_link_revoke"
|
||||
|
||||
|
||||
def record(
|
||||
@@ -94,6 +103,7 @@ def record(
|
||||
summary=(summary or None) and summary[:255],
|
||||
before=before,
|
||||
after=after,
|
||||
request_id=logs.current_request_id(),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -268,10 +268,12 @@ class Settings(BaseSettings):
|
||||
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
|
||||
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
|
||||
quota_retention_days: int = 365
|
||||
# Background-run history (job_runs): days kept (0 = forever). The audit log's 180, which is how
|
||||
# long a run lived there before it moved here: ~1.8k rows a day at the default intervals
|
||||
# (measured 2026-10-17), ~320k at the limit. The same daily cleanup job prunes it.
|
||||
job_run_retention_days: int = 180
|
||||
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
|
||||
download_layout: DownloadLayout = "plex"
|
||||
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
|
||||
sponsorblock_default: bool = False
|
||||
|
||||
# --- Diagnostics / logging ---
|
||||
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
|
||||
|
||||
@@ -1,20 +1,23 @@
|
||||
"""The folders of the Files library — where each of a user's files sits.
|
||||
|
||||
A folder is a row, never a directory: on disk a file stays where the download layout put it (one
|
||||
shared asset can be held by several users), so a rename or a move is a column update. Two kinds:
|
||||
shared asset can be held by several users), so a rename or a move is a column update. Three kinds:
|
||||
|
||||
- a SOURCE folder is made automatically, one per origin a user has files from (YouTube, Plex clips,
|
||||
uploads, one per other site). Its `source_key` names the origin; the client renders its name
|
||||
from the key, in the reader's language.
|
||||
- an OWN folder is the user's, with the name they gave it.
|
||||
- the HUB folder (one per user, made when they first give a machine a token for it, `hub_folder`) is
|
||||
where they swap files with that machine. Its name, like a source folder's, is the client's; a file
|
||||
may be moved in like into an own folder, and it keeps a file until it is deleted.
|
||||
|
||||
Every file of a user is in exactly one folder. The folder a new file lands in is decided here
|
||||
(`source_key_for` + `source_folder`), at every place a job is created, so the rule lives once. A file
|
||||
moves only into one of the user's own folders or back to its own source folder (`move`): a YouTube
|
||||
file in the TikTok folder would be a lie the folder's name tells.
|
||||
moves only into one of the user's own folders, their Hub folder, or back to its own source folder
|
||||
(`move`): a YouTube file in the TikTok folder would be a lie the folder's name tells.
|
||||
|
||||
Retention is the folder's: a source folder keeps a file `download_retention_days` (the value when the
|
||||
folder was made; the user can change it), Uploads and own folders keep it until it is deleted. A
|
||||
folder was made; the user can change it), Uploads, own folders and the Hub keep it until it is deleted. A
|
||||
done file carries its own `expires_at`, stamped by `done_expiry` when it finishes — unless the user
|
||||
kept it — and its `ready_at`, the moment it finished (`stamp_done`).
|
||||
"""
|
||||
@@ -35,6 +38,7 @@ from app.models import DownloadJob, FileFolder, MediaAsset
|
||||
class FolderKind(StrEnum):
|
||||
SOURCE = "source"
|
||||
OWN = "own"
|
||||
HUB = "hub"
|
||||
|
||||
|
||||
class SourceKey(StrEnum):
|
||||
@@ -123,6 +127,26 @@ def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
|
||||
return folder
|
||||
|
||||
|
||||
def hub_folder(db: Session, user_id: int) -> FileFolder:
|
||||
"""The user's Hub folder, made on first use; a race on its one-per-user index is settled like
|
||||
`source_folder`'s."""
|
||||
stmt = select(FileFolder).where(
|
||||
FileFolder.user_id == user_id, FileFolder.kind == FolderKind.HUB
|
||||
)
|
||||
folder = db.execute(stmt).scalar_one_or_none()
|
||||
if folder is not None:
|
||||
return folder
|
||||
folder = FileFolder(user_id=user_id, kind=FolderKind.HUB)
|
||||
try:
|
||||
with db.begin_nested():
|
||||
db.add(folder)
|
||||
db.flush()
|
||||
# silent-ok: another request made the Hub folder first; re-selecting it is the whole handling.
|
||||
except IntegrityError:
|
||||
folder = db.execute(stmt).scalar_one()
|
||||
return folder
|
||||
|
||||
|
||||
def place(db: Session, job: DownloadJob) -> None:
|
||||
"""Put a new job's file in its source folder, and record that origin on it. Called wherever a
|
||||
job is created; the job must carry its `user_id`, `source_kind` and source refs already."""
|
||||
@@ -178,8 +202,9 @@ def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True)
|
||||
|
||||
|
||||
def upload_target(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
|
||||
"""The folder an upload asked to land in, if it may: one of the user's own folders, or their
|
||||
Uploads folder. A file put in a site's folder would claim an origin it does not have."""
|
||||
"""The folder an upload asked to land in, if it may: one of the user's own folders, their Hub
|
||||
folder, or their Uploads folder. A file put in a site's folder would claim an origin it does not
|
||||
have."""
|
||||
folder = own_folder(db, user_id, folder_id, hold=hold)
|
||||
if folder.kind == FolderKind.SOURCE and folder.source_key != SourceKey.UPLOAD:
|
||||
raise FolderError("file_folder_not_its_source")
|
||||
@@ -201,7 +226,8 @@ def set_kept(db: Session, job: DownloadJob, kept: bool) -> None:
|
||||
|
||||
|
||||
def move(db: Session, job: DownloadJob, folder: FileFolder) -> None:
|
||||
"""Put a file in `folder` — one of its user's own folders, or the source folder it came from.
|
||||
"""Put a file in `folder` — one of its user's own folders, their Hub folder, or the source folder
|
||||
it came from.
|
||||
The caller has checked that `folder` is the job's user's."""
|
||||
if folder.kind == FolderKind.SOURCE and folder.source_key != source_key_for(db, job):
|
||||
raise FolderError("file_folder_not_its_source")
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Download retention garbage collector (a scheduler job).
|
||||
|
||||
Five passes each run:
|
||||
Six passes each run:
|
||||
1. Pre-expiry warning — a done file whose `expires_at` (its folder's retention, `folders`) falls
|
||||
within the grace window, and that its user did not keep, warns its user once (`expiry_warned`).
|
||||
2. Expiry — such a file past its `expires_at` leaves the library: the job is deleted and its hold
|
||||
@@ -14,6 +14,8 @@ Five passes each run:
|
||||
re-download renames an asset, e.g. old accented channel dirs, or an upload's stranded file).
|
||||
It skips the dot-prefixed system trees (all but `.uploads`) and anything newer than an hour,
|
||||
so an in-flight download is never touched.
|
||||
6. Token expiry warning — a machine's token for the library about to run out warns its user once
|
||||
(`apitokens.warn_expiring`): the library's other clock.
|
||||
|
||||
Runs in the API process (which mounts FILES_ROOT). Pure disk/DB work — no YouTube quota.
|
||||
"""
|
||||
@@ -25,7 +27,7 @@ from pathlib import Path
|
||||
from sqlalchemy import func, or_, select, update
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import logs, sysconfig
|
||||
from app import apitokens, logs, sysconfig
|
||||
from app.config import settings
|
||||
from app.downloads import quota, service, storage
|
||||
from app.downloads.status import JobStatus
|
||||
@@ -366,12 +368,16 @@ def run_download_gc(db: Session) -> dict:
|
||||
# posters). Runs last so the keep-set reflects the assets that survived passes 2-4.
|
||||
swept = _sweep_orphan_files(db, settings.files_root)
|
||||
|
||||
# 6. A machine's token about to run out.
|
||||
tokens_warned = apitokens.warn_expiring(db)
|
||||
|
||||
result = {
|
||||
"warned": warned,
|
||||
"expired": expired,
|
||||
"evicted": evicted,
|
||||
"reaped": reaped,
|
||||
"swept": swept,
|
||||
"tokens_warned": tokens_warned,
|
||||
}
|
||||
log.info("download_gc %s", result)
|
||||
return result
|
||||
|
||||
@@ -1,10 +1,15 @@
|
||||
"""Public "share by link" helpers for the download center.
|
||||
|
||||
A DownloadLink is a capability URL: the unguessable `token` grants anyone read access to one
|
||||
download's file on the login-free `/watch/{token}` page. Password-protected links additionally
|
||||
require a short-lived signed **grant** on the file request — because a `<video src>` can't send a
|
||||
header, the password is exchanged once at `/unlock` for an HMAC grant appended to the file URL,
|
||||
so the raw password never rides in the media URL (or server logs).
|
||||
A DownloadLink is a capability URL: the unguessable token grants anyone read access to one
|
||||
download's file, or to the files in one of its owner's own folders (`folder_files`), on the
|
||||
login-free `/watch/{token}` page. So the token is never stored as written:
|
||||
the link is found by its SHA-256, and the owner's copy is Fernet-encrypted — a database copy alone
|
||||
opens nothing. Without that copy (no TOKEN_ENCRYPTION_KEY, or a key since changed) the link still
|
||||
opens; only its address cannot be shown again, and the owner makes a new link to share.
|
||||
|
||||
Password-protected links additionally require a short-lived signed **grant** on the file request —
|
||||
because a `<video src>` can't send a header, the password is exchanged once at `/unlock` for an HMAC
|
||||
grant appended to the file URL, so the raw password never rides in the media URL (or server logs).
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
@@ -12,16 +17,71 @@ import hmac
|
||||
import secrets
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import security
|
||||
from app.config import settings
|
||||
from app.models import DownloadLink
|
||||
from app.downloads import storage
|
||||
from app.downloads.status import JobStatus
|
||||
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
|
||||
|
||||
_GRANT_TTL = 6 * 3600 # a watch session's grant is good for 6h, then the viewer re-unlocks
|
||||
# The most files a folder link lists: as many as the owner's own Files list shows (`routes.files.
|
||||
# MAX_LIST`; a test holds the two equal — this layer may not import that one).
|
||||
FOLDER_MAX = 500
|
||||
|
||||
|
||||
def new_token() -> str:
|
||||
return secrets.token_urlsafe(24) # ~32 chars, 192 bits
|
||||
|
||||
|
||||
def seal(link: DownloadLink, token: str) -> None:
|
||||
"""Store `token` on `link` the only ways it is kept: its hash, and its encrypted copy."""
|
||||
link.token_hash = security.hash_token(token)
|
||||
link.token_enc = security.encrypt(token) if security.can_encrypt() else None
|
||||
|
||||
|
||||
def find(db: Session, token: str) -> DownloadLink | None:
|
||||
"""The link a `/watch` token names, by its hash."""
|
||||
return db.execute(
|
||||
select(DownloadLink).where(DownloadLink.token_hash == security.hash_token(token))
|
||||
).scalar_one_or_none()
|
||||
|
||||
|
||||
def recover(link: DownloadLink) -> str | None:
|
||||
"""The link's token from its encrypted copy, or None when there is none this key can read."""
|
||||
return security.decrypt_optional(link.token_enc)
|
||||
|
||||
|
||||
def in_folder(folder: FileFolder) -> tuple:
|
||||
"""Which jobs a link to `folder` reaches: the done files in it NOW, the folder owner's. A file
|
||||
moved out stops opening through the link at its next request; one moved in starts."""
|
||||
return (
|
||||
DownloadJob.folder_id == folder.id,
|
||||
DownloadJob.user_id == folder.user_id,
|
||||
DownloadJob.status == JobStatus.DONE,
|
||||
)
|
||||
|
||||
|
||||
def folder_files(db: Session, folder: FileFolder) -> list[tuple[DownloadJob, MediaAsset]]:
|
||||
"""What a link to `folder` shows — its page lists exactly these, and its link card counts them:
|
||||
each file `in_folder` whose asset is ready and whose bytes are on disk (so the list never offers
|
||||
a dead row), newest first, at most `FOLDER_MAX`."""
|
||||
rows = db.execute(
|
||||
select(DownloadJob, MediaAsset)
|
||||
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
|
||||
.where(*in_folder(folder), MediaAsset.status == "ready")
|
||||
.order_by(DownloadJob.ready_at.desc().nulls_last(), DownloadJob.id.desc())
|
||||
.limit(FOLDER_MAX)
|
||||
).all()
|
||||
return [
|
||||
(job, asset)
|
||||
for job, asset in rows
|
||||
if storage.safe_abs_path(settings.files_root, asset.rel_path) is not None
|
||||
]
|
||||
|
||||
|
||||
def is_expired(link: DownloadLink) -> bool:
|
||||
return link.expires_at is not None and link.expires_at <= datetime.now(timezone.utc)
|
||||
|
||||
@@ -55,18 +115,22 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
|
||||
return False
|
||||
if exp < datetime.now(timezone.utc).timestamp():
|
||||
return False
|
||||
return hmac.compare_digest(sig, _sign(f"{token}.{exp}.{password_version}"))
|
||||
# As bytes: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it, and
|
||||
# `sig` is whatever the URL carried.
|
||||
return hmac.compare_digest(sig.encode(), _sign(f"{token}.{exp}.{password_version}").encode())
|
||||
|
||||
|
||||
# --- serialization -------------------------------------------------------------------------
|
||||
|
||||
|
||||
def owner_view(link: DownloadLink, base_url: str = "") -> dict:
|
||||
"""What the file owner sees when managing their links."""
|
||||
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
|
||||
"""What the file owner sees when managing their links. `token` is the one just made (it is in
|
||||
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
|
||||
token = token or recover(link)
|
||||
return {
|
||||
"id": link.id,
|
||||
"token": link.token,
|
||||
"url": f"{base_url}/watch/{link.token}",
|
||||
"token": token,
|
||||
"url": f"{base_url}/watch/{token}" if token else None,
|
||||
"allow_download": link.allow_download,
|
||||
"has_password": link.password_hash is not None,
|
||||
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
|
||||
|
||||
@@ -11,13 +11,12 @@ import html
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import settings
|
||||
from app.downloads import links as linksmod
|
||||
from app.downloads import storage
|
||||
from app.models import DownloadJob, DownloadLink, MediaAsset
|
||||
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
|
||||
|
||||
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
|
||||
|
||||
@@ -74,22 +73,25 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
|
||||
|
||||
def _watch_tags(token: str, db: Session) -> str | None:
|
||||
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
|
||||
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
|
||||
link = linksmod.find(db, token)
|
||||
if link is None or linksmod.is_expired(link):
|
||||
return None
|
||||
url = f"{settings.app_base}/watch/{token}"
|
||||
folder = link.folder_id is not None
|
||||
if link.password_hash:
|
||||
# Password-gated: don't leak the title/thumbnail in the unfurl.
|
||||
return "\n ".join(
|
||||
[
|
||||
"<title>Siftlode</title>",
|
||||
_tag("og:title", "Password-protected video"),
|
||||
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
|
||||
_tag("og:description", "Shared via Siftlode"),
|
||||
_tag("og:type", "video.other"),
|
||||
_tag("og:type", "website" if folder else "video.other"),
|
||||
_tag("og:url", url),
|
||||
_tag("og:site_name", "Siftlode"),
|
||||
]
|
||||
)
|
||||
if folder:
|
||||
return _folder_tags(db, link, url)
|
||||
job = db.get(DownloadJob, link.job_id)
|
||||
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
|
||||
if job is None or asset is None or asset.status != "ready":
|
||||
@@ -147,6 +149,30 @@ def _watch_tags(token: str, db: Session) -> str | None:
|
||||
return "\n ".join(tags)
|
||||
|
||||
|
||||
def _folder_tags(db: Session, link: DownloadLink, url: str) -> str | None:
|
||||
"""A folder link's card: the folder's name and how many files it holds — never their names,
|
||||
which the page shows only to whoever opens it."""
|
||||
folder = db.get(FileFolder, link.folder_id)
|
||||
if folder is None or not folder.name:
|
||||
return None
|
||||
count = len(linksmod.folder_files(db, folder)) # the page's own list, so the two agree
|
||||
title = folder.name.strip()
|
||||
about = f"{count} file{'' if count == 1 else 's'} · Shared via Siftlode"
|
||||
return "\n ".join(
|
||||
[
|
||||
f"<title>{html.escape(title)}</title>",
|
||||
_tag("og:title", title),
|
||||
_tag("og:description", about),
|
||||
_tag("og:type", "website"),
|
||||
_tag("og:url", url),
|
||||
_tag("og:site_name", "Siftlode"),
|
||||
_tag("twitter:title", title, attr="name"),
|
||||
_tag("twitter:description", about, attr="name"),
|
||||
_tag("twitter:card", "summary", attr="name"),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def render_watch_html(token: str, page_html: Path, db: Session) -> str | None:
|
||||
"""Return the watch page (watch.html) with the OG block replaced for this token, or None to
|
||||
serve the generic page unchanged (invalid/expired/not-ready link, or a template without the
|
||||
|
||||
@@ -294,9 +294,13 @@ def served_media_type(media_kind: str | None, container: str | None) -> str | No
|
||||
def download_filename(display_name: str, container: str | None, path: Path) -> str:
|
||||
"""The Content-Disposition filename for a served download: the cleaned display name carrying a single
|
||||
correct extension — the asset container, else the file's own suffix — without doubling it when the
|
||||
name already ends in that extension. Shared by the private download and the public watch endpoints."""
|
||||
name already ends in that extension. A file with no extension at all is served without one —
|
||||
never as `name.` (Windows drops a trailing dot, so the saved file would not be the one named).
|
||||
Shared by the private download and the public watch endpoints."""
|
||||
ext = container or path.suffix.lstrip(".")
|
||||
base = display_filename(display_name)
|
||||
if not ext:
|
||||
return base
|
||||
if base.lower().endswith(f".{ext.lower()}"):
|
||||
base = base[: -(len(ext) + 1)]
|
||||
return f"{base}.{ext}"
|
||||
|
||||
@@ -57,12 +57,14 @@ def ingest_upload(
|
||||
display_name: str | None = None,
|
||||
sha256: str | None = None,
|
||||
folder_id: int | None = None,
|
||||
via: str | None = None,
|
||||
) -> DownloadJob:
|
||||
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return its job —
|
||||
`done`, or `queued` for the worker's phone-safe conversion (see the module docstring). `src_path`
|
||||
is MOVED into place on success (the caller stages it to a temp file first). `sha256` is the
|
||||
hash of those bytes, when the caller took it on the way in; `folder_id` the folder it asked for
|
||||
(`folders.upload_target`), else its Uploads folder.
|
||||
(`folders.upload_target`), else its Uploads folder; `via` what the audit row names as the doer
|
||||
when it was not the user themselves (an API token).
|
||||
|
||||
Raises folders.FolderError when that folder is not one an upload may land in (or is gone).
|
||||
|
||||
@@ -145,7 +147,14 @@ def ingest_upload(
|
||||
folders.stamp_done(db, job)
|
||||
service.incr_ref_count(db, asset.id)
|
||||
db.flush()
|
||||
audit.record(db, user_id, AuditAction.FILE_UPLOAD, target_type="file", target_id=job.id)
|
||||
audit.record(
|
||||
db,
|
||||
user_id,
|
||||
AuditAction.FILE_UPLOAD,
|
||||
target_type="file",
|
||||
target_id=job.id,
|
||||
summary=via,
|
||||
)
|
||||
db.commit()
|
||||
except Exception:
|
||||
db.rollback()
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
"""The durable history of background runs — the `job_runs` table.
|
||||
|
||||
A run is written twice: a `running` row when it starts, its outcome when it ends. Each write is its
|
||||
OWN short session, committed at once, for the reason `quota.record_usage` gives: the run's own work
|
||||
may roll back (its `except: db.rollback()`), and the history of what happened must not go with it —
|
||||
nor may this bookkeeping commit the job's half-done session.
|
||||
|
||||
Only the API process runs jobs (the worker never starts the scheduler), and one API process serves a
|
||||
database (`.env.example`, the Scheduler section: the single-run guard is in its memory too), so at its
|
||||
startup nothing can still be running: `interrupt_leftovers` closes what a stopped process left open.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import delete, func, select, union_all, update
|
||||
from sqlalchemy.engine import CursorResult
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import JobRun, QuotaEvent
|
||||
from app.runstatus import RunStatus, RunTrigger
|
||||
|
||||
|
||||
def start(job: str, trigger: RunTrigger, actor_id: int | None, target: str | None = None) -> int:
|
||||
"""Open a run's row; returns its id."""
|
||||
with SessionLocal() as s:
|
||||
row = JobRun(
|
||||
job=job,
|
||||
trigger=trigger,
|
||||
actor_id=actor_id,
|
||||
target=target,
|
||||
status=RunStatus.RUNNING,
|
||||
started_at=datetime.now(timezone.utc),
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
return row.id
|
||||
|
||||
|
||||
def finish(
|
||||
run_id: int,
|
||||
status: RunStatus,
|
||||
*,
|
||||
summary: str | None = None,
|
||||
result: dict | None = None,
|
||||
error: str | None = None,
|
||||
failure: dict | None = None,
|
||||
) -> None:
|
||||
"""Close a run with its outcome and what it spent. Conditional on the row still running: a row
|
||||
the startup sweep already called interrupted is not rewritten by a thread that outlived it."""
|
||||
with SessionLocal() as s:
|
||||
units = s.scalar(
|
||||
select(func.coalesce(func.sum(QuotaEvent.units), 0)).where(
|
||||
QuotaEvent.job_run_id == run_id
|
||||
)
|
||||
)
|
||||
s.execute(
|
||||
update(JobRun)
|
||||
.where(JobRun.id == run_id, JobRun.status == RunStatus.RUNNING)
|
||||
.values(
|
||||
status=status,
|
||||
finished_at=datetime.now(timezone.utc),
|
||||
summary=summary,
|
||||
result=result,
|
||||
error=error,
|
||||
failure=failure,
|
||||
quota_units=units or 0,
|
||||
)
|
||||
)
|
||||
s.commit()
|
||||
|
||||
|
||||
def interrupt_leftovers() -> int:
|
||||
"""Every row a previous process left `running` → `interrupted`, with what it spent until then
|
||||
(its quota rows outlive it, as `finish` sums them). Called once at startup, before the scheduler
|
||||
can open a new run. Returns how many."""
|
||||
spent = (
|
||||
select(func.coalesce(func.sum(QuotaEvent.units), 0))
|
||||
.where(QuotaEvent.job_run_id == JobRun.id)
|
||||
.scalar_subquery()
|
||||
)
|
||||
with SessionLocal() as s:
|
||||
res = s.execute(
|
||||
update(JobRun)
|
||||
.where(JobRun.status == RunStatus.RUNNING)
|
||||
.values(
|
||||
status=RunStatus.INTERRUPTED,
|
||||
finished_at=datetime.now(timezone.utc),
|
||||
quota_units=spent,
|
||||
)
|
||||
)
|
||||
s.commit()
|
||||
return res.rowcount if isinstance(res, CursorResult) else 0
|
||||
|
||||
|
||||
def recent_runs(db: Session, jobs: list[str], per_job: int) -> dict[str, list[JobRun]]:
|
||||
"""Each of `jobs`' last `per_job` runs, newest first, by job id — one query, each job's part a
|
||||
short walk of `ix_job_runs_job_started`, so a poll costs the same at any history length (a window
|
||||
over the whole table grew with every run kept)."""
|
||||
if not jobs:
|
||||
return {}
|
||||
newest = union_all(
|
||||
*(
|
||||
select(JobRun.id)
|
||||
.where(JobRun.job == job)
|
||||
.order_by(JobRun.started_at.desc(), JobRun.id.desc())
|
||||
.limit(per_job)
|
||||
for job in jobs
|
||||
)
|
||||
).subquery()
|
||||
rows = db.execute(
|
||||
select(JobRun)
|
||||
.where(JobRun.id.in_(select(newest.c.id)))
|
||||
.order_by(JobRun.job, JobRun.started_at.desc(), JobRun.id.desc())
|
||||
).scalars()
|
||||
out: dict[str, list[JobRun]] = {}
|
||||
for r in rows:
|
||||
out.setdefault(r.job, []).append(r)
|
||||
return out
|
||||
|
||||
|
||||
def gc(db: Session, retention_days: int) -> int:
|
||||
"""Delete finished runs older than `retention_days` (0 = keep forever). A run still going is
|
||||
never reaped. The caller commits."""
|
||||
if retention_days <= 0:
|
||||
return 0
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
|
||||
res = db.execute(
|
||||
delete(JobRun).where(JobRun.started_at < cutoff, JobRun.status != RunStatus.RUNNING)
|
||||
)
|
||||
return res.rowcount if isinstance(res, CursorResult) else 0
|
||||
+15
-2
@@ -10,8 +10,8 @@ the logs were both unjoinable and leaky. `configure_logging()` replaces all four
|
||||
Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
|
||||
ts · level · sev · source · logger · event · msg · attrs
|
||||
plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
|
||||
`job_id`) and the app version. That shape is what the admin log console filters on and what a Gitea
|
||||
issue attachment will carry, so it is deliberately flat and stable.
|
||||
`job_id`, `run_id`) and the app version. That shape is what the admin log console filters on and
|
||||
what a Gitea issue attachment will carry, so it is deliberately flat and stable.
|
||||
|
||||
Call sites stay ordinary stdlib logging:
|
||||
|
||||
@@ -81,15 +81,28 @@ _debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
|
||||
"log_debug_session", default=None
|
||||
)
|
||||
_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
|
||||
# The background run (app/jobruns.py) this context is doing; quota's spend is tagged with it too.
|
||||
_run_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_run_id", default=None)
|
||||
|
||||
_CONTEXT_VARS = {
|
||||
"request_id": _request_id,
|
||||
"user_id": _user_id,
|
||||
"debug_session_id": _debug_session,
|
||||
"job_id": _job_id,
|
||||
"run_id": _run_id,
|
||||
}
|
||||
|
||||
|
||||
def current_request_id() -> str | None:
|
||||
"""The HTTP request this context is serving, if any — the request middleware sets it."""
|
||||
return _request_id.get()
|
||||
|
||||
|
||||
def current_run_id() -> int | None:
|
||||
"""The background run this context is doing, if any — `bind(run_id=…)` sets it."""
|
||||
return _run_id.get()
|
||||
|
||||
|
||||
class _RequestScope:
|
||||
"""What the auth dependency learns about a request, for every line logged after it.
|
||||
|
||||
|
||||
@@ -362,6 +362,7 @@ def serialize(row: LogEvent) -> dict:
|
||||
"request_id": row.request_id,
|
||||
"user_id": row.user_id,
|
||||
"job_id": row.job_id,
|
||||
"run_id": row.run_id,
|
||||
"debug_session_id": row.debug_session_id,
|
||||
"version": row.version,
|
||||
}
|
||||
|
||||
@@ -52,12 +52,14 @@ from app.routes import ( # noqa: E402 — logging is configured first
|
||||
plex as plex_routes,
|
||||
public as public_routes,
|
||||
quota,
|
||||
runs as runs_routes,
|
||||
saved_views,
|
||||
scheduler as scheduler_routes,
|
||||
search as search_routes,
|
||||
setup as setup_routes,
|
||||
sync,
|
||||
tags,
|
||||
tokens as tokens_routes,
|
||||
version,
|
||||
)
|
||||
from app.scheduler import shutdown_scheduler, start_scheduler # noqa: E402 — logging is configured first
|
||||
@@ -337,12 +339,14 @@ app.include_router(plex_routes.router)
|
||||
app.include_router(downloads.router)
|
||||
app.include_router(downloads.admin_router)
|
||||
app.include_router(files_routes.router)
|
||||
app.include_router(tokens_routes.router)
|
||||
app.include_router(public_routes.router)
|
||||
app.include_router(admin.router)
|
||||
app.include_router(config_routes.router)
|
||||
app.include_router(scheduler_routes.router)
|
||||
app.include_router(audit_routes.router)
|
||||
app.include_router(admin_logs_routes.router)
|
||||
app.include_router(runs_routes.router)
|
||||
app.include_router(quota.router)
|
||||
app.include_router(version.router)
|
||||
app.include_router(setup_routes.router)
|
||||
|
||||
+122
-22
@@ -24,6 +24,7 @@ from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
|
||||
|
||||
from app.db import Base
|
||||
from app.runstatus import RunStatus, RunTrigger, sql_in
|
||||
from app.utils import canonical_email
|
||||
|
||||
# Every user-supplied display string in this schema shares one width, and it lives HERE so the
|
||||
@@ -563,14 +564,20 @@ class QuotaEvent(Base):
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now(), index=True
|
||||
)
|
||||
# The background run that spent it (app/jobruns.py); NULL = a request's own spend. SET NULL: a
|
||||
# reaped run leaves the spend in the budget's history.
|
||||
job_run_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("job_runs.id", ondelete="SET NULL"), index=True
|
||||
)
|
||||
|
||||
|
||||
class AuditLog(Base):
|
||||
"""Append-only admin/scheduler audit trail: who did what, when, and (where it applies)
|
||||
the before->after values. `actor_id` is the acting admin; NULL means the scheduler /
|
||||
background did it (SET NULL on user delete keeps the trail). Logged at ACTION / job-run
|
||||
granularity — never per-row — so a scheduler run that touches thousands of videos is ONE
|
||||
row summarising the outcome. `before`/`after` are small JSON snapshots (secrets excluded)."""
|
||||
"""Append-only admin audit trail: who did what, when, and (where it applies) the before->after
|
||||
values. `actor_id` is the acting admin; NULL means the background did it (SET NULL on user
|
||||
delete keeps the trail). Logged per ACTION, never per row; a background run is not an action
|
||||
and has its own history (`JobRun`). `before`/`after` are small JSON snapshots (secrets
|
||||
excluded). `request_id` is the request that made the change, so its log lines are one click
|
||||
away (NULL for rows written outside a request, and for rows older than the column)."""
|
||||
|
||||
__tablename__ = "audit_log"
|
||||
|
||||
@@ -587,6 +594,7 @@ class AuditLog(Base):
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now(), index=True
|
||||
)
|
||||
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
|
||||
|
||||
class LogEvent(Base):
|
||||
@@ -609,6 +617,8 @@ class LogEvent(Base):
|
||||
"debug_session_id",
|
||||
postgresql_where="debug_session_id IS NOT NULL",
|
||||
),
|
||||
# A run's lines are `WHERE run_id = …`; request lines carry none.
|
||||
Index("ix_log_events_run", "run_id", postgresql_where="run_id IS NOT NULL"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
|
||||
@@ -624,7 +634,11 @@ class LogEvent(Base):
|
||||
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
user_id: Mapped[int | None] = mapped_column(Integer)
|
||||
debug_session_id: Mapped[str | None] = mapped_column(String(64))
|
||||
# A DOWNLOAD job's id (download_jobs) — not a background run, which is `run_id`.
|
||||
job_id: Mapped[int | None] = mapped_column(Integer)
|
||||
# The background run (job_runs) that logged it. No foreign key, as `user_id`: the writer runs
|
||||
# behind the event, and a run reaped meanwhile must not fail the batch its lines travel in.
|
||||
run_id: Mapped[int | None] = mapped_column(BigInteger)
|
||||
version: Mapped[str | None] = mapped_column(String(64))
|
||||
|
||||
|
||||
@@ -689,15 +703,44 @@ class AppState(Base):
|
||||
setup_token_hash: Mapped[str | None] = mapped_column(String(64))
|
||||
|
||||
|
||||
class SchedulerSetting(Base):
|
||||
"""Admin override for a scheduler job's run interval (minutes). One row per job id;
|
||||
absent = use the env/config default. DB-backed so it's editable from the admin UI at
|
||||
runtime without a redeploy (env stays a fallback)."""
|
||||
class JobRun(Base):
|
||||
"""One run of a background job — scheduled, an admin's "run now", or a user's ad-hoc work on one
|
||||
target — written when it starts and again when it ends (app/jobruns.py), each in its own
|
||||
committed session, so the history survives the job's own rollback and a restart. A row left
|
||||
`running` by a process that died is set `interrupted` at the next startup.
|
||||
|
||||
__tablename__ = "scheduler_settings"
|
||||
`quota_units` is what the run spent of the YouTube budget: the `quota_events` rows carrying its
|
||||
id, summed when it ends. `summary` is the English one-line fallback; `result` / `failure` are the
|
||||
same outcome as data, which the client names in the reader's language."""
|
||||
|
||||
job_id: Mapped[str] = mapped_column(String(40), primary_key=True)
|
||||
interval_minutes: Mapped[int] = mapped_column(Integer)
|
||||
__tablename__ = "job_runs"
|
||||
__table_args__ = (
|
||||
Index("ix_job_runs_job_started", "job", "started_at"),
|
||||
CheckConstraint(f"status IN ({sql_in(RunStatus)})", name="ck_job_runs_status"),
|
||||
CheckConstraint(f"trigger IN ({sql_in(RunTrigger)})", name="ck_job_runs_trigger"),
|
||||
# Only a run still going has no end.
|
||||
CheckConstraint(
|
||||
f"(status = '{RunStatus.RUNNING}') = (finished_at IS NULL)", name="ck_job_runs_finished"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
|
||||
job: Mapped[str] = mapped_column(String(40))
|
||||
trigger: Mapped[str] = mapped_column(String(16))
|
||||
# Who started it: the admin of a manual run, the user of an ad-hoc one; NULL = the schedule.
|
||||
actor_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("users.id", ondelete="SET NULL"), index=True
|
||||
)
|
||||
# What an ad-hoc run worked on (a channel id); NULL for the recurring jobs.
|
||||
target: Mapped[str | None] = mapped_column(String(128))
|
||||
status: Mapped[str] = mapped_column(String(16))
|
||||
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
summary: Mapped[str | None] = mapped_column(Text)
|
||||
result: Mapped[dict | None] = mapped_column(JSONB)
|
||||
error: Mapped[str | None] = mapped_column(Text)
|
||||
failure: Mapped[dict | None] = mapped_column(JSONB)
|
||||
quota_units: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
|
||||
|
||||
|
||||
class SystemConfig(Base):
|
||||
@@ -919,7 +962,8 @@ class FileFolder(Base, TimestampMixin):
|
||||
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
|
||||
|
||||
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
|
||||
user gets a file from that origin; an OWN folder (`name` set) is one the user made. Every file —
|
||||
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
|
||||
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
|
||||
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
|
||||
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
|
||||
|
||||
@@ -933,9 +977,12 @@ class FileFolder(Base, TimestampMixin):
|
||||
unique=True,
|
||||
postgresql_where=text("name IS NOT NULL"),
|
||||
),
|
||||
# One Hub folder per user.
|
||||
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
|
||||
CheckConstraint(
|
||||
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
|
||||
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)",
|
||||
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
|
||||
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
|
||||
name="ck_file_folder_kind",
|
||||
),
|
||||
)
|
||||
@@ -949,6 +996,47 @@ class FileFolder(Base, TimestampMixin):
|
||||
retention_days: Mapped[int | None] = mapped_column(Integer)
|
||||
|
||||
|
||||
class ApiToken(Base, TimestampMixin):
|
||||
"""A user's token for a machine to reach their Files library with (`app.apitokens`): sent as
|
||||
`Authorization: Bearer sfl_…`, honoured on `/api/files` only, and only in the folders its
|
||||
grants name. Only the SHA-256 of the token is kept — it is shown once, when it is made — plus
|
||||
its first characters (`prefix`), so the user can tell their tokens apart."""
|
||||
|
||||
__tablename__ = "api_tokens"
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
|
||||
name: Mapped[str] = mapped_column(String(80))
|
||||
prefix: Mapped[str] = mapped_column(String(16))
|
||||
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
|
||||
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) # NULL = never
|
||||
last_used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# The one notice its user gets before it runs out (`apitokens.warn_expiring`).
|
||||
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
|
||||
|
||||
|
||||
class ApiTokenGrant(Base):
|
||||
"""What one token may do in one folder. A folder with no grant is out of the token's reach."""
|
||||
|
||||
__tablename__ = "api_token_grants"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
|
||||
CheckConstraint(
|
||||
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
token_id: Mapped[int] = mapped_column(ForeignKey("api_tokens.id", ondelete="CASCADE"))
|
||||
folder_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
|
||||
)
|
||||
can_list: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
|
||||
can_read: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
|
||||
can_write: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
|
||||
can_delete: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
|
||||
|
||||
|
||||
def _ready_on_insert(context) -> datetime | None:
|
||||
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
|
||||
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
|
||||
@@ -1084,21 +1172,33 @@ class DownloadQuota(Base, UpdatedAtMixin):
|
||||
|
||||
|
||||
class DownloadLink(Base, TimestampMixin):
|
||||
"""A public capability link to one download — the "share by link" surface.
|
||||
"""A public capability link to one download, or to one of its owner's own folders — the "share
|
||||
by link" surface.
|
||||
|
||||
Anyone holding the unguessable `token` can watch the file on the login-free `/watch/{token}`
|
||||
player page (Google-Drive style). Optional per-link controls: `expires_at`, `allow_download`
|
||||
(stream-only vs downloadable), and a `password_hash` (argon2, like a user password). Revoke =
|
||||
delete the row. Distinct from `DownloadShare`, which grants a *registered* user access via the
|
||||
in-app "Shared with me" list."""
|
||||
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
|
||||
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
|
||||
the folder at the time it is opened. The token itself is never stored as written: a link is
|
||||
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
|
||||
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
|
||||
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
|
||||
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
|
||||
grants a *registered* user access via the in-app "Shared with me" list."""
|
||||
|
||||
__tablename__ = "download_links"
|
||||
__table_args__ = (
|
||||
# A link opens exactly one thing: a file or a folder.
|
||||
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
job_id: Mapped[int] = mapped_column(
|
||||
job_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
|
||||
)
|
||||
token: Mapped[str] = mapped_column(String(64), unique=True)
|
||||
folder_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
|
||||
)
|
||||
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
|
||||
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
|
||||
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
|
||||
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
|
||||
password_hash: Mapped[str | None] = mapped_column(Text)
|
||||
|
||||
+11
-3
@@ -16,7 +16,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.engine import CursorResult
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import sysconfig
|
||||
from app import logs, sysconfig
|
||||
from app.db import SessionLocal
|
||||
from app.models import ApiQuotaUsage, QuotaEvent
|
||||
|
||||
@@ -204,6 +204,14 @@ def record_usage(units: int) -> None:
|
||||
)
|
||||
with SessionLocal() as qs:
|
||||
qs.execute(stmt)
|
||||
# Audit detail: who/what spent it (per-user attribution; NULL actor = system).
|
||||
qs.add(QuotaEvent(user_id=_actor_id.get(), action=_action.get(), units=units))
|
||||
# Audit detail: who/what spent it (per-user attribution; NULL actor = system), and the
|
||||
# background run it belongs to, so the run's row can say what it cost.
|
||||
qs.add(
|
||||
QuotaEvent(
|
||||
user_id=_actor_id.get(),
|
||||
action=_action.get(),
|
||||
units=units,
|
||||
job_run_id=logs.current_run_id(),
|
||||
)
|
||||
)
|
||||
qs.commit()
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
"""The account filter the activity page's lists share (runs, admin actions, server log, quota): one
|
||||
filter bar drives them all, so the value it sends means the same on every tab.
|
||||
|
||||
The value is an account id, or `none` — the rows no account made (the background) — a value to filter
|
||||
on, not the absence of a filter.
|
||||
"""
|
||||
|
||||
from sqlalchemy import ColumnElement
|
||||
|
||||
NONE = "none"
|
||||
# The `user` query parameter's shape on every route that takes it.
|
||||
PATTERN = r"^(none|[0-9]{1,10})$"
|
||||
|
||||
|
||||
def condition(column, value: str) -> ColumnElement[bool]:
|
||||
"""`column` (an account id column) narrowed to `value`."""
|
||||
return column.is_(None) if value == NONE else column == int(value)
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Keyset paging over `(timestamp desc, id desc)`, shared by the admin history lists (server log,
|
||||
audit log, background runs).
|
||||
|
||||
A timestamp alone is not unique (a browser log row's `ts` is clamped into its consent window; a run
|
||||
and its siblings can start in the same microsecond), and an offset would skip or repeat rows while
|
||||
writers add new ones at the top — so a page continues from the last row's `(timestamp, id)`.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import and_, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import User
|
||||
|
||||
PAGE_MAX = 200
|
||||
|
||||
|
||||
def utc(value: datetime | None) -> datetime | None:
|
||||
"""A bound without a zone is UTC — the columns are `timestamptz`, and a naive value would be read
|
||||
in the database session's zone instead."""
|
||||
if value is None or value.tzinfo is not None:
|
||||
return value
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def encode_cursor(ts: datetime, row_id: int) -> str:
|
||||
return base64.urlsafe_b64encode(json.dumps([ts.isoformat(), row_id]).encode()).decode()
|
||||
|
||||
|
||||
def decode_cursor(cursor: str) -> tuple[datetime, int]:
|
||||
try:
|
||||
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
|
||||
at = datetime.fromisoformat(ts)
|
||||
return (at if at.tzinfo is not None else at.replace(tzinfo=timezone.utc)), int(row_id)
|
||||
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
|
||||
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
|
||||
|
||||
|
||||
def older_than(ts_col, id_col, cursor: str):
|
||||
"""The rows after `cursor` in `(ts desc, id desc)` order. The `ts <=` conjunct is implied by the
|
||||
OR; it is there so the planner can bound the timestamp index scan instead of filtering every
|
||||
newer row first."""
|
||||
ts, row_id = decode_cursor(cursor)
|
||||
return and_(ts_col <= ts, or_(ts_col < ts, and_(ts_col == ts, id_col < row_id)))
|
||||
|
||||
|
||||
def page(rows: list, limit: int, ts_of) -> tuple[list, str | None]:
|
||||
"""`rows` was read with `limit + 1` (to know whether there is a next page without a count query):
|
||||
the page itself, and the cursor that continues after it (None on the last page)."""
|
||||
more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
return rows, encode_cursor(ts_of(rows[-1]), rows[-1].id) if more else None
|
||||
|
||||
|
||||
def accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
|
||||
"""The accounts a page names, by id → address. An id with no account left (erased after the row
|
||||
was written) is simply absent, and the client shows the bare id."""
|
||||
if not user_ids:
|
||||
return {}
|
||||
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
|
||||
return {str(uid): email for uid, email in rows}
|
||||
@@ -8,33 +8,29 @@ What a row holds was redacted when it was written (`logs.record_fields`, and the
|
||||
browser's rows), so this serves the columns as they are; a value the redactor took out reads as
|
||||
`[redacted]` / `[email]`, which is the point — the console shows that something was there.
|
||||
|
||||
Paging is a keyset over `(ts desc, id desc)`: a browser row's `ts` is clamped into its consent
|
||||
window, so `ts` alone is not unique, and an offset would skip or repeat rows while the writer adds
|
||||
new ones at the top. The live tail re-reads the first page rather than asking for "ids after N":
|
||||
Paging is a keyset over `(ts desc, id desc)` (`_keyset`). The live tail re-reads the first page rather than asking for "ids after N":
|
||||
several processes write this table, and a batch's ids are taken when it inserts, not when it
|
||||
commits, so a slower batch can land below an id the tail already passed — and would never show.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import Text, and_, cast, func, or_, select
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import Text, cast, func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import logsink
|
||||
from app.auth import admin_user
|
||||
from app.db import get_db
|
||||
from app.models import LogEvent, User
|
||||
from app.routes import _account as account
|
||||
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
|
||||
from app.utils import like_escape
|
||||
|
||||
router = APIRouter(prefix="/api/admin/logs", tags=["admin"])
|
||||
|
||||
PAGE_MAX = 200
|
||||
# A facet list is a chip row, not a report: past this many distinct events the rest are reachable
|
||||
# through the free-text search, and the response stays small whatever a self-host logs.
|
||||
FACET_EVENTS_MAX = 100
|
||||
@@ -42,38 +38,9 @@ FACET_EVENTS_MAX = 100
|
||||
Level = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
|
||||
# The filters a facet can leave out of its own count (`LogFilters.conditions`).
|
||||
Facet = Literal["levels", "sources", "user", "event"]
|
||||
# `none` selects the rows that have no account / no event code (a scheduler line, an anonymous
|
||||
# request) — a real value to filter on, not the absence of a filter.
|
||||
NONE = "none"
|
||||
|
||||
|
||||
def _utc(value: datetime | None) -> datetime | None:
|
||||
"""A bound without a zone is UTC — the column is `timestamptz`, and a naive value would be read
|
||||
in the database session's zone instead."""
|
||||
if value is None or value.tzinfo is not None:
|
||||
return value
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _encode_cursor(row: LogEvent) -> str:
|
||||
return base64.urlsafe_b64encode(json.dumps([row.ts.isoformat(), row.id]).encode()).decode()
|
||||
|
||||
|
||||
def _decode_cursor(cursor: str) -> tuple[datetime, int]:
|
||||
try:
|
||||
ts, row_id = json.loads(base64.urlsafe_b64decode(cursor.encode()))
|
||||
return _utc(datetime.fromisoformat(ts)), int(row_id)
|
||||
except (binascii.Error, json.JSONDecodeError, ValueError, TypeError) as exc:
|
||||
raise HTTPException(status_code=400, detail={"code": "bad_cursor"}) from exc
|
||||
|
||||
|
||||
def _accounts(db: Session, user_ids: set[int]) -> dict[str, str]:
|
||||
"""The accounts a page names, by id → address. An id with no account left (erased after the
|
||||
line was written) is simply absent, and the client shows the bare id."""
|
||||
if not user_ids:
|
||||
return {}
|
||||
rows = db.execute(select(User.id, User.email).where(User.id.in_(user_ids))).all()
|
||||
return {str(uid): email for uid, email in rows}
|
||||
# `none` selects the rows that have no event code, as the account filter's `none` (app/routes/_account)
|
||||
# selects the rows of no account — a real value to filter on, not the absence of a filter.
|
||||
NONE = account.NONE
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -86,6 +53,7 @@ class LogFilters:
|
||||
event: str | None
|
||||
request_id: str | None
|
||||
job_id: int | None
|
||||
run_id: int | None
|
||||
q: str | None
|
||||
since: datetime | None
|
||||
until: datetime | None
|
||||
@@ -100,11 +68,7 @@ class LogFilters:
|
||||
if self.sources and skip != "sources":
|
||||
out.append(LogEvent.source.in_(self.sources))
|
||||
if self.user is not None and skip != "user":
|
||||
out.append(
|
||||
LogEvent.user_id.is_(None)
|
||||
if self.user == NONE
|
||||
else LogEvent.user_id == int(self.user)
|
||||
)
|
||||
out.append(account.condition(LogEvent.user_id, self.user))
|
||||
if self.event is not None and skip != "event":
|
||||
out.append(
|
||||
LogEvent.event.is_(None) if self.event == NONE else LogEvent.event == self.event
|
||||
@@ -113,10 +77,12 @@ class LogFilters:
|
||||
out.append(LogEvent.request_id == self.request_id)
|
||||
if self.job_id is not None:
|
||||
out.append(LogEvent.job_id == self.job_id)
|
||||
if self.run_id is not None:
|
||||
out.append(LogEvent.run_id == self.run_id)
|
||||
if self.since is not None:
|
||||
out.append(LogEvent.ts >= _utc(self.since))
|
||||
out.append(LogEvent.ts >= utc(self.since))
|
||||
if self.until is not None:
|
||||
out.append(LogEvent.ts < _utc(self.until))
|
||||
out.append(LogEvent.ts < utc(self.until))
|
||||
if self.q and self.q.strip():
|
||||
like = f"%{like_escape(self.q.strip())}%"
|
||||
out.append(
|
||||
@@ -134,17 +100,20 @@ class LogFilters:
|
||||
def log_filters(
|
||||
levels: list[Level] = Query(default=[]),
|
||||
sources: list[str] = Query(default=[], max_length=8),
|
||||
user: str | None = Query(default=None, pattern=r"^(none|[0-9]{1,10})$"),
|
||||
user: str | None = Query(default=None, pattern=account.PATTERN),
|
||||
event: str | None = Query(
|
||||
default=None, max_length=64, pattern=r"^(none|[a-z][a-z0-9_]*(\.[a-z0-9_]+)*)$"
|
||||
),
|
||||
request_id: str | None = Query(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$"),
|
||||
job_id: int | None = Query(default=None, ge=1),
|
||||
run_id: int | None = Query(default=None, ge=1),
|
||||
q: str | None = Query(default=None, max_length=200),
|
||||
since: datetime | None = None,
|
||||
until: datetime | None = None,
|
||||
) -> LogFilters:
|
||||
return LogFilters(list(levels), list(sources), user, event, request_id, job_id, q, since, until)
|
||||
return LogFilters(
|
||||
list(levels), list(sources), user, event, request_id, job_id, run_id, q, since, until
|
||||
)
|
||||
|
||||
|
||||
@router.get("")
|
||||
@@ -159,28 +128,20 @@ def list_logs(
|
||||
continues from a previous page's `next_cursor`."""
|
||||
query = select(LogEvent).where(*filters.conditions())
|
||||
if before is not None:
|
||||
ts, row_id = _decode_cursor(before)
|
||||
# The `ts <=` conjunct is implied by the OR; it is there so the planner can bound the
|
||||
# `ts` index scan instead of filtering every newer row first.
|
||||
query = query.where(
|
||||
and_(
|
||||
LogEvent.ts <= ts,
|
||||
or_(LogEvent.ts < ts, and_(LogEvent.ts == ts, LogEvent.id < row_id)),
|
||||
)
|
||||
)
|
||||
|
||||
# One more than asked, to know whether there is a next page without a second count query.
|
||||
rows = (
|
||||
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
|
||||
.scalars()
|
||||
.all()
|
||||
query = query.where(older_than(LogEvent.ts, LogEvent.id, before))
|
||||
rows, cursor = page(
|
||||
list(
|
||||
db.execute(query.order_by(LogEvent.ts.desc(), LogEvent.id.desc()).limit(limit + 1))
|
||||
.scalars()
|
||||
.all()
|
||||
),
|
||||
limit,
|
||||
lambda r: r.ts,
|
||||
)
|
||||
more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
return {
|
||||
"items": [logsink.serialize(r) for r in rows],
|
||||
"accounts": _accounts(db, {r.user_id for r in rows if r.user_id is not None}),
|
||||
"next_cursor": _encode_cursor(rows[-1]) if more else None,
|
||||
"accounts": accounts(db, {r.user_id for r in rows if r.user_id is not None}),
|
||||
"next_cursor": cursor,
|
||||
}
|
||||
|
||||
|
||||
@@ -211,6 +172,6 @@ def log_facets(
|
||||
"sources": counts(LogEvent.source, "sources"),
|
||||
"events": counts(LogEvent.event, "event", FACET_EVENTS_MAX),
|
||||
"users": [{**u, "value": str(u["value"])} for u in users],
|
||||
"accounts": _accounts(db, {u["value"] for u in users if u["value"] != NONE}),
|
||||
"accounts": accounts(db, {u["value"] for u in users if u["value"] != NONE}),
|
||||
"total": db.scalar(total) or 0,
|
||||
}
|
||||
|
||||
+106
-29
@@ -1,67 +1,144 @@
|
||||
"""Admin audit log API: read the append-only who/what/when trail, and clear it.
|
||||
|
||||
Admin-only. The client renders these rows in a DataTable (client-side sort/filter/paginate), so
|
||||
this returns the most recent window of rows (capped) plus the true total, rather than paging
|
||||
server-side — the log stays small in practice (state-changing runs + admin actions only)."""
|
||||
Admin-only. The activity page reads it a page at a time, newest first, filtered on the server like
|
||||
the server log (`_keyset` paging), so the whole trail is reachable however long it grows."""
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import delete, func, select
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import Text, cast, delete, func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import audit
|
||||
from app.audit import AuditAction
|
||||
from app.auth import admin_user
|
||||
from app.db import get_db
|
||||
from app.models import AuditLog, User
|
||||
from app.utils import iso
|
||||
from app.models import AuditLog, LogEvent, User
|
||||
from app.routes import _account as account
|
||||
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
|
||||
from app.utils import iso, like_escape
|
||||
|
||||
router = APIRouter(prefix="/api/admin/audit", tags=["admin"])
|
||||
|
||||
# Cap the client-side window. The audit log is small by design; if it ever outgrows this we'd
|
||||
# switch the DataTable to server-side paging.
|
||||
MAX_ROWS = 2000
|
||||
|
||||
|
||||
def _serialize(row: AuditLog, actors: dict[int, str]) -> dict:
|
||||
def _serialize(row: AuditLog, lines: dict[str, int]) -> dict:
|
||||
return {
|
||||
"id": row.id,
|
||||
"created_at": iso(row.created_at),
|
||||
# actor_id NULL = the scheduler/background did it (client shows "System"); a set id that
|
||||
# can't be resolved would mean a deleted user, but SET NULL makes that NULL — so present.
|
||||
"actor": actors.get(row.actor_id) if row.actor_id is not None else None,
|
||||
# NULL = the background did it (the client shows "System"); the address is in `accounts`.
|
||||
"actor_id": row.actor_id,
|
||||
"action": row.action,
|
||||
"target_type": row.target_type,
|
||||
"target_id": row.target_id,
|
||||
"summary": row.summary,
|
||||
"before": row.before,
|
||||
"after": row.after,
|
||||
"request_id": row.request_id,
|
||||
# How many of that request's log lines are still kept: they go after days, the row after
|
||||
# months, and a link to none would open an empty list.
|
||||
"log_lines": lines.get(row.request_id, 0) if row.request_id else 0,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AuditFilters:
|
||||
action: str | None
|
||||
actor: str | None
|
||||
q: str | None
|
||||
since: datetime | None
|
||||
until: datetime | None
|
||||
|
||||
def conditions(self, skip: Literal["action"] | None = None) -> list:
|
||||
"""The WHERE conditions, ANDed. `skip="action"` leaves the action filter out, so the action
|
||||
menu counts every action under the other filters (the way back from a pick)."""
|
||||
out = []
|
||||
if self.action is not None and skip != "action":
|
||||
out.append(AuditLog.action == self.action)
|
||||
if self.actor is not None:
|
||||
out.append(account.condition(AuditLog.actor_id, self.actor))
|
||||
if self.since is not None:
|
||||
out.append(AuditLog.created_at >= utc(self.since))
|
||||
if self.until is not None:
|
||||
out.append(AuditLog.created_at < utc(self.until))
|
||||
if self.q and self.q.strip():
|
||||
like = f"%{like_escape(self.q.strip())}%"
|
||||
out.append(
|
||||
or_(
|
||||
AuditLog.summary.ilike(like),
|
||||
AuditLog.target_id.ilike(like),
|
||||
AuditLog.action.ilike(like),
|
||||
AuditLog.request_id.ilike(like),
|
||||
cast(AuditLog.before, Text).ilike(like),
|
||||
cast(AuditLog.after, Text).ilike(like),
|
||||
)
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def audit_filters(
|
||||
action: str | None = Query(default=None, max_length=48, pattern=r"^[a-z][a-z0-9_]*$"),
|
||||
user: str | None = Query(default=None, pattern=account.PATTERN),
|
||||
q: str | None = Query(default=None, max_length=200),
|
||||
since: datetime | None = None,
|
||||
until: datetime | None = None,
|
||||
) -> AuditFilters:
|
||||
return AuditFilters(action, user, q, since, until)
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_audit(
|
||||
limit: int = 500,
|
||||
filters: AuditFilters = Depends(audit_filters),
|
||||
before: str | None = Query(default=None, max_length=200),
|
||||
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
|
||||
_: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
limit = max(1, min(limit, MAX_ROWS))
|
||||
total = db.scalar(select(func.count()).select_from(AuditLog)) or 0
|
||||
rows = (
|
||||
db.execute(select(AuditLog).order_by(AuditLog.created_at.desc()).limit(limit))
|
||||
.scalars()
|
||||
.all()
|
||||
"""One page of rows, newest first, matching every filter given; `before` continues from a
|
||||
previous page's `next_cursor`."""
|
||||
query = select(AuditLog).where(*filters.conditions())
|
||||
if before is not None:
|
||||
query = query.where(older_than(AuditLog.created_at, AuditLog.id, before))
|
||||
order = (AuditLog.created_at.desc(), AuditLog.id.desc())
|
||||
rows, cursor = page(
|
||||
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
|
||||
limit,
|
||||
lambda r: r.created_at,
|
||||
)
|
||||
actor_ids = {r.actor_id for r in rows if r.actor_id is not None}
|
||||
actors: dict[int, str] = {}
|
||||
if actor_ids:
|
||||
actors = dict(db.execute(select(User.id, User.email).where(User.id.in_(actor_ids))).all())
|
||||
requests = {r.request_id for r in rows if r.request_id}
|
||||
lines: dict[str, int] = {}
|
||||
if requests:
|
||||
counted = db.execute(
|
||||
select(LogEvent.request_id, func.count())
|
||||
.where(LogEvent.request_id.in_(requests))
|
||||
.group_by(LogEvent.request_id)
|
||||
)
|
||||
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
|
||||
return {
|
||||
"items": [_serialize(r, actors) for r in rows],
|
||||
"total": total,
|
||||
"returned": len(rows),
|
||||
"items": [_serialize(r, lines) for r in rows],
|
||||
"accounts": accounts(db, {r.actor_id for r in rows if r.actor_id is not None}),
|
||||
"next_cursor": cursor,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/facets")
|
||||
def audit_facets(
|
||||
filters: AuditFilters = Depends(audit_filters),
|
||||
_: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""The actions present under the other filters, with counts — what the action menu offers."""
|
||||
n = func.count().label("n")
|
||||
stmt = (
|
||||
select(AuditLog.action, n)
|
||||
.where(*filters.conditions("action"))
|
||||
.group_by(AuditLog.action)
|
||||
.order_by(n.desc(), AuditLog.action)
|
||||
)
|
||||
return {"actions": [{"value": a, "count": c} for a, c in db.execute(stmt).all()]}
|
||||
|
||||
|
||||
@router.delete("")
|
||||
def clear_audit(admin: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
|
||||
"""Clear the audit log. Leaves ONE row recording who cleared it + how many (tamper-evidence)."""
|
||||
|
||||
+28
-14
@@ -16,6 +16,7 @@ from sqlalchemy import select, update
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import apitokens
|
||||
from app import email as email_mod
|
||||
from app import logs
|
||||
from app import mailtext
|
||||
@@ -446,7 +447,7 @@ async def callback(
|
||||
# SEC-10: drop any password nobody proved the mailbox for, now that Google attests
|
||||
# ownership (bumps the epoch to revoke any session it minted). Runs before the
|
||||
# activation below, which is what it reads.
|
||||
revoke_untrusted_preset_password(user)
|
||||
revoke_untrusted_preset_password(db, user)
|
||||
# Google verified the email and is_allowed granted access, so finish activating a
|
||||
# previously pending password registration (otherwise current_user would bounce it).
|
||||
user.email_verified = True
|
||||
@@ -486,7 +487,7 @@ async def callback(
|
||||
# real owner, signed in through Google right now, sets a new password in Settings.
|
||||
if canonical_email(email) in env_admin_canonicals():
|
||||
if user.role != "admin" and user.password_hash is not None:
|
||||
_revoke_password(user)
|
||||
_revoke_password(db, user)
|
||||
user.role = "admin"
|
||||
elif not user.role:
|
||||
user.role = "user"
|
||||
@@ -1118,9 +1119,8 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
|
||||
# about. The person still confirms their address the normal way.
|
||||
if kind == "reset":
|
||||
user.email_verified = True
|
||||
bump_session_epoch(
|
||||
user
|
||||
) # SA4: a reset (often a compromise response) kills all existing sessions.
|
||||
# SA4: a reset (often a compromise response) kills all existing sessions — and API tokens.
|
||||
bump_session_epoch(db, user)
|
||||
# Burn any other outstanding reset tokens for this user.
|
||||
for row in db.execute(
|
||||
select(AuthToken).where(
|
||||
@@ -1143,13 +1143,27 @@ def _remember_epoch(request: Request, user: User) -> None:
|
||||
request.session["epochs"] = epochs
|
||||
|
||||
|
||||
def bump_session_epoch(user: User) -> None:
|
||||
def bump_session_epoch(db: Session, user: User) -> None:
|
||||
"""Invalidate every EXISTING signed cookie for this account by advancing its epoch (SA4). Cookies
|
||||
minted before the bump carry a now-stale epoch and are rejected by `current_user`."""
|
||||
user.session_epoch = (user.session_epoch or 0) + 1
|
||||
minted before the bump carry a now-stale epoch and are rejected by `current_user`. The account's
|
||||
API tokens go with them (`apitokens.revoke_all`): a token minted from a stolen session must not
|
||||
outlive the answer to that theft. Not committed — the caller commits it with its change.
|
||||
|
||||
The user's row is locked before the tokens go, the lock `create_token` takes too: a token made at
|
||||
the same moment either commits first and is deleted here, or waits and sees the new epoch. FOR NO
|
||||
KEY UPDATE (`key_share`): the two still exclude each other, but a row that only refers to the user
|
||||
(a notice, an audit line) is not held up — a token's expiry notice would deadlock with it."""
|
||||
if user.id is None:
|
||||
user.session_epoch = (user.session_epoch or 0) + 1
|
||||
return
|
||||
epoch = db.execute(
|
||||
select(User.session_epoch).where(User.id == user.id).with_for_update(key_share=True)
|
||||
).scalar_one()
|
||||
user.session_epoch = (epoch or 0) + 1
|
||||
apitokens.revoke_all(db, user.id)
|
||||
|
||||
|
||||
def revoke_untrusted_preset_password(user: User) -> None:
|
||||
def revoke_untrusted_preset_password(db: Session, user: User) -> None:
|
||||
"""SEC-10 (account pre-emption): a password set on an account whose email was NEVER verified is
|
||||
untrusted — whoever registered the address (to pre-empt the real owner) could have chosen it. When
|
||||
the real owner attests ownership through Google, drop that password and bump the epoch to revoke any
|
||||
@@ -1183,15 +1197,15 @@ def revoke_untrusted_preset_password(user: User) -> None:
|
||||
_send_reset_if_eligible keyed off exactly that, so a revoked account silently got no reset link and
|
||||
was locked out for good (it could not sign in either). The stamp separates the two cases."""
|
||||
if not user.email_verified or (not user.is_active and user.password_hash is not None):
|
||||
_revoke_password(user)
|
||||
_revoke_password(db, user)
|
||||
|
||||
|
||||
def _revoke_password(user: User) -> None:
|
||||
def _revoke_password(db: Session, user: User) -> None:
|
||||
"""Drop the account's password, stamp why (so password-reset still accepts it) and end every
|
||||
session it signed in."""
|
||||
user.password_hash = None
|
||||
user.password_revoked_at = datetime.now(timezone.utc)
|
||||
bump_session_epoch(user)
|
||||
bump_session_epoch(db, user)
|
||||
|
||||
|
||||
@router.post("/logout-others")
|
||||
@@ -1203,7 +1217,7 @@ def logout_others(
|
||||
in somewhere" / a suspected session compromise."""
|
||||
# Commit the bump BEFORE re-stamping this cookie, so a failed commit can't strand the current
|
||||
# session at a newer epoch than the DB (see set_password).
|
||||
bump_session_epoch(user)
|
||||
bump_session_epoch(db, user)
|
||||
db.commit()
|
||||
_remember_epoch(request, user)
|
||||
log.info("Logout-others: uid=%s (epoch=%s)", user.id, user.session_epoch)
|
||||
@@ -1255,7 +1269,7 @@ def set_password(
|
||||
# SA4: a password change logs out every OTHER session; re-stamp this cookie at the new epoch so
|
||||
# the person making the change stays signed in here. Commit BEFORE re-stamping so a failed commit
|
||||
# can't leave the cookie at a newer epoch than the DB (which would wrongly 401 THIS session).
|
||||
bump_session_epoch(user)
|
||||
bump_session_epoch(db, user)
|
||||
db.commit()
|
||||
_remember_epoch(request, user)
|
||||
log.info("Password set/changed: uid=%s", user.id)
|
||||
|
||||
@@ -3,15 +3,16 @@ hidden, tags) and per-channel sync state. Local reads/writes plus YouTube-side w
|
||||
(subscribe / unsubscribe) that require the optional write scope."""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from functools import partial
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import Select, func, select, update
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.collation import collated, sorted_words
|
||||
from app.mailtext import lang_for_user
|
||||
from app import logs, quota
|
||||
from app import logs, quota, scheduler
|
||||
from app.auth import admin_user, current_user, require_write_scope
|
||||
from app.db import get_db
|
||||
from app.models import (
|
||||
@@ -481,7 +482,6 @@ class ChannelUpdateIn(BaseModel):
|
||||
def update_channel(
|
||||
channel_id: str,
|
||||
body: ChannelUpdateIn,
|
||||
background: BackgroundTasks,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
@@ -508,12 +508,19 @@ def update_channel(
|
||||
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_BACKFILL_RECENT):
|
||||
run_recent_backfill(db, [channel], max_channels=1)
|
||||
|
||||
# …and kick the FULL deep backfill for this one channel immediately in the background, so the
|
||||
# whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks. The
|
||||
# request returns right away; the channel page's live poll shows the count climb. The scheduler
|
||||
# still finishes a very large channel's tail on its normal runs (deep_requested stays set).
|
||||
if deep_turned_on:
|
||||
background.add_task(backfill_channel_now, channel_id, user.id)
|
||||
# …and kick the FULL deep backfill for this one channel immediately, as a recorded run of its own,
|
||||
# so the whole catalogue lands in seconds instead of over the scheduler's paced 10-minute ticks.
|
||||
# The request returns right away; the channel page's live poll shows the count climb. The
|
||||
# scheduler still finishes a very large channel's tail on its normal runs (deep_requested stays set).
|
||||
# A channel whose history is all in gets no run: it would only say "nothing to do", one kept row
|
||||
# per flip of the flag.
|
||||
if deep_turned_on and not (channel is not None and channel.backfill_done):
|
||||
scheduler.run_ad_hoc(
|
||||
scheduler.CHANNEL_BACKFILL,
|
||||
partial(backfill_channel_now, channel_id, user.id),
|
||||
actor_id=user.id,
|
||||
target=channel_id,
|
||||
)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
|
||||
@@ -61,7 +61,7 @@ def set_config(
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
s = sysconfig.spec(key)
|
||||
s = sysconfig.config_page_spec(key)
|
||||
if s is None:
|
||||
raise not_found("config_key")
|
||||
if "value" not in body.model_fields_set:
|
||||
@@ -89,7 +89,9 @@ def set_config(
|
||||
before=None if s.secret else {"value": _redact(old)},
|
||||
after=None if s.secret else {"value": _redact(new)},
|
||||
)
|
||||
db.commit()
|
||||
# The override is committed even when the trail skips it: re-saving the value in force still
|
||||
# pins it as an override, and an uncommitted write would be dropped with the session.
|
||||
db.commit()
|
||||
_apply_runtime(db, key)
|
||||
return sysconfig.describe(db)
|
||||
|
||||
@@ -100,7 +102,7 @@ def reset_config(
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
s = sysconfig.spec(key)
|
||||
s = sysconfig.config_page_spec(key)
|
||||
if s is None:
|
||||
raise not_found("config_key")
|
||||
old = None if s.secret else sysconfig.get(db, key)
|
||||
|
||||
+129
-26
@@ -27,13 +27,16 @@ from app.db import get_db
|
||||
from app.downloads import clip as clipmod
|
||||
from app.downloads import credentials as dlcreds
|
||||
from app.downloads import edit as editmod
|
||||
from app.downloads import folders
|
||||
from app.downloads import links as linksmod
|
||||
from app.downloads import quota, service, storage
|
||||
from app.downloads import urlguard
|
||||
from app.downloads.folders import FolderKind
|
||||
from app.downloads.status import FINISHED, NOT_REFETCHABLE, JobStatus
|
||||
from app.models import (
|
||||
DownloadJob,
|
||||
DownloadLink,
|
||||
FileFolder,
|
||||
DownloadProfile,
|
||||
DownloadQuota,
|
||||
DownloadShare,
|
||||
@@ -62,6 +65,7 @@ from app.routes.files import (
|
||||
)
|
||||
from app.security import hash_password
|
||||
from app.userscope import is_messageable_user, messageable_clauses
|
||||
from app.utils import iso
|
||||
from app.refusal import refused
|
||||
|
||||
router = APIRouter(prefix="/api/downloads", tags=["downloads"])
|
||||
@@ -716,8 +720,12 @@ def _own_link(db: Session, user: User, link_id: int) -> DownloadLink:
|
||||
link = db.get(DownloadLink, link_id)
|
||||
if link is None:
|
||||
raise not_found("link")
|
||||
job = db.get(DownloadJob, link.job_id)
|
||||
if job is None or job.user_id != user.id:
|
||||
target = (
|
||||
db.get(DownloadJob, link.job_id)
|
||||
if link.job_id is not None
|
||||
else db.get(FileFolder, link.folder_id)
|
||||
)
|
||||
if target is None or target.user_id != user.id:
|
||||
raise not_found("link")
|
||||
return link
|
||||
|
||||
@@ -730,21 +738,17 @@ def _link_expiry(days: int | None) -> datetime | None:
|
||||
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
|
||||
|
||||
|
||||
def _links_of(db: Session, *where) -> list[dict]:
|
||||
rows = db.execute(select(DownloadLink).where(*where).order_by(DownloadLink.id.desc()))
|
||||
return [linksmod.owner_view(row) for row in rows.scalars()]
|
||||
|
||||
|
||||
@router.get("/{job_id}/links")
|
||||
def list_links(
|
||||
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
) -> list[dict]:
|
||||
owned_or_404(db, DownloadJob, job_id, user, what="download")
|
||||
rows = (
|
||||
db.execute(
|
||||
select(DownloadLink)
|
||||
.where(DownloadLink.job_id == job_id)
|
||||
.order_by(DownloadLink.id.desc())
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
return [linksmod.owner_view(row) for row in rows]
|
||||
return _links_of(db, DownloadLink.job_id == job_id)
|
||||
|
||||
|
||||
class CreateLinkIn(BaseModel):
|
||||
@@ -758,6 +762,34 @@ class CreateLinkIn(BaseModel):
|
||||
expires_days: int | None = None
|
||||
|
||||
|
||||
def _new_link(
|
||||
db: Session, user: User, body: CreateLinkIn, **target: int
|
||||
) -> tuple[DownloadLink, str]:
|
||||
"""Make a link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and
|
||||
the token it is opened by — which is in hand only now."""
|
||||
pw = body.password.strip()
|
||||
token = linksmod.new_token()
|
||||
link = DownloadLink(
|
||||
**target,
|
||||
created_by=user.id,
|
||||
allow_download=body.allow_download,
|
||||
password_hash=hash_password(pw) if pw else None,
|
||||
expires_at=_link_expiry(body.expires_days),
|
||||
)
|
||||
linksmod.seal(link, token)
|
||||
db.add(link)
|
||||
db.flush()
|
||||
audit.record(
|
||||
db,
|
||||
user.id,
|
||||
audit.AuditAction.DOWNLOAD_LINK_CREATE,
|
||||
target_type="download_link",
|
||||
target_id=link.id,
|
||||
after={**target, **_link_audit(link)},
|
||||
)
|
||||
return link, token
|
||||
|
||||
|
||||
@router.post("/{job_id}/links")
|
||||
def create_link(
|
||||
job_id: int,
|
||||
@@ -765,22 +797,73 @@ def create_link(
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
|
||||
job = _shareable_job(db, user, job_id)
|
||||
if job.status != JobStatus.DONE or job.asset_id is None:
|
||||
raise refused(409, "share_unfinished")
|
||||
pw = body.password.strip()
|
||||
link = DownloadLink(
|
||||
job_id=job_id,
|
||||
token=linksmod.new_token(),
|
||||
created_by=user.id,
|
||||
allow_download=body.allow_download,
|
||||
password_hash=hash_password(pw) if pw else None,
|
||||
expires_at=_link_expiry(body.expires_days),
|
||||
)
|
||||
db.add(link)
|
||||
link, token = _new_link(db, user, body, job_id=job_id)
|
||||
db.commit()
|
||||
db.refresh(link)
|
||||
return linksmod.owner_view(link)
|
||||
return linksmod.owner_view(link, token=token)
|
||||
|
||||
|
||||
def _shareable_job(db: Session, user: User, job_id: int) -> DownloadJob:
|
||||
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
|
||||
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
|
||||
rather than failing the link's foreign key — or, done first, leaves a 404."""
|
||||
owned_or_404(db, DownloadJob, job_id, user, what="download")
|
||||
job = db.execute(
|
||||
select(DownloadJob)
|
||||
.where(DownloadJob.id == job_id)
|
||||
.with_for_update(read=True)
|
||||
.execution_options(populate_existing=True)
|
||||
).scalar_one_or_none()
|
||||
if job is None:
|
||||
raise not_found("download")
|
||||
return job
|
||||
|
||||
|
||||
def _shareable_folder(db: Session, user: User, folder_id: int) -> FileFolder:
|
||||
"""One of the user's own folders. A source folder lets its files go on its own clock, and the Hub
|
||||
is the user's channel to a machine of theirs: neither is a shelf to hand out. Held (FOR SHARE)
|
||||
for the rest of the transaction, so a delete of it at the same moment waits and takes the new
|
||||
link with it, rather than failing the link's foreign key."""
|
||||
owned_or_404(db, FileFolder, folder_id, user, what="folder")
|
||||
try:
|
||||
folder = folders.own_folder(db, user.id, folder_id)
|
||||
except folders.FolderError as e:
|
||||
raise refused(409, e.reason) from e
|
||||
if folder.kind != FolderKind.OWN:
|
||||
raise refused(409, "share_folder_kind")
|
||||
return folder
|
||||
|
||||
|
||||
@router.get("/folders/{folder_id}/links")
|
||||
def list_folder_links(
|
||||
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
) -> list[dict]:
|
||||
owned_or_404(db, FileFolder, folder_id, user, what="folder")
|
||||
return _links_of(db, DownloadLink.folder_id == folder_id)
|
||||
|
||||
|
||||
@router.post("/folders/{folder_id}/links")
|
||||
def create_folder_link(
|
||||
folder_id: int,
|
||||
body: CreateLinkIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
_shareable_folder(db, user, folder_id)
|
||||
link, token = _new_link(db, user, body, folder_id=folder_id)
|
||||
db.commit()
|
||||
return linksmod.owner_view(link, token=token)
|
||||
|
||||
|
||||
def _link_audit(link: DownloadLink) -> dict:
|
||||
"""A link's settings as the audit records them: whether it has a password, never which."""
|
||||
return {
|
||||
"allow_download": link.allow_download,
|
||||
"has_password": link.password_hash is not None,
|
||||
"expires_at": iso(link.expires_at),
|
||||
}
|
||||
|
||||
|
||||
class UpdateLinkIn(BaseModel):
|
||||
@@ -797,6 +880,7 @@ def update_link(
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
link = _own_link(db, user, link_id)
|
||||
before = _link_audit(link)
|
||||
sent = body.model_fields_set
|
||||
if "allow_download" in sent:
|
||||
link.allow_download = bool(body.allow_download)
|
||||
@@ -808,6 +892,17 @@ def update_link(
|
||||
# Bump the version so any grant issued under the previous password stops working now, rather
|
||||
# than living out its 6h TTL (setting/clearing/changing the password all count as a rotation).
|
||||
link.password_version += 1
|
||||
after = _link_audit(link)
|
||||
if after != before or "password" in sent:
|
||||
audit.record(
|
||||
db,
|
||||
user.id,
|
||||
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
|
||||
target_type="download_link",
|
||||
target_id=link_id,
|
||||
before=before,
|
||||
after=after,
|
||||
)
|
||||
db.commit()
|
||||
return linksmod.owner_view(link)
|
||||
|
||||
@@ -818,8 +913,16 @@ def revoke_link(
|
||||
) -> dict:
|
||||
link = _own_link(db, user, link_id)
|
||||
db.delete(link)
|
||||
audit.record(
|
||||
db,
|
||||
user.id,
|
||||
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
|
||||
target_type="download_link",
|
||||
target_id=link_id,
|
||||
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
|
||||
)
|
||||
db.commit()
|
||||
return {"deleted": link_id}
|
||||
return {"ok": True, "deleted": link_id}
|
||||
|
||||
|
||||
# --- admin ---------------------------------------------------------------------------------
|
||||
|
||||
+111
-38
@@ -7,6 +7,11 @@ bytes. Folders are rows (`downloads.folders`), so a rename or a move never touch
|
||||
In-flight jobs are not here: they are the download tray's (`routes.downloads`). The helpers above
|
||||
the routes are shared with that module — serialization, the ownership locks, the upload spool —
|
||||
because a download and a library file are one row at two moments.
|
||||
|
||||
A machine with an API token (`app.apitokens`) may use the routes that take `files_caller`: list the
|
||||
folders and files, fetch a file and its hash, upload, delete — each only in a folder its token was
|
||||
granted that right in, and never the files shared with the user. A folder or file out of its reach
|
||||
answers like a missing one. Everything else here is the signed-in user's alone (`current_user`).
|
||||
"""
|
||||
|
||||
import base64
|
||||
@@ -29,6 +34,7 @@ from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import audit, sysconfig
|
||||
from app.apitokens import FilesCaller, Right, files_caller
|
||||
from app.audit import AuditAction
|
||||
from app.auth import current_user
|
||||
from app.collation import collated
|
||||
@@ -41,6 +47,7 @@ from app.mailtext import lang_for_user
|
||||
from app.models import (
|
||||
DISPLAY_TEXT_MAX,
|
||||
DownloadJob,
|
||||
DownloadLink,
|
||||
DownloadShare,
|
||||
FileFolder,
|
||||
LibraryState,
|
||||
@@ -208,7 +215,7 @@ UPLOAD_FOLDER_HEADER = "x-upload-folder"
|
||||
@dataclass(frozen=True)
|
||||
class SpooledUpload:
|
||||
"""What the body dependency hands the handler: the bytes already on disk and their hash, plus
|
||||
the names and the folder that came alongside them."""
|
||||
the names and the folder that came alongside them, and who sent them."""
|
||||
|
||||
path: Path
|
||||
size: int
|
||||
@@ -217,6 +224,7 @@ class SpooledUpload:
|
||||
filename: str
|
||||
display_name: str | None
|
||||
folder_id: int | None
|
||||
via: str | None = None
|
||||
|
||||
|
||||
def _header_text(request: Request, name: str) -> str:
|
||||
@@ -233,15 +241,21 @@ def _header_text(request: Request, name: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _upload_folder(db: Session, user_id: int, raw: str) -> int | None:
|
||||
def _upload_folder(db: Session, caller: FilesCaller, raw: str) -> int | None:
|
||||
"""The folder id an upload asked for (`UPLOAD_FOLDER_HEADER`), checked; None when it asked for
|
||||
none."""
|
||||
none — which a token may not: it writes only where it was granted to."""
|
||||
if not raw.strip():
|
||||
if caller.grants is not None:
|
||||
raise refused(400, "api_token_folder_required")
|
||||
return None
|
||||
try:
|
||||
return folders.upload_target(db, user_id, int(raw), hold=False).id
|
||||
folder_id = int(raw)
|
||||
except ValueError as e:
|
||||
raise refused(409, "file_folder_gone") from e
|
||||
if not caller.may(folder_id, Right.WRITE):
|
||||
raise not_found("folder")
|
||||
try:
|
||||
return folders.upload_target(db, caller.user.id, folder_id, hold=False).id
|
||||
except folders.FolderError as e:
|
||||
raise refused(409, e.reason) from e
|
||||
|
||||
@@ -260,24 +274,26 @@ class _HashingSink:
|
||||
|
||||
async def spool_upload(
|
||||
request: Request,
|
||||
user: User = Depends(current_user),
|
||||
caller: FilesCaller = Depends(files_caller),
|
||||
db: Session = Depends(get_db),
|
||||
) -> AsyncIterator[SpooledUpload]:
|
||||
"""Stream the request body to a staging temp file, enforcing the per-file cap as it arrives.
|
||||
|
||||
`user` is taken but unused: it is the whole point. Dependencies resolve in declaration order, so
|
||||
naming `current_user` here makes the auth check a precondition of reading the body at all — a
|
||||
stranger is refused before a byte moves. The `yield` shape owns the temp file's life: ingest
|
||||
MOVES it into place on success (the unlink is then a no-op), and on any other exit — cap hit,
|
||||
empty, quota, probe or move error, a client that vanished mid-upload — it is removed."""
|
||||
`caller` comes first on purpose. Dependencies resolve in declaration order, so naming
|
||||
`files_caller` here makes the auth check a precondition of reading the body at all — a stranger,
|
||||
or a token without the right to write in that folder, is refused before a byte moves. The
|
||||
`yield` shape owns the temp file's life: ingest MOVES it into place on success (the unlink is
|
||||
then a no-op), and on any other exit — cap hit, empty, quota, probe or move error, a client that
|
||||
vanished mid-upload — it is removed."""
|
||||
cap = sysconfig.get_int(db, "download_upload_max_bytes")
|
||||
user_id = user.id
|
||||
user_id = caller.user.id
|
||||
via = caller.via
|
||||
# A folder the file may not go in is refused before the transfer, not after it.
|
||||
folder_id = _upload_folder(db, user_id, request.headers.get(UPLOAD_FOLDER_HEADER, ""))
|
||||
folder_id = _upload_folder(db, caller, request.headers.get(UPLOAD_FOLDER_HEADER, ""))
|
||||
# Hand the pooled connection back BEFORE the transfer starts — see `db.release_read_connection`
|
||||
# for why, and for the precondition it enforces. Everything this dependency still needs is
|
||||
# already an int or a header in hand: touching an ORM attribute below would reload `user` and
|
||||
# check a connection straight back out.
|
||||
# already an int, a string or a header in hand: touching an ORM attribute below (`caller.user`)
|
||||
# would reload it and check a connection straight back out.
|
||||
release_read_connection(db)
|
||||
staging = Path(settings.files_root) / ".staging" / "uploads"
|
||||
staging.mkdir(parents=True, exist_ok=True)
|
||||
@@ -300,6 +316,7 @@ async def spool_upload(
|
||||
filename=_header_text(request, UPLOAD_FILENAME_HEADER) or "upload",
|
||||
display_name=display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER)),
|
||||
folder_id=folder_id,
|
||||
via=via,
|
||||
)
|
||||
finally:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
@@ -322,10 +339,21 @@ def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob:
|
||||
return job
|
||||
|
||||
|
||||
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob:
|
||||
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
|
||||
own in a folder it may read."""
|
||||
if caller.grants is None:
|
||||
return accessible_job(db, caller.user, job_id)
|
||||
job = db.get(DownloadJob, job_id)
|
||||
if job is None or job.user_id != caller.user.id or not caller.may(job.folder_id, Right.READ):
|
||||
raise not_found("download")
|
||||
return job
|
||||
|
||||
|
||||
# --- folders -------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _serialize_folder(folder: FileFolder, count: int, size: int) -> dict:
|
||||
def _serialize_folder(folder: FileFolder, count: int, size: int, linked: bool = False) -> dict:
|
||||
return {
|
||||
"id": folder.id,
|
||||
"kind": folder.kind,
|
||||
@@ -336,27 +364,47 @@ def _serialize_folder(folder: FileFolder, count: int, size: int) -> dict:
|
||||
"count": count,
|
||||
"size_bytes": size,
|
||||
"created_at": iso(folder.created_at),
|
||||
# A live public link opens it: whatever is moved in is shown to whoever holds the link.
|
||||
"has_link": linked,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/folders")
|
||||
def list_folders(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
|
||||
"""The caller's folders with how many files each holds and their size. A source folder its
|
||||
files have all left stays (count 0): the client decides whether to show it."""
|
||||
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
|
||||
def _linked(db: Session, folder_ids: list[int]) -> set[int]:
|
||||
"""Which of these folders a live public link opens (one not past its expiry)."""
|
||||
now = datetime.now(timezone.utc)
|
||||
rows = db.execute(
|
||||
select(
|
||||
FileFolder,
|
||||
func.count(DownloadJob.id),
|
||||
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
|
||||
select(DownloadLink.folder_id).where(
|
||||
DownloadLink.folder_id.in_(folder_ids),
|
||||
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
|
||||
)
|
||||
.outerjoin(DownloadJob, done)
|
||||
).scalars()
|
||||
return {folder_id for folder_id in rows if folder_id is not None}
|
||||
|
||||
|
||||
@router.get("/folders")
|
||||
def list_folders(
|
||||
caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
|
||||
) -> list[dict]:
|
||||
"""The caller's folders with how many files each holds and their size (a token: the ones it may
|
||||
list). A source folder its files have all left stays (count 0): the client decides whether to
|
||||
show it."""
|
||||
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
|
||||
stmt = select(
|
||||
FileFolder,
|
||||
func.count(DownloadJob.id),
|
||||
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
|
||||
).where(FileFolder.user_id == caller.user.id)
|
||||
listable = caller.folders_with(Right.LIST)
|
||||
if listable is not None:
|
||||
stmt = stmt.where(FileFolder.id.in_(listable))
|
||||
rows = db.execute(
|
||||
stmt.outerjoin(DownloadJob, done)
|
||||
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
|
||||
.where(FileFolder.user_id == user.id)
|
||||
.group_by(FileFolder.id)
|
||||
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
|
||||
).all()
|
||||
return [_serialize_folder(f, count, int(size)) for f, count, size in rows]
|
||||
linked = _linked(db, [f.id for f, _, _ in rows])
|
||||
return [_serialize_folder(f, count, int(size), f.id in linked) for f, count, size in rows]
|
||||
|
||||
|
||||
class FolderIn(BaseModel):
|
||||
@@ -439,7 +487,7 @@ def update_folder(
|
||||
)
|
||||
db.commit()
|
||||
count, size = _folder_totals(db, folder.id)
|
||||
return _serialize_folder(folder, count, size)
|
||||
return _serialize_folder(folder, count, size, folder.id in _linked(db, [folder.id]))
|
||||
|
||||
|
||||
def _folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
|
||||
@@ -520,12 +568,20 @@ def list_files(
|
||||
sort: Literal["added", "name", "size"] = "added",
|
||||
order: Literal["asc", "desc"] = "desc",
|
||||
scope: Literal["mine", "shared"] = "mine",
|
||||
user: User = Depends(current_user),
|
||||
caller: FilesCaller = Depends(files_caller),
|
||||
db: Session = Depends(get_db),
|
||||
) -> list[dict]:
|
||||
"""The caller's done files (`scope=mine`, optionally one folder's), or the ones shared with
|
||||
them (`scope=shared`); searched by name and channel, sorted by when they were added, by name
|
||||
(in the reader's alphabet) or by size."""
|
||||
(in the reader's alphabet) or by size. A token lists only in the folders it may list, and never
|
||||
the shared ones."""
|
||||
user = caller.user
|
||||
listable = caller.folders_with(Right.LIST)
|
||||
if listable is not None:
|
||||
if scope == "shared":
|
||||
raise refused(403, "api_token_scope")
|
||||
if folder_id is not None and folder_id not in listable:
|
||||
raise not_found("folder")
|
||||
stmt = (
|
||||
select(DownloadJob)
|
||||
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
|
||||
@@ -540,6 +596,8 @@ def list_files(
|
||||
stmt = stmt.where(DownloadJob.user_id == user.id)
|
||||
if folder_id is not None:
|
||||
stmt = stmt.where(DownloadJob.folder_id == folder_id)
|
||||
elif listable is not None:
|
||||
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
|
||||
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
|
||||
term = q.strip()
|
||||
if term:
|
||||
@@ -693,11 +751,23 @@ def update_file(
|
||||
|
||||
@router.delete("/{job_id}")
|
||||
def delete_file(
|
||||
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
|
||||
) -> dict:
|
||||
"""Delete a file from the caller's library; the bytes go with their last holder."""
|
||||
job = own_job_for_update(db, user, job_id)
|
||||
audit.record(db, user.id, AuditAction.FILE_DELETE, target_type="file", target_id=job_id)
|
||||
"""Delete a file from the caller's library; the bytes go with their last holder. A token
|
||||
deletes only in a folder it may delete in — judged on the row as locked, so a file moved out of
|
||||
that folder meanwhile is out of its reach."""
|
||||
job = own_job_for_update(db, caller.user, job_id)
|
||||
if not caller.may(job.folder_id, Right.DELETE):
|
||||
db.rollback() # lets go of the locks; nothing was written
|
||||
raise not_found("download")
|
||||
audit.record(
|
||||
db,
|
||||
caller.user.id,
|
||||
AuditAction.FILE_DELETE,
|
||||
target_type="file",
|
||||
target_id=job_id,
|
||||
summary=caller.via,
|
||||
)
|
||||
service.delete(db, job) # commits, the audit row with it
|
||||
return {"ok": True, "deleted": job_id}
|
||||
|
||||
@@ -732,6 +802,7 @@ def ingest_spooled(db: Session, upload: SpooledUpload) -> DownloadJob:
|
||||
display_name=upload.display_name,
|
||||
sha256=upload.sha256,
|
||||
folder_id=upload.folder_id,
|
||||
via=upload.via,
|
||||
)
|
||||
except quota.QuotaExceeded as e:
|
||||
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
|
||||
@@ -782,8 +853,10 @@ def serve_file(db: Session, job: DownloadJob) -> FileResponse:
|
||||
|
||||
|
||||
@router.get("/{job_id}/file")
|
||||
def get_file(job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)):
|
||||
response = serve_file(db, accessible_job(db, user, job_id))
|
||||
def get_file(
|
||||
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
|
||||
):
|
||||
response = serve_file(db, readable_job(db, caller, job_id))
|
||||
db.commit()
|
||||
return response
|
||||
|
||||
@@ -840,13 +913,13 @@ def preview_file(
|
||||
|
||||
@router.get("/{job_id}/sha256")
|
||||
def get_sha256(
|
||||
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
job_id: int, caller: FilesCaller = Depends(files_caller), db: Session = Depends(get_db)
|
||||
) -> dict:
|
||||
"""The file's SHA-256 (hex), hashed now if it never was and kept on the asset for every holder.
|
||||
A sync handler: hashing a large file is blocking disk work, so it runs in the threadpool — and
|
||||
every other sync route shares that pool, so only `_HASHERS` files are hashed at a time (a caller
|
||||
past that is told to come back), and none of them holds a database connection while it reads."""
|
||||
asset, path = _ready_file(db, accessible_job(db, user, job_id))
|
||||
asset, path = _ready_file(db, readable_job(db, caller, job_id))
|
||||
if asset.sha256:
|
||||
return {"sha256": asset.sha256}
|
||||
asset_id = asset.id
|
||||
|
||||
+149
-53
@@ -9,7 +9,7 @@ a short-lived signed grant that the media URL carries (a `<video src>` can't sen
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from fastapi.responses import FileResponse
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import select, update
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import settings
|
||||
@@ -17,7 +17,7 @@ from app.db import get_db
|
||||
from app.downloads import links as linksmod
|
||||
from app.downloads import storage
|
||||
from app.downloads.status import JobStatus
|
||||
from app.models import DownloadJob, DownloadLink, MediaAsset, User
|
||||
from app.models import DownloadJob, DownloadLink, FileFolder, MediaAsset, User
|
||||
from app.ratelimit import RateLimiter, client_key
|
||||
from app.security import verify_password
|
||||
from app.refusal import refused
|
||||
@@ -38,7 +38,7 @@ _unlock_failures = RateLimiter(max_events=50, window_seconds=300)
|
||||
|
||||
|
||||
def _link_or_404(db: Session, token: str) -> DownloadLink:
|
||||
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
|
||||
link = linksmod.find(db, token)
|
||||
if link is None or linksmod.is_expired(link):
|
||||
raise refused(404, "link_gone")
|
||||
return link
|
||||
@@ -55,8 +55,7 @@ def _being_made(db: Session, job: DownloadJob) -> bool:
|
||||
return owner is not None and not owner.is_suspended
|
||||
|
||||
|
||||
def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAsset]:
|
||||
job = db.get(DownloadJob, link.job_id)
|
||||
def _ready_job(db: Session, job: DownloadJob | None) -> tuple[DownloadJob, MediaAsset]:
|
||||
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
|
||||
if (
|
||||
job is not None
|
||||
@@ -79,36 +78,114 @@ def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAs
|
||||
return job, asset
|
||||
|
||||
|
||||
def _ready_asset(db: Session, link: DownloadLink) -> MediaAsset:
|
||||
return _ready_target(db, link)[1]
|
||||
def _ready_target(db: Session, link: DownloadLink) -> tuple[DownloadJob, MediaAsset]:
|
||||
"""A file link's file. A folder link names no one file: its files are asked for by id."""
|
||||
if link.job_id is None:
|
||||
raise refused(404, "video_gone")
|
||||
return _ready_job(db, db.get(DownloadJob, link.job_id))
|
||||
|
||||
|
||||
def _file_url(token: str, grant: str | None = None) -> str:
|
||||
base = f"/api/public/watch/{token}/file"
|
||||
return f"{base}?g={grant}" if grant else base
|
||||
def _link_folder(db: Session, link: DownloadLink) -> FileFolder:
|
||||
folder = db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
|
||||
if folder is None:
|
||||
raise refused(404, "link_gone")
|
||||
return folder
|
||||
|
||||
|
||||
def _poster_url(token: str, asset, grant: str | None = None) -> str | None:
|
||||
def _folder_job(db: Session, link: DownloadLink, job_id: int) -> tuple[DownloadJob, MediaAsset]:
|
||||
folder = _link_folder(db, link)
|
||||
job = db.execute(
|
||||
select(DownloadJob).where(DownloadJob.id == job_id, *linksmod.in_folder(folder))
|
||||
).scalar_one_or_none()
|
||||
if job is None:
|
||||
raise refused(404, "video_gone")
|
||||
return _ready_job(db, job)
|
||||
|
||||
|
||||
def _base(token: str, job_id: int | None) -> str:
|
||||
"""A file link's file lives under its token; a folder link's files, each under its id."""
|
||||
return f"/api/public/watch/{token}" + (f"/files/{job_id}" if job_id is not None else "")
|
||||
|
||||
|
||||
def _file_url(token: str, grant: str | None = None, job_id: int | None = None) -> str:
|
||||
url = f"{_base(token, job_id)}/file"
|
||||
return f"{url}?g={grant}" if grant else url
|
||||
|
||||
|
||||
def _poster_url(
|
||||
token: str, asset: MediaAsset, grant: str | None = None, job_id: int | None = None
|
||||
) -> str | None:
|
||||
if not asset.poster_path:
|
||||
return None
|
||||
base = f"/api/public/watch/{token}/poster.jpg"
|
||||
return f"{base}?g={grant}" if grant else base
|
||||
url = f"{_base(token, job_id)}/poster.jpg"
|
||||
return f"{url}?g={grant}" if grant else url
|
||||
|
||||
|
||||
def _folder_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
|
||||
"""A folder link's page: the folder's name and each file's own watch metadata
|
||||
(`linksmod.folder_files`: ready, on disk, newest first)."""
|
||||
folder = _link_folder(db, link)
|
||||
files = [
|
||||
{
|
||||
"id": job.id,
|
||||
**linksmod.public_meta(
|
||||
link,
|
||||
job,
|
||||
asset,
|
||||
_file_url(token, grant, job.id),
|
||||
_poster_url(token, asset, grant, job.id),
|
||||
),
|
||||
}
|
||||
for job, asset in linksmod.folder_files(db, folder)
|
||||
]
|
||||
return {
|
||||
"needs_password": False,
|
||||
"kind": "folder",
|
||||
"name": folder.name,
|
||||
"allow_download": link.allow_download,
|
||||
"files": files,
|
||||
}
|
||||
|
||||
|
||||
def _file_meta(db: Session, link: DownloadLink, token: str, grant: str | None) -> dict:
|
||||
job, asset = _ready_target(db, link)
|
||||
return {
|
||||
"needs_password": False,
|
||||
"kind": "file",
|
||||
**linksmod.public_meta(
|
||||
link, job, asset, _file_url(token, grant), _poster_url(token, asset, grant)
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _opened(db: Session, link: DownloadLink, token: str, grant: str | None = None) -> dict:
|
||||
"""The page an open link shows, counted as one view (the handler commits the count)."""
|
||||
meta = (
|
||||
_folder_meta(db, link, token, grant)
|
||||
if link.folder_id is not None
|
||||
else _file_meta(db, link, token, grant)
|
||||
)
|
||||
# In SQL, not `+= 1` on the loaded row: two opens at once would both write the same count.
|
||||
db.execute(
|
||||
update(DownloadLink)
|
||||
.where(DownloadLink.id == link.id)
|
||||
.values(view_count=DownloadLink.view_count + 1)
|
||||
.execution_options(synchronize_session=False)
|
||||
)
|
||||
return meta
|
||||
|
||||
|
||||
@router.get("/watch/{token}")
|
||||
def watch_meta(token: str, db: Session = Depends(get_db)) -> dict:
|
||||
"""Public metadata for the watch page. For a password link, returns only `needs_password`
|
||||
until the viewer unlocks it (no title/thumbnail leak)."""
|
||||
"""Public metadata for the watch page: one file's, or a folder's list. For a password link,
|
||||
returns only `needs_password` until the viewer unlocks it (no title/thumbnail leak)."""
|
||||
link = _link_or_404(db, token)
|
||||
if link.password_hash:
|
||||
return {"needs_password": True}
|
||||
job, asset = _ready_target(db, link)
|
||||
link.view_count += 1
|
||||
# Which kind it opens, for the password prompt's title — never what is in it.
|
||||
return {"needs_password": True, "kind": "folder" if link.folder_id is not None else "file"}
|
||||
meta = _opened(db, link, token)
|
||||
db.commit()
|
||||
return {
|
||||
"needs_password": False,
|
||||
**linksmod.public_meta(link, job, asset, _file_url(token), _poster_url(token, asset)),
|
||||
}
|
||||
return meta
|
||||
|
||||
|
||||
class WatchUnlockIn(BaseModel):
|
||||
@@ -119,7 +196,7 @@ class WatchUnlockIn(BaseModel):
|
||||
def watch_unlock(
|
||||
token: str, body: WatchUnlockIn, request: Request, db: Session = Depends(get_db)
|
||||
) -> dict:
|
||||
"""Exchange the link password for a signed grant carried by the media URL."""
|
||||
"""Exchange the link password for a signed grant carried by the media URLs."""
|
||||
# The link first: a token naming none must leave no limiter entry behind, or posting made-up tokens
|
||||
# would grow the limiter without bound.
|
||||
link = _link_or_404(db, token)
|
||||
@@ -128,36 +205,27 @@ def watch_unlock(
|
||||
raise refused(429, "too_many_attempts")
|
||||
if not link.password_hash:
|
||||
# Not a password link — nothing to unlock; just hand back the plain meta.
|
||||
job, asset = _ready_target(db, link)
|
||||
return {
|
||||
"ok": True,
|
||||
"needs_password": False,
|
||||
**linksmod.public_meta(link, job, asset, _file_url(token), _poster_url(token, asset)),
|
||||
}
|
||||
meta = _opened(db, link, token)
|
||||
db.commit()
|
||||
return {"ok": True, **meta}
|
||||
if not verify_password(body.password, link.password_hash):
|
||||
_unlock_failures.allow(token)
|
||||
raise refused(403, "wrong_share_password")
|
||||
job, asset = _ready_target(db, link)
|
||||
link.view_count += 1
|
||||
db.commit()
|
||||
grant = linksmod.make_grant(token, link.password_version)
|
||||
return {
|
||||
"ok": True,
|
||||
"needs_password": False,
|
||||
**linksmod.public_meta(
|
||||
link, job, asset, _file_url(token, grant), _poster_url(token, asset, grant)
|
||||
),
|
||||
}
|
||||
meta = _opened(db, link, token, grant)
|
||||
db.commit()
|
||||
return {"ok": True, **meta}
|
||||
|
||||
|
||||
@router.get("/watch/{token}/file")
|
||||
def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
|
||||
"""Range-aware stream of the shared file. Inline by default (plays in the page); an
|
||||
`allow_download` link serves it as an attachment. Password links require a valid grant."""
|
||||
link = _link_or_404(db, token)
|
||||
def _granted(link: DownloadLink, token: str, g: str | None) -> None:
|
||||
"""A password link's files want the grant its unlock handed out."""
|
||||
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
|
||||
raise refused(403, "share_password_required")
|
||||
asset = _ready_asset(db, link)
|
||||
|
||||
|
||||
def _serve(link: DownloadLink, asset: MediaAsset) -> FileResponse:
|
||||
"""Range-aware stream of a shared file. Inline by default (plays in the page); an
|
||||
`allow_download` link serves it as an attachment."""
|
||||
path = storage.safe_abs_path(settings.files_root, asset.rel_path)
|
||||
if path is None:
|
||||
raise refused(404, "video_gone")
|
||||
@@ -184,17 +252,45 @@ def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
|
||||
)
|
||||
|
||||
|
||||
@router.get("/watch/{token}/poster.jpg")
|
||||
def watch_poster(token: str, g: str | None = None, db: Session = Depends(get_db)):
|
||||
"""The generated poster frame for a shared thumbnail-less video (used as the page/video poster
|
||||
and the link-preview image). Password links require a valid grant, like the file."""
|
||||
link = _link_or_404(db, token)
|
||||
if link.password_hash and not linksmod.check_grant(token, g, link.password_version):
|
||||
raise refused(403, "share_password_required")
|
||||
job, asset = _ready_target(db, link)
|
||||
def _poster(asset: MediaAsset) -> FileResponse:
|
||||
if not asset.poster_path:
|
||||
raise not_found("poster")
|
||||
path = storage.safe_abs_path(settings.files_root, asset.poster_path)
|
||||
if path is None:
|
||||
raise not_found("poster")
|
||||
return FileResponse(path, media_type="image/jpeg")
|
||||
|
||||
|
||||
@router.get("/watch/{token}/file")
|
||||
def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
|
||||
"""A file link's file. Password links require a valid grant."""
|
||||
link = _link_or_404(db, token)
|
||||
_granted(link, token, g)
|
||||
return _serve(link, _ready_target(db, link)[1])
|
||||
|
||||
|
||||
@router.get("/watch/{token}/poster.jpg")
|
||||
def watch_poster(token: str, g: str | None = None, db: Session = Depends(get_db)):
|
||||
"""The generated poster frame for a shared thumbnail-less video (used as the page/video poster
|
||||
and the link-preview image). Password links require a valid grant, like the file."""
|
||||
link = _link_or_404(db, token)
|
||||
_granted(link, token, g)
|
||||
return _poster(_ready_target(db, link)[1])
|
||||
|
||||
|
||||
@router.get("/watch/{token}/files/{job_id}/file")
|
||||
def watch_folder_file(token: str, job_id: int, g: str | None = None, db: Session = Depends(get_db)):
|
||||
"""One file of a folder link's folder, while it is in it. Password links require a grant."""
|
||||
link = _link_or_404(db, token)
|
||||
_granted(link, token, g)
|
||||
return _serve(link, _folder_job(db, link, job_id)[1])
|
||||
|
||||
|
||||
@router.get("/watch/{token}/files/{job_id}/poster.jpg")
|
||||
def watch_folder_poster(
|
||||
token: str, job_id: int, g: str | None = None, db: Session = Depends(get_db)
|
||||
):
|
||||
"""The poster frame of one file of a folder link's folder, under the same terms as the file."""
|
||||
link = _link_or_404(db, token)
|
||||
_granted(link, token, g)
|
||||
return _poster(_folder_job(db, link, job_id)[1])
|
||||
|
||||
@@ -6,7 +6,7 @@ The aggregate daily budget lives in ApiQuotaUsage; here we read the per-spend au
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -14,6 +14,8 @@ from app import quota
|
||||
from app.auth import admin_user, current_user
|
||||
from app.db import get_db
|
||||
from app.models import QuotaEvent, User
|
||||
from app.routes import _account as account
|
||||
from app.routes._keyset import utc
|
||||
|
||||
router = APIRouter(prefix="/api/quota", tags=["quota"])
|
||||
|
||||
@@ -45,23 +47,32 @@ def my_usage(user: User = Depends(current_user), db: Session = Depends(get_db))
|
||||
|
||||
@router.get("/admin")
|
||||
def admin_usage(
|
||||
days: int = 30,
|
||||
since: datetime | None = None,
|
||||
until: datetime | None = None,
|
||||
user: str | None = Query(default=None, pattern=account.PATTERN),
|
||||
_: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
days = max(1, min(days, 365))
|
||||
since = datetime.now(timezone.utc) - timedelta(days=days)
|
||||
"""Spend per account (and per action) and per day, between `since` (default: 30 days back) and
|
||||
`until` (default: now), optionally for one account (`none` = the background)."""
|
||||
now = datetime.now(timezone.utc)
|
||||
lo = utc(since) or now - timedelta(days=30)
|
||||
window = [QuotaEvent.created_at >= lo]
|
||||
if until is not None:
|
||||
window.append(QuotaEvent.created_at < utc(until))
|
||||
if user is not None:
|
||||
window.append(account.condition(QuotaEvent.user_id, user))
|
||||
|
||||
totals = db.execute(
|
||||
select(QuotaEvent.user_id, func.sum(QuotaEvent.units))
|
||||
.where(QuotaEvent.created_at >= since)
|
||||
.where(*window)
|
||||
.group_by(QuotaEvent.user_id)
|
||||
).all()
|
||||
|
||||
per_action: dict[int | None, dict[str, int]] = {}
|
||||
for uid, action, units in db.execute(
|
||||
select(QuotaEvent.user_id, QuotaEvent.action, func.sum(QuotaEvent.units))
|
||||
.where(QuotaEvent.created_at >= since)
|
||||
.where(*window)
|
||||
.group_by(QuotaEvent.user_id, QuotaEvent.action)
|
||||
).all():
|
||||
per_action.setdefault(uid, {})[action] = int(units)
|
||||
@@ -84,16 +95,16 @@ def admin_usage(
|
||||
]
|
||||
rows.sort(key=lambda r: r["total"], reverse=True)
|
||||
|
||||
# Instance-wide daily series in Pacific days (matches the budget reset).
|
||||
# The daily series in Pacific days (matches the budget reset), of the one account when filtered.
|
||||
day_expr = func.date(func.timezone("America/Los_Angeles", QuotaEvent.created_at))
|
||||
daily = [
|
||||
{"day": str(day), "total": int(units)}
|
||||
for day, units in db.execute(
|
||||
select(day_expr, func.sum(QuotaEvent.units))
|
||||
.where(QuotaEvent.created_at >= since)
|
||||
.where(*window)
|
||||
.group_by(day_expr)
|
||||
.order_by(day_expr)
|
||||
).all()
|
||||
]
|
||||
|
||||
return {"range_days": days, "rows": rows, "daily": daily}
|
||||
return {"rows": rows, "daily": daily}
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
"""Admin background-run history: read `job_runs`, a page at a time.
|
||||
|
||||
Admin-only and read-only — rows leave the table through the retention job
|
||||
(`job_run_retention_days`). Filtered on the server and paged by `_keyset`, like the server log and
|
||||
the audit log beside it on the activity page. A row names what a reader needs without a second
|
||||
request: the channel a user's ad-hoc run worked on, and how many log lines the run left (only
|
||||
WARNING and up are stored, so most runs have none — the count says so before the click).
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import scheduler
|
||||
from app.auth import admin_user
|
||||
from app.db import get_db
|
||||
from app.models import Channel, JobRun, LogEvent, User
|
||||
from app.routes import _account as account
|
||||
from app.routes._keyset import PAGE_MAX, accounts, older_than, page, utc
|
||||
from app.runstatus import RunStatus
|
||||
from app.utils import iso, like_escape
|
||||
|
||||
router = APIRouter(prefix="/api/admin/runs", tags=["admin"])
|
||||
|
||||
Facet = Literal["job", "status"]
|
||||
|
||||
|
||||
def _serialize(run: JobRun, titles: dict[str, str], lines: dict[int, int]) -> dict:
|
||||
return {
|
||||
"id": run.id,
|
||||
"job": run.job,
|
||||
"trigger": run.trigger,
|
||||
"actor_id": run.actor_id,
|
||||
"target": run.target,
|
||||
"target_title": titles.get(run.target) if run.target is not None else None,
|
||||
"status": run.status,
|
||||
"started_at": iso(run.started_at),
|
||||
"finished_at": iso(run.finished_at),
|
||||
"summary": run.summary,
|
||||
"result": run.result,
|
||||
"error": run.error,
|
||||
"failure": run.failure,
|
||||
"quota_units": run.quota_units,
|
||||
"log_lines": lines.get(run.id, 0),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RunFilters:
|
||||
job: str | None
|
||||
status: RunStatus | None
|
||||
actor: str | None
|
||||
q: str | None
|
||||
since: datetime | None
|
||||
until: datetime | None
|
||||
|
||||
def conditions(self, skip: Facet | None = None) -> list:
|
||||
"""The WHERE conditions, ANDed. `skip` leaves one menu's own filter out, so that menu counts
|
||||
its values under every other filter (the way back from a pick)."""
|
||||
out = []
|
||||
if self.job is not None and skip != "job":
|
||||
out.append(JobRun.job == self.job)
|
||||
if self.status is not None and skip != "status":
|
||||
out.append(JobRun.status == self.status)
|
||||
if self.actor is not None:
|
||||
out.append(account.condition(JobRun.actor_id, self.actor))
|
||||
if self.since is not None:
|
||||
out.append(JobRun.started_at >= utc(self.since))
|
||||
if self.until is not None:
|
||||
out.append(JobRun.started_at < utc(self.until))
|
||||
if self.q and self.q.strip():
|
||||
like = f"%{like_escape(self.q.strip())}%"
|
||||
out.append(
|
||||
or_(
|
||||
JobRun.job.ilike(like),
|
||||
JobRun.target.ilike(like),
|
||||
JobRun.summary.ilike(like),
|
||||
JobRun.error.ilike(like),
|
||||
)
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def run_filters(
|
||||
job: str | None = Query(default=None, max_length=40, pattern=r"^[a-z][a-z0-9_]*$"),
|
||||
status: RunStatus | None = None,
|
||||
user: str | None = Query(default=None, pattern=account.PATTERN),
|
||||
q: str | None = Query(default=None, max_length=200),
|
||||
since: datetime | None = None,
|
||||
until: datetime | None = None,
|
||||
) -> RunFilters:
|
||||
return RunFilters(job, status, user, q, since, until)
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_runs(
|
||||
filters: RunFilters = Depends(run_filters),
|
||||
before: str | None = Query(default=None, max_length=200),
|
||||
limit: int = Query(default=100, ge=1, le=PAGE_MAX),
|
||||
_: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""One page of runs, newest first, matching every filter given; `before` continues from a
|
||||
previous page's `next_cursor`."""
|
||||
query = select(JobRun).where(*filters.conditions())
|
||||
if before is not None:
|
||||
query = query.where(older_than(JobRun.started_at, JobRun.id, before))
|
||||
order = (JobRun.started_at.desc(), JobRun.id.desc())
|
||||
runs, cursor = page(
|
||||
list(db.execute(query.order_by(*order).limit(limit + 1)).scalars().all()),
|
||||
limit,
|
||||
lambda r: r.started_at,
|
||||
)
|
||||
channels = {r.target for r in runs if r.job in scheduler.AD_HOC_JOBS and r.target}
|
||||
titles: dict[str, str] = {}
|
||||
if channels:
|
||||
rows = db.execute(select(Channel.id, Channel.title).where(Channel.id.in_(channels)))
|
||||
titles = {cid: title for cid, title in rows.tuples() if title is not None}
|
||||
lines: dict[int, int] = {}
|
||||
if runs:
|
||||
counted = db.execute(
|
||||
select(LogEvent.run_id, func.count())
|
||||
.where(LogEvent.run_id.in_([r.id for r in runs]))
|
||||
.group_by(LogEvent.run_id)
|
||||
)
|
||||
lines = {rid: n for rid, n in counted.tuples() if rid is not None}
|
||||
return {
|
||||
"items": [_serialize(r, titles, lines) for r in runs],
|
||||
"accounts": accounts(db, {r.actor_id for r in runs if r.actor_id is not None}),
|
||||
"next_cursor": cursor,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/facets")
|
||||
def run_facets(
|
||||
filters: RunFilters = Depends(run_filters),
|
||||
_: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""What the job menu and the status chips offer, each counted under every filter but its own."""
|
||||
|
||||
def counts(col, skip: Facet) -> list[dict]:
|
||||
n = func.count().label("n")
|
||||
stmt = select(col, n).where(*filters.conditions(skip)).group_by(col).order_by(n.desc(), col)
|
||||
return [{"value": v, "count": c} for v, c in db.execute(stmt).all()]
|
||||
|
||||
total = select(func.count()).select_from(JobRun).where(*filters.conditions("status"))
|
||||
return {
|
||||
"jobs": counts(JobRun.job, "job"),
|
||||
"statuses": counts(JobRun.status, "status"),
|
||||
"total": db.scalar(total) or 0,
|
||||
}
|
||||
@@ -17,8 +17,9 @@ from app.scheduler import (
|
||||
JOB_INTERVALS,
|
||||
MAX_INTERVAL,
|
||||
MIN_INTERVAL,
|
||||
apply_interval,
|
||||
reschedule,
|
||||
scheduler_snapshot,
|
||||
stage_interval,
|
||||
trigger_all,
|
||||
trigger_job,
|
||||
)
|
||||
@@ -42,14 +43,14 @@ def set_job_interval(
|
||||
admin: User = Depends(admin_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Change how often a scheduler job runs (minutes). Persisted as an override and applied
|
||||
to the live scheduler immediately."""
|
||||
"""Change how often a scheduler job runs (minutes). Stored as an override together with its
|
||||
audit row, then applied to the live scheduler."""
|
||||
if job_id not in JOB_INTERVALS:
|
||||
raise not_found("job")
|
||||
minutes = body.interval_minutes
|
||||
if not (MIN_INTERVAL <= minutes <= MAX_INTERVAL):
|
||||
raise refused(400, "invalid_interval", min=MIN_INTERVAL, max=MAX_INTERVAL)
|
||||
applied = apply_interval(db, job_id, minutes)
|
||||
applied = stage_interval(db, job_id, minutes)
|
||||
audit.record(
|
||||
db,
|
||||
admin.id,
|
||||
@@ -60,6 +61,7 @@ def set_job_interval(
|
||||
after={"interval_minutes": applied},
|
||||
)
|
||||
db.commit()
|
||||
reschedule(job_id, applied)
|
||||
return {"ok": True, "id": job_id, "interval_minutes": applied}
|
||||
|
||||
|
||||
@@ -117,7 +119,7 @@ def set_maintenance_settings(
|
||||
|
||||
@router.get("")
|
||||
def get_scheduler(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
|
||||
snap = scheduler_snapshot()
|
||||
snap = scheduler_snapshot(db)
|
||||
|
||||
# Work still queued (shared catalog, so these are instance-wide, not per-user).
|
||||
deep_requested = (
|
||||
|
||||
@@ -91,12 +91,12 @@ def setup_admin(body: SetupAdminIn, db: Session = Depends(require_setup)) -> dic
|
||||
|
||||
@router.post("/config")
|
||||
def setup_config(settings_map: dict[str, Any], db: Session = Depends(require_setup)) -> dict:
|
||||
"""Persist provided DB-backed settings (Google creds, SMTP, quota…). Keys must be in the
|
||||
sysconfig registry; empty values are skipped (left at the default). Genuinely dynamic
|
||||
"""Persist provided DB-backed settings (Google creds, SMTP, quota…). Keys must be the
|
||||
Configuration page's; empty values are skipped (left at the default). Genuinely dynamic
|
||||
key→value map, so it stays a typed dict rather than a fixed model."""
|
||||
saved: list[str] = []
|
||||
for key, value in settings_map.items():
|
||||
if sysconfig.spec(key) is None:
|
||||
if sysconfig.config_page_spec(key) is None:
|
||||
raise refused(400, "unknown_setting", key=key)
|
||||
if value is None or value == "":
|
||||
continue
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
"""A user's API tokens for their Files library (`app.apitokens`): list, make, revoke.
|
||||
|
||||
Signed-in only — `current_user`, never a token: a token cannot make itself a wider one. A new
|
||||
token's value is in the answer that made it and nowhere after; only its hash is kept.
|
||||
|
||||
The Hub preset is the one machine a user swaps files with: it makes (or reuses) their Hub folder
|
||||
and gives the token every right in it, so the hub both drops files there and picks up the ones put
|
||||
there for it.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import apitokens, audit
|
||||
from app.apitokens import GRANT_COLUMNS, Right
|
||||
from app.audit import AuditAction
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.downloads import folders
|
||||
from app.models import ApiToken, ApiTokenGrant, FileFolder, User
|
||||
from app.refusal import refused
|
||||
from app.routes._common import owned_or_404
|
||||
from app.security import hash_token
|
||||
from app.utils import iso
|
||||
|
||||
router = APIRouter(prefix="/api/tokens", tags=["tokens"])
|
||||
|
||||
# Far more than one person's machines; a ceiling, so a script gone wrong cannot mint without end.
|
||||
MAX_TOKENS = 20
|
||||
_MAX_GRANTS = 50
|
||||
_MAX_EXPIRY_DAYS = 3650
|
||||
|
||||
|
||||
class GrantIn(BaseModel):
|
||||
folder_id: int
|
||||
rights: list[Right] = Field(min_length=1, max_length=len(Right))
|
||||
|
||||
|
||||
class TokenIn(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=80)
|
||||
# None = never expires.
|
||||
expires_days: int | None = Field(default=None, ge=1, le=_MAX_EXPIRY_DAYS)
|
||||
preset: Literal["hub"] | None = None
|
||||
grants: list[GrantIn] = Field(default_factory=list, max_length=_MAX_GRANTS)
|
||||
|
||||
|
||||
def _serialize(db: Session, token: ApiToken) -> dict:
|
||||
rows = db.execute(
|
||||
select(ApiTokenGrant, FileFolder)
|
||||
.join(FileFolder, FileFolder.id == ApiTokenGrant.folder_id)
|
||||
.where(ApiTokenGrant.token_id == token.id)
|
||||
.order_by(FileFolder.id)
|
||||
).all()
|
||||
now = datetime.now(timezone.utc)
|
||||
return {
|
||||
"id": token.id,
|
||||
"name": token.name,
|
||||
"prefix": token.prefix,
|
||||
"created_at": iso(token.created_at),
|
||||
"expires_at": iso(token.expires_at),
|
||||
"expired": token.expires_at is not None and token.expires_at <= now,
|
||||
"expiring": apitokens.expiring(token, now),
|
||||
"last_used_at": iso(token.last_used_at),
|
||||
"grants": [
|
||||
{
|
||||
"folder_id": folder.id,
|
||||
"kind": folder.kind,
|
||||
"source_key": folder.source_key,
|
||||
"name": folder.name,
|
||||
"rights": sorted(apitokens.grant_rights(grant)),
|
||||
}
|
||||
for grant, folder in rows
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_tokens(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
|
||||
tokens = (
|
||||
db.execute(select(ApiToken).where(ApiToken.user_id == user.id).order_by(ApiToken.id))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
return [_serialize(db, t) for t in tokens]
|
||||
|
||||
|
||||
def _wanted_grants(db: Session, user: User, body: TokenIn) -> dict[int, set[Right]]:
|
||||
"""Folder id → the rights the new token gets there; every folder checked to be the user's, and
|
||||
writing only where an upload may land (`folders.upload_target`)."""
|
||||
if body.preset == "hub":
|
||||
if body.grants:
|
||||
raise refused(400, "api_token_preset_grants")
|
||||
return {folders.hub_folder(db, user.id).id: set(Right)}
|
||||
if not body.grants:
|
||||
raise refused(400, "api_token_no_grants")
|
||||
wanted: dict[int, set[Right]] = {}
|
||||
for grant in body.grants:
|
||||
wanted.setdefault(grant.folder_id, set()).update(grant.rights)
|
||||
for folder_id, rights in wanted.items():
|
||||
try:
|
||||
if Right.WRITE in rights:
|
||||
folders.upload_target(db, user.id, folder_id)
|
||||
else:
|
||||
folders.own_folder(db, user.id, folder_id)
|
||||
except folders.FolderError as e:
|
||||
raise refused(409, e.reason) from e
|
||||
return wanted
|
||||
|
||||
|
||||
@router.post("")
|
||||
def create_token(
|
||||
body: TokenIn, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
) -> dict:
|
||||
"""Make a token; its value is in this answer only."""
|
||||
name = body.name.strip()
|
||||
if not name:
|
||||
raise refused(400, "api_token_name_required")
|
||||
# The user's row, locked: two requests at once cannot both slip under the ceiling, and a
|
||||
# session ended since `current_user` read the epoch (unlocked) cannot mint a token that outlives
|
||||
# it — `bump_session_epoch` takes this lock before it revokes them all. No key update: a row that
|
||||
# only refers to the user (a folder delete's audit line) must not wait on it, or the two deadlock.
|
||||
epoch = db.execute(
|
||||
select(User.session_epoch).where(User.id == user.id).with_for_update(key_share=True)
|
||||
).scalar_one()
|
||||
if (epoch or 0) != (user.session_epoch or 0):
|
||||
raise refused(401, "not_authenticated")
|
||||
count = db.execute(
|
||||
select(func.count(ApiToken.id)).where(ApiToken.user_id == user.id)
|
||||
).scalar_one()
|
||||
if count >= MAX_TOKENS:
|
||||
raise refused(409, "api_token_limit", limit=MAX_TOKENS)
|
||||
wanted = _wanted_grants(db, user, body)
|
||||
raw = apitokens.mint()
|
||||
token = ApiToken(
|
||||
user_id=user.id,
|
||||
name=name,
|
||||
prefix=raw[: apitokens.SHOWN_LEN],
|
||||
token_hash=hash_token(raw),
|
||||
expires_at=None
|
||||
if body.expires_days is None
|
||||
else datetime.now(timezone.utc) + timedelta(days=body.expires_days),
|
||||
)
|
||||
db.add(token)
|
||||
db.flush()
|
||||
for folder_id, rights in wanted.items():
|
||||
db.add(
|
||||
ApiTokenGrant(
|
||||
token_id=token.id,
|
||||
folder_id=folder_id,
|
||||
**{col: right in rights for right, col in GRANT_COLUMNS.items()},
|
||||
)
|
||||
)
|
||||
db.flush()
|
||||
audit.record(
|
||||
db,
|
||||
user.id,
|
||||
AuditAction.API_TOKEN_CREATE,
|
||||
target_type="api_token",
|
||||
target_id=token.id,
|
||||
after={
|
||||
"grants": {str(f): sorted(r) for f, r in wanted.items()},
|
||||
"expires_at": iso(token.expires_at),
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
return {"ok": True, **_serialize(db, token), "token": raw}
|
||||
|
||||
|
||||
@router.delete("/{token_id}")
|
||||
def revoke_token(
|
||||
token_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
|
||||
) -> dict:
|
||||
"""Revoke a token: it stops working at its next request (its grants go with it)."""
|
||||
token = owned_or_404(db, ApiToken, token_id, user, what="api_token")
|
||||
db.delete(token)
|
||||
audit.record(
|
||||
db, user.id, AuditAction.API_TOKEN_REVOKE, target_type="api_token", target_id=token_id
|
||||
)
|
||||
db.commit()
|
||||
return {"ok": True, "deleted": token_id}
|
||||
@@ -0,0 +1,37 @@
|
||||
"""The words a background run's row is written with — the `job_runs` table's status and trigger.
|
||||
|
||||
One source for the literals: the scheduler writes them, the startup sweep rewrites `running`, the
|
||||
model's CHECK constraints are built from these sets, and the dashboard reads them back (the frontend's
|
||||
copies are held to them by `backendIds.contract.test.ts`). The migrated database's CHECKs are frozen
|
||||
in its migration: a test holds them equal to these sets, so a new word fails until a migration adds
|
||||
it (`test_runstatus.py`). The download pipeline's `cancelled`/`canceled` drift (`downloads/status.py`)
|
||||
is why a status is never typed as a bare string.
|
||||
"""
|
||||
|
||||
from enum import StrEnum
|
||||
|
||||
|
||||
class RunStatus(StrEnum):
|
||||
RUNNING = "running"
|
||||
OK = "ok"
|
||||
ERROR = "error"
|
||||
# Sync was paused: the run was due and deliberately did nothing.
|
||||
SKIPPED = "skipped"
|
||||
# The process stopped while the run was going (a deploy, a crash): set at the next startup, so a
|
||||
# row never claims to be running in a process that no longer exists.
|
||||
INTERRUPTED = "interrupted"
|
||||
|
||||
|
||||
class RunTrigger(StrEnum):
|
||||
# The run's interval came round.
|
||||
SCHEDULED = "scheduled"
|
||||
# An admin pressed "run now" (or "run all").
|
||||
MANUAL = "manual"
|
||||
# A user's own action started work on one target (a channel's full history), outside the
|
||||
# recurring jobs.
|
||||
AD_HOC = "ad_hoc"
|
||||
|
||||
|
||||
def sql_in(values: type[StrEnum]) -> str:
|
||||
"""`'a', 'b'` — the IN-list a CHECK constraint names these values with."""
|
||||
return ", ".join(f"'{v.value}'" for v in values)
|
||||
+348
-191
@@ -3,28 +3,31 @@ recent-first / deep backfill. Runs inside the API process (single worker)."""
|
||||
|
||||
import contextvars
|
||||
import threading
|
||||
from datetime import datetime, timezone
|
||||
from datetime import timezone
|
||||
from typing import Callable
|
||||
|
||||
from apscheduler.executors.pool import ThreadPoolExecutor
|
||||
from apscheduler.jobstores.base import ConflictingIdError
|
||||
from apscheduler.schedulers.background import BackgroundScheduler
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import audit, debugsession, logs, logsink, progress, quota
|
||||
from app.audit import AuditAction
|
||||
from app import audit, debugsession, jobruns, logs, logsink, progress, quota, sysconfig
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.downloads.gc import run_download_gc
|
||||
from app.models import SchedulerSetting
|
||||
from app.models import Channel, JobRun
|
||||
from app.plex.sync import sync as run_plex_sync
|
||||
from app.plex.tmdb import run_enrich as run_tmdb_enrich
|
||||
from app.plex.tmdb import run_people_enrich
|
||||
from app.plex.watch_sync import run_plex_watch_reconcile, run_plex_watch_sync
|
||||
from app.notifications import create_notification
|
||||
from app.runstatus import RunStatus, RunTrigger
|
||||
from app.state import is_sync_paused
|
||||
from app.sync.autotag import run_autotag_all
|
||||
from app.sync.explore import purge_ephemeral
|
||||
from app.sync.maintenance import run_maintenance
|
||||
from app.sync.playlists import sync_all_playlists
|
||||
from app.utils import iso
|
||||
from app.sync.runner import (
|
||||
run_deep_backfill,
|
||||
run_enrich,
|
||||
@@ -37,12 +40,31 @@ from app.sync.runner import (
|
||||
log = logs.get_logger(__name__)
|
||||
_scheduler: BackgroundScheduler | None = None
|
||||
|
||||
# Per-job run activity, kept in-memory for the admin Scheduler dashboard to read (the
|
||||
# scheduler lives in the same single-worker process as the API, so a request can read it
|
||||
# directly). Ephemeral by design — it reflects "what the scheduler is doing right now and
|
||||
# how the last runs went" since this process started; durable history isn't the goal here.
|
||||
# What is running RIGHT NOW, per job: the single-run claim and the live progress. In memory on
|
||||
# purpose — the scheduler lives in the API's one process, and progress changes every second. What a
|
||||
# run did is durable instead: its `job_runs` row (app/jobruns.py), which outlives a restart.
|
||||
_activity: dict[str, dict] = {}
|
||||
_activity_lock = threading.Lock()
|
||||
# The runs this process has open (run id -> job), under the same lock. A row can stay `running` in
|
||||
# the table after its outcome failed to write (_close), so the table cannot say what is going; this
|
||||
# can, as the one process per database (app/jobruns.py) is the one running them.
|
||||
_open_runs: dict[int, str] = {}
|
||||
|
||||
|
||||
def _opened(run_id: int, job: str) -> None:
|
||||
with _activity_lock:
|
||||
_open_runs[run_id] = job
|
||||
|
||||
|
||||
def _closed(run_id: int) -> None:
|
||||
with _activity_lock:
|
||||
_open_runs.pop(run_id, None)
|
||||
|
||||
|
||||
def _open_run_ids() -> dict[int, str]:
|
||||
with _activity_lock:
|
||||
return dict(_open_runs)
|
||||
|
||||
|
||||
# Set (per-thread) by a manual "run now" trigger to the id of the admin who clicked, so that
|
||||
# run — and only that run, not the recurring scheduled ones — posts a completion notification
|
||||
@@ -51,59 +73,36 @@ _manual_actor: contextvars.ContextVar[int | None] = contextvars.ContextVar(
|
||||
"manual_actor", default=None
|
||||
)
|
||||
|
||||
# Each interval job's env/config default period (minutes). The admin can override any of
|
||||
# these at runtime (stored in scheduler_settings, applied live); see load_intervals.
|
||||
# Each interval job's env/config default period (minutes). The admin can override any of these at
|
||||
# runtime from the Scheduler page — a sysconfig key per job (`sysconfig.interval_key`), applied
|
||||
# live; see load_intervals.
|
||||
JOB_INTERVALS: dict[str, int] = {
|
||||
"rss_poll": settings.rss_poll_minutes,
|
||||
"enrich": settings.enrich_interval_minutes,
|
||||
"backfill": settings.backfill_interval_minutes,
|
||||
"autotag": settings.autotag_interval_minutes,
|
||||
"shorts": settings.shorts_probe_interval_minutes,
|
||||
"subscriptions": settings.subscriptions_resync_minutes,
|
||||
"playlist_sync": settings.playlist_sync_minutes,
|
||||
"maintenance": settings.maintenance_interval_minutes,
|
||||
"explore_cleanup": settings.explore_cleanup_minutes,
|
||||
"download_gc": settings.download_gc_minutes,
|
||||
"plex_sync": settings.plex_sync_interval_min,
|
||||
"plex_watch_sync": settings.plex_watch_sync_interval_min,
|
||||
"plex_watch_reconcile": settings.plex_watch_reconcile_interval_min,
|
||||
"tmdb_enrich": settings.tmdb_enrich_interval_min,
|
||||
"people_enrich": settings.people_enrich_interval_min,
|
||||
"audit_gc": settings.audit_gc_minutes,
|
||||
"log_gc": settings.log_gc_minutes,
|
||||
job: getattr(settings, attr) for job, attr in sysconfig.JOB_INTERVAL_DEFAULTS.items()
|
||||
}
|
||||
|
||||
# Sane bounds for an admin-set interval (minutes).
|
||||
MIN_INTERVAL = 1
|
||||
MAX_INTERVAL = 1440 # one day
|
||||
MIN_INTERVAL = sysconfig.MIN_INTERVAL
|
||||
MAX_INTERVAL = sysconfig.MAX_INTERVAL
|
||||
|
||||
|
||||
def load_intervals(db) -> dict[str, int]:
|
||||
"""Effective per-job intervals: the env/config defaults overlaid with any admin overrides
|
||||
saved in scheduler_settings."""
|
||||
iv = dict(JOB_INTERVALS)
|
||||
for row in db.execute(select(SchedulerSetting)).scalars():
|
||||
if row.job_id in iv and row.interval_minutes:
|
||||
iv[row.job_id] = row.interval_minutes
|
||||
return iv
|
||||
"""Effective per-job intervals: each job's sysconfig override, else its env/config default."""
|
||||
return {job: sysconfig.get_int(db, sysconfig.interval_key(job)) for job in JOB_INTERVALS}
|
||||
|
||||
|
||||
def apply_interval(db, job_id: str, minutes: int) -> int:
|
||||
"""Persist a job's interval override and reschedule the live job immediately (if the
|
||||
scheduler runs in this process). Returns the applied value."""
|
||||
def stage_interval(db, job_id: str, minutes: int) -> int:
|
||||
"""Stage a job's interval override (clamped) in `db`; returns the value. The caller commits it
|
||||
with its audit row, then calls `reschedule` — the live job follows only what was stored."""
|
||||
if job_id not in JOB_INTERVALS:
|
||||
raise KeyError(job_id)
|
||||
minutes = max(MIN_INTERVAL, min(MAX_INTERVAL, int(minutes)))
|
||||
row = db.get(SchedulerSetting, job_id)
|
||||
if row is None:
|
||||
db.add(SchedulerSetting(job_id=job_id, interval_minutes=minutes))
|
||||
else:
|
||||
row.interval_minutes = minutes
|
||||
db.commit()
|
||||
sysconfig.set_value(db, sysconfig.interval_key(job_id), minutes)
|
||||
return minutes
|
||||
|
||||
|
||||
def reschedule(job_id: str, minutes: int) -> None:
|
||||
"""Move the live interval job to `minutes` (if this process registered it)."""
|
||||
if _scheduler is not None and _scheduler.get_job(job_id) is not None:
|
||||
_scheduler.reschedule_job(job_id, trigger="interval", minutes=minutes)
|
||||
log.info("rescheduled job %s to every %s min", job_id, minutes)
|
||||
return minutes
|
||||
|
||||
|
||||
def _record(name: str, **fields) -> None:
|
||||
@@ -112,7 +111,12 @@ def _record(name: str, **fields) -> None:
|
||||
|
||||
|
||||
def _notify_done(
|
||||
db, actor_id: int, job_id: str, status: str, summary: str | None, result: dict | None = None
|
||||
db,
|
||||
actor_id: int,
|
||||
job_id: str,
|
||||
status: RunStatus,
|
||||
summary: str | None,
|
||||
result: dict | None = None,
|
||||
) -> None:
|
||||
"""Post a completion notice to the triggering admin's inbox (manual runs only). Best
|
||||
effort — a notification failure must never mask the job's own outcome."""
|
||||
@@ -130,49 +134,6 @@ def _notify_done(
|
||||
log.exception("failed to write completion notification for job %s", job_id)
|
||||
|
||||
|
||||
def _run_changed(result) -> bool:
|
||||
"""Did an automatic job run actually do something worth an audit row? (No-op polls don't.)
|
||||
A "change" = a truthy NUMERIC count; status-string dicts like {"skipped": "disabled"} (Plex
|
||||
off — the default) are no-ops, not changes, so they don't flood the trail every interval."""
|
||||
if result is None:
|
||||
return False
|
||||
if isinstance(result, dict):
|
||||
# A phased job (backfill's recent/deep, people_enrich's bios/portraits) nests a result per
|
||||
# phase: a count inside a phase is a change too.
|
||||
return any(
|
||||
_run_changed(v) if isinstance(v, dict) else isinstance(v, (int, float)) and bool(v)
|
||||
for v in result.values()
|
||||
)
|
||||
if isinstance(result, (int, float)):
|
||||
return result != 0
|
||||
return bool(result)
|
||||
|
||||
|
||||
def _audit_run(db, actor_id, job_id, status, summary, result) -> None:
|
||||
"""Write ONE run-summary audit row per job run (never per-item). Manual runs (an admin
|
||||
clicked "run now" → actor_id set) and errors are always logged; automatic runs only when
|
||||
they changed something, so routine no-op polls don't spam the trail. Best-effort."""
|
||||
if actor_id is None and status == "ok" and not _run_changed(result):
|
||||
return
|
||||
# The result as data too: the audit log names it in the reader's language, as the inbox
|
||||
# does; `summary` stays the English line of the row's own summary column.
|
||||
data = _result_data(result)
|
||||
try:
|
||||
audit.record(
|
||||
db,
|
||||
actor_id,
|
||||
AuditAction.JOB_RUN,
|
||||
target_type="job",
|
||||
target_id=job_id,
|
||||
summary=f"{job_id}: {status}" + (f" — {summary}" if summary else ""),
|
||||
after={"status": status, "summary": summary, **({"result": data} if data else {})},
|
||||
)
|
||||
db.commit()
|
||||
except Exception:
|
||||
db.rollback()
|
||||
log.exception("failed to write audit row for job %s", job_id)
|
||||
|
||||
|
||||
def _job(name: str, fn) -> None:
|
||||
# The one place both run paths converge (manual trigger AND scheduled fire), so claim the run
|
||||
# HERE, atomically: whichever reaches this first wins, the other skips. This is the real
|
||||
@@ -180,71 +141,88 @@ def _job(name: str, fn) -> None:
|
||||
if not _claim_running(name):
|
||||
log.info("job %s already running — skipping this trigger", name)
|
||||
return
|
||||
db = SessionLocal()
|
||||
actor_id = _manual_actor.get() # set only for a manual "run now" trigger
|
||||
_record(
|
||||
name, last_started=datetime.now(timezone.utc).isoformat(), last_error=None, progress=None
|
||||
) # running already claimed
|
||||
trigger = RunTrigger.SCHEDULED if actor_id is None else RunTrigger.MANUAL
|
||||
_record(name, progress=None) # running already claimed
|
||||
db = SessionLocal()
|
||||
run_id = None
|
||||
try:
|
||||
try:
|
||||
run_id = jobruns.start(name, trigger, actor_id)
|
||||
except Exception:
|
||||
# No row to tell it in: the database is what failed. The log is the record.
|
||||
log.exception("job %s not started: its run could not be recorded", name)
|
||||
return
|
||||
_opened(run_id, name)
|
||||
with logs.bind(run_id=run_id):
|
||||
_run(db, name, fn, run_id, actor_id)
|
||||
finally:
|
||||
# ALWAYS release the claim, even if the body raised before its own status update — so a
|
||||
# failure can never wedge the job as permanently "running".
|
||||
if run_id is not None:
|
||||
_closed(run_id)
|
||||
_record(name, running=False, progress=None)
|
||||
db.close()
|
||||
|
||||
|
||||
def _run(db, name: str, fn, run_id: int, actor_id: int | None) -> None:
|
||||
"""The run itself, inside its `job_runs` row: the pause-skip, the work, and its outcome — on
|
||||
the row, and in the triggering admin's inbox for a manual run."""
|
||||
|
||||
def sink(current, total, phase):
|
||||
_record(name, progress={"current": current, "total": total, "phase": phase})
|
||||
|
||||
# Only the work is inside the try: writing its outcome is not the job, and a write that fails
|
||||
# must not turn a run that succeeded into a failed one (see _close).
|
||||
result = None
|
||||
try:
|
||||
if is_sync_paused(db):
|
||||
log.info("job %s skipped (sync paused)", name)
|
||||
_record(
|
||||
name,
|
||||
status="skipped",
|
||||
last_finished=datetime.now(timezone.utc).isoformat(),
|
||||
last_result="paused",
|
||||
result=_PAUSED,
|
||||
progress=None,
|
||||
)
|
||||
if actor_id is not None:
|
||||
_notify_done(db, actor_id, name, "skipped", "sync paused", _PAUSED)
|
||||
_audit_run(db, actor_id, name, "skipped", "sync paused", _PAUSED)
|
||||
return
|
||||
with progress.bind(sink):
|
||||
result = fn(db)
|
||||
summary = _summarize(result)
|
||||
data = _result_data(result)
|
||||
log.info("job %s -> %s", name, result)
|
||||
_record(
|
||||
name,
|
||||
status="ok",
|
||||
last_finished=datetime.now(timezone.utc).isoformat(),
|
||||
last_result=summary,
|
||||
result=data,
|
||||
progress=None,
|
||||
)
|
||||
if actor_id is not None:
|
||||
_notify_done(db, actor_id, name, "ok", summary, data)
|
||||
_audit_run(db, actor_id, name, "ok", summary, result)
|
||||
paused = is_sync_paused(db)
|
||||
if not paused:
|
||||
with progress.bind(sink):
|
||||
result = fn(db)
|
||||
except Exception as exc:
|
||||
db.rollback()
|
||||
log.exception("job %s failed", name, extra=logs.event("scheduler.job.failed", job=name))
|
||||
err = str(exc) or exc.__class__.__name__
|
||||
# A failure the job named (progress.JobFailed) rides along as data, as a result does, so the
|
||||
# client says it in the reader's language; `err` stays the English line.
|
||||
failure = (
|
||||
{"failure": exc.reason, **exc.params} if isinstance(exc, progress.JobFailed) else None
|
||||
)
|
||||
_record(
|
||||
name,
|
||||
status="error",
|
||||
last_finished=datetime.now(timezone.utc).isoformat(),
|
||||
last_error=err,
|
||||
result=failure,
|
||||
progress=None,
|
||||
)
|
||||
err, failure = _failure(exc)
|
||||
_close(run_id, RunStatus.ERROR, error=err, failure=failure)
|
||||
if actor_id is not None:
|
||||
_notify_done(db, actor_id, name, "error", err, failure)
|
||||
_audit_run(db, actor_id, name, "error", err, failure)
|
||||
finally:
|
||||
# ALWAYS release the claim, even if the body raised before its own status update — so a
|
||||
# failure can never wedge the job as permanently "running".
|
||||
_record(name, running=False)
|
||||
db.close()
|
||||
_notify_done(db, actor_id, name, RunStatus.ERROR, err, failure)
|
||||
return
|
||||
if paused:
|
||||
log.info("job %s skipped (sync paused)", name)
|
||||
_close(run_id, RunStatus.SKIPPED, summary="sync paused", result=_PAUSED)
|
||||
if actor_id is not None:
|
||||
_notify_done(db, actor_id, name, RunStatus.SKIPPED, "sync paused", _PAUSED)
|
||||
return
|
||||
summary, data = _finish_ok(name, run_id, result)
|
||||
if actor_id is not None:
|
||||
_notify_done(db, actor_id, name, RunStatus.OK, summary, data)
|
||||
|
||||
|
||||
def _close(run_id: int, status: RunStatus, **outcome) -> None:
|
||||
"""Write a run's outcome. Its own failure (the database) is the log's to tell, never the job's:
|
||||
the row then stays `running` until the next start's sweep closes it as interrupted."""
|
||||
try:
|
||||
jobruns.finish(run_id, status, **outcome)
|
||||
except Exception:
|
||||
log.exception("run %s: its outcome (%s) could not be recorded", run_id, status)
|
||||
|
||||
|
||||
def _finish_ok(name: str, run_id: int, result) -> tuple[str | None, dict | None]:
|
||||
"""Close a run that returned: its result on the row, as data and as the English one-liner."""
|
||||
summary = _summarize(result)
|
||||
data = _result_data(result)
|
||||
log.info("job %s -> %s", name, result)
|
||||
_close(run_id, RunStatus.OK, summary=summary, result=data)
|
||||
return summary, data
|
||||
|
||||
|
||||
def _failure(exc: Exception) -> tuple[str, dict | None]:
|
||||
"""A raised run's English error line, and the failure the job named (progress.JobFailed) as
|
||||
data, as a result is, so the client says it in the reader's language."""
|
||||
err = str(exc) or exc.__class__.__name__
|
||||
failure = {"failure": exc.reason, **exc.params} if isinstance(exc, progress.JobFailed) else None
|
||||
return err, failure
|
||||
|
||||
|
||||
# A skipped run's result, in the same shape as a job's own `{"skipped": …}`.
|
||||
@@ -271,11 +249,13 @@ def _summarize(result) -> str | None:
|
||||
return str(result)
|
||||
|
||||
|
||||
def scheduler_snapshot() -> dict:
|
||||
"""What the scheduler is doing, for the admin dashboard. `running_here` is False in
|
||||
environments where the scheduler is disabled (e.g. local dev shares the prod-ish DB but
|
||||
runs no scheduler), so the UI can say so while still showing DB-derived queue/quota."""
|
||||
running_here = _scheduler is not None
|
||||
def scheduler_snapshot(db) -> dict:
|
||||
"""What the scheduler is doing, for the admin dashboard: each job's live state from this process
|
||||
and its last runs from `job_runs`, so a restart no longer blanks "when did it last run, and how
|
||||
did it go"; and the same for the ad-hoc runs users start on one target. `running_here` is False where the scheduler runs no interval jobs
|
||||
(SCHEDULER_ENABLED=false), so the UI can say so while still showing the DB-derived history,
|
||||
queue and quota."""
|
||||
running_here = _scheduler is not None and settings.scheduler_enabled
|
||||
next_runs: dict[str, str | None] = {}
|
||||
live_intervals: dict[str, int] = {}
|
||||
if _scheduler is not None:
|
||||
@@ -287,6 +267,12 @@ def scheduler_snapshot() -> dict:
|
||||
live_intervals[job.id] = round(iv.total_seconds() / 60)
|
||||
with _activity_lock:
|
||||
acts = {k: dict(v) for k, v in _activity.items()}
|
||||
# Open before the read or after it: a run that started or ended while the rows were read is
|
||||
# still counted as the open run it was, never as a stray one.
|
||||
open_runs = _open_run_ids()
|
||||
recent = jobruns.recent_runs(db, [*JOB_INTERVALS, *AD_HOC_JOBS], RECENT_RUNS)
|
||||
open_runs |= _open_run_ids()
|
||||
going_id = max((r for r, j in open_runs.items() if j == CHANNEL_BACKFILL), default=None)
|
||||
jobs = []
|
||||
for job_id, default_interval in JOB_INTERVALS.items():
|
||||
a = acts.get(job_id, {})
|
||||
@@ -296,16 +282,94 @@ def scheduler_snapshot() -> dict:
|
||||
"interval_minutes": live_intervals.get(job_id, default_interval),
|
||||
"next_run": next_runs.get(job_id),
|
||||
"running": a.get("running", False),
|
||||
"status": a.get("status"),
|
||||
"last_started": a.get("last_started"),
|
||||
"last_finished": a.get("last_finished"),
|
||||
"last_result": a.get("last_result"),
|
||||
"result": a.get("result"),
|
||||
"last_error": a.get("last_error"),
|
||||
"progress": a.get("progress"),
|
||||
**_last_runs(recent.get(job_id, []), open_runs),
|
||||
}
|
||||
)
|
||||
return {"running_here": running_here, "enabled": settings.scheduler_enabled, "jobs": jobs}
|
||||
# Work a user starts on one target: no interval, no "run now" (it needs its target), but its
|
||||
# runs are shown beside the jobs' — it spends the same quota. Its one in-memory guard is the
|
||||
# target's own (sync/runner), so "running" is the runs this process has open: two channels can
|
||||
# backfill at once, so a run still going counts even when a newer one already ended. That run then
|
||||
# leads the row, so its channel, its "no outcome yet" and its dot are one run's, never the
|
||||
# newer run's outcome under the older one's channel.
|
||||
backfills = recent.get(CHANNEL_BACKFILL, [])
|
||||
going = db.get(JobRun, going_id) if going_id is not None else None
|
||||
if going is not None and going.status != RunStatus.RUNNING:
|
||||
going = None # its outcome is written; only the bookkeeping has not caught up
|
||||
if going is not None:
|
||||
backfills = [going, *(r for r in backfills if r.id != going.id)][:RECENT_RUNS]
|
||||
shown = backfills[0] if backfills else None
|
||||
ad_hoc = [
|
||||
{
|
||||
"id": CHANNEL_BACKFILL,
|
||||
"running": going is not None,
|
||||
"target": shown.target if shown else None,
|
||||
"target_title": (
|
||||
db.scalar(select(Channel.title).where(Channel.id == shown.target))
|
||||
if shown is not None and shown.target
|
||||
else None
|
||||
),
|
||||
**_last_runs(backfills, open_runs),
|
||||
}
|
||||
]
|
||||
return {
|
||||
"running_here": running_here,
|
||||
"enabled": settings.scheduler_enabled,
|
||||
"jobs": jobs,
|
||||
"ad_hoc": ad_hoc,
|
||||
}
|
||||
|
||||
|
||||
# How many past runs a Scheduler row shows as dots; the full history is the run list.
|
||||
RECENT_RUNS = 3
|
||||
|
||||
|
||||
def _seen(run, open_runs: dict[int, str]) -> RunStatus:
|
||||
"""A run's status as the page tells it. A row `running` that no run here has open is one whose
|
||||
outcome failed to write: the next start's sweep closes it as interrupted, so it reads so now —
|
||||
not as a run still going beside a row that says nothing runs."""
|
||||
if run.status == RunStatus.RUNNING and run.id not in open_runs:
|
||||
return RunStatus.INTERRUPTED
|
||||
return RunStatus(run.status)
|
||||
|
||||
|
||||
def _last_runs(runs: list, open_runs: dict[int, str]) -> dict:
|
||||
"""A row's run fields from its newest runs (newest first): how the last one went, and the few
|
||||
before it for the history dots."""
|
||||
run = runs[0] if runs else None
|
||||
status = _seen(run, open_runs) if run is not None else None
|
||||
# A run still going has no outcome yet: only an ended run says how it went.
|
||||
ended = run if status is not None and status != RunStatus.RUNNING else None
|
||||
return {
|
||||
"status": status if ended else None,
|
||||
"last_started": iso(run.started_at) if run else None,
|
||||
# A stray row has no end of its own: it ended, as far as anyone can tell, where it began.
|
||||
"last_finished": iso(ended.finished_at or ended.started_at) if ended else None,
|
||||
"last_result": ended.summary if ended else None,
|
||||
# The outcome as data: the result, or on a failed run the failure the job named.
|
||||
"result": _outcome_data(ended),
|
||||
"last_error": ended.error if ended else None,
|
||||
"last_trigger": run.trigger if run else None,
|
||||
"last_run_id": run.id if run else None,
|
||||
"quota_units": ended.quota_units if ended else None,
|
||||
"recent": [_run_brief(r, open_runs) for r in runs],
|
||||
}
|
||||
|
||||
|
||||
def _run_brief(run, open_runs: dict[int, str]) -> dict:
|
||||
return {
|
||||
"id": run.id,
|
||||
"status": _seen(run, open_runs),
|
||||
"trigger": run.trigger,
|
||||
"started_at": iso(run.started_at),
|
||||
"finished_at": iso(run.finished_at),
|
||||
}
|
||||
|
||||
|
||||
def _outcome_data(run) -> dict | None:
|
||||
if run is None:
|
||||
return None
|
||||
return run.failure if run.status == RunStatus.ERROR else run.result
|
||||
|
||||
|
||||
def _rss_job() -> None:
|
||||
@@ -370,7 +434,8 @@ def _explore_cleanup_job() -> None:
|
||||
|
||||
|
||||
def _download_gc_job() -> None:
|
||||
# Retention GC for the download center: pre-expiry warnings, TTL deletion, LRU eviction.
|
||||
# Retention GC for the download center: pre-expiry warnings, TTL deletion, LRU eviction — and
|
||||
# the week-ahead warning before an API token runs out (`apitokens.warn_expiring`).
|
||||
# Pure disk/DB work (no YouTube quota).
|
||||
_job("download_gc", run_download_gc)
|
||||
|
||||
@@ -407,15 +472,16 @@ def _people_enrich_job() -> None:
|
||||
|
||||
|
||||
def _audit_gc_job() -> None:
|
||||
# The history cleanup: audit-log rows past audit_retention_days and YouTube quota history past
|
||||
# quota_retention_days (0 = keep forever, each). Pure DB, no quota.
|
||||
# The history cleanup: audit-log rows past audit_retention_days, YouTube quota history past
|
||||
# quota_retention_days and finished runs past job_run_retention_days (0 = keep forever, each).
|
||||
# Pure DB, no quota.
|
||||
def work(db):
|
||||
from app import sysconfig
|
||||
|
||||
reaped = audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))
|
||||
quota_reaped = quota.gc(db, sysconfig.get_int(db, "quota_retention_days"))
|
||||
runs_reaped = jobruns.gc(db, sysconfig.get_int(db, "job_run_retention_days"))
|
||||
db.commit()
|
||||
return {"reaped": reaped, "quota_reaped": quota_reaped}
|
||||
return {"reaped": reaped, "quota_reaped": quota_reaped, "runs_reaped": runs_reaped}
|
||||
|
||||
_job("audit_gc", work)
|
||||
|
||||
@@ -424,7 +490,6 @@ def _log_gc_job() -> None:
|
||||
# Prune log_events rows older than log_retention_days — the limit the privacy policy states, so
|
||||
# this runs hourly rather than daily. Pure DB, no quota.
|
||||
def work(db):
|
||||
from app import sysconfig
|
||||
|
||||
days = sysconfig.get_int(db, "log_retention_days")
|
||||
reaped = logsink.gc(db, days)
|
||||
@@ -484,15 +549,29 @@ def running_job_ids() -> set[str]:
|
||||
return {jid for jid, a in _activity.items() if a.get("running")}
|
||||
|
||||
|
||||
def trigger_job(job_id: str, actor_id: int | None = None) -> str:
|
||||
"""Run a job immediately in a background thread, independent of its interval schedule.
|
||||
The wrapper handles its own DB session, activity tracking and pause-skip, so the live
|
||||
dashboard reflects the run. Returns "started", "already_running", or raises KeyError for
|
||||
an unknown job. Refusing a concurrent run keeps a manual trigger from overlapping a
|
||||
scheduled run (APScheduler enforces max_instances=1 for the scheduled side).
|
||||
def _as_actor(fn: Callable[[], None], actor_id: int | None) -> Callable[[], None]:
|
||||
"""`fn` run as `actor_id`'s manual run. The actor is RESET afterwards: the executor's pool
|
||||
threads are reused and keep their context from task to task, so an actor left set would make
|
||||
the next scheduled run that thread picks up look like this admin's manual run."""
|
||||
|
||||
`actor_id` (the admin who clicked "run now") is stashed in a contextvar inside the new
|
||||
thread so the run posts a completion notification to that admin's inbox."""
|
||||
def run() -> None:
|
||||
token = _manual_actor.set(actor_id)
|
||||
try:
|
||||
fn()
|
||||
finally:
|
||||
_manual_actor.reset(token)
|
||||
|
||||
return run
|
||||
|
||||
|
||||
def trigger_job(job_id: str, actor_id: int | None = None) -> str:
|
||||
"""Run a job now, independent of its interval schedule: a one-shot job on the same scheduler,
|
||||
so a manual run shares the scheduled runs' executor and shutdown — no thread of its own. The
|
||||
wrapper handles its own DB session, run row and pause-skip. Returns "started",
|
||||
"already_running", or raises KeyError for an unknown job.
|
||||
|
||||
`actor_id` (the admin who clicked "run now") is set in a contextvar inside the run's thread, so
|
||||
the run is recorded as theirs and posts a completion notification to their inbox."""
|
||||
fn = JOB_FUNCS.get(job_id)
|
||||
if fn is None:
|
||||
raise KeyError(job_id)
|
||||
@@ -501,13 +580,18 @@ def trigger_job(job_id: str, actor_id: int | None = None) -> str:
|
||||
# _job claims, sees it's taken, and skips — no double run.
|
||||
if _is_running(job_id):
|
||||
return "already_running"
|
||||
sched = _scheduler
|
||||
if sched is None:
|
||||
# Only between shutdown and exit: start_scheduler runs at every API startup.
|
||||
raise RuntimeError("the scheduler is not running")
|
||||
|
||||
def run() -> None:
|
||||
if actor_id is not None:
|
||||
_manual_actor.set(actor_id)
|
||||
fn()
|
||||
|
||||
threading.Thread(target=run, name=f"trigger-{job_id}", daemon=True).start()
|
||||
try:
|
||||
# One id per job: a second click before the first one-shot has even started is the same
|
||||
# run, not a queued second one. misfire_grace_time=None: a busy pool delays it, never drops it.
|
||||
sched.add_job(_as_actor(fn, actor_id), "date", id=f"{job_id}@now", misfire_grace_time=None)
|
||||
# silent-ok: not a failure — the answer is the 409 the admin sees ("already running").
|
||||
except ConflictingIdError:
|
||||
return "already_running"
|
||||
return "started"
|
||||
|
||||
|
||||
@@ -520,22 +604,95 @@ def trigger_all(actor_id: int | None = None) -> list[str]:
|
||||
return started
|
||||
|
||||
|
||||
def start_scheduler() -> None:
|
||||
global _scheduler
|
||||
if not settings.scheduler_enabled or _scheduler is not None:
|
||||
return
|
||||
# Effective intervals = env defaults overlaid with any admin overrides from the DB.
|
||||
db = SessionLocal()
|
||||
# Work a user starts on one target outside the job list — today the "get full history" backfill of
|
||||
# one channel. It runs on the same scheduler as a recorded run (trigger `ad_hoc`), so what it did and
|
||||
# spent is in `job_runs` like any job's, not a background task only the log ever heard of.
|
||||
CHANNEL_BACKFILL = "channel_backfill"
|
||||
# Every ad-hoc job id: the page names each one (a frontend contract test reads this tuple).
|
||||
AD_HOC_JOBS = (CHANNEL_BACKFILL,)
|
||||
|
||||
|
||||
def run_ad_hoc(job: str, fn: Callable[[], dict], *, actor_id: int, target: str) -> None:
|
||||
"""Run `fn` now as `actor_id`'s run of `job` on `target`, a one-shot job on the scheduler. Its
|
||||
own exclusion (if any) lives in `fn`, which says so in its result; one id per job and target, so
|
||||
a second request before the first has even started is that same run."""
|
||||
if job not in AD_HOC_JOBS:
|
||||
raise KeyError(job) # an unnamed job would show on the page as its bare id
|
||||
sched = _scheduler
|
||||
if sched is None:
|
||||
# Only between shutdown and exit: start_scheduler runs at every API startup.
|
||||
raise RuntimeError("the scheduler is not running")
|
||||
try:
|
||||
iv = load_intervals(db)
|
||||
sched.add_job(
|
||||
_ad_hoc,
|
||||
"date",
|
||||
args=(job, fn, actor_id, target),
|
||||
id=f"{job}@{target}",
|
||||
misfire_grace_time=None,
|
||||
)
|
||||
# silent-ok: not a failure — the run for this target is already queued, and it does this work.
|
||||
except ConflictingIdError:
|
||||
log.info("ad-hoc %s for %s already queued", job, target)
|
||||
|
||||
|
||||
def _ad_hoc(job: str, fn: Callable[[], dict], actor_id: int, target: str) -> None:
|
||||
try:
|
||||
run_id = jobruns.start(job, RunTrigger.AD_HOC, actor_id, target)
|
||||
except Exception:
|
||||
# No row to tell it in: the database is what failed. The log is the record.
|
||||
log.exception("ad-hoc %s for %s not started: its run could not be recorded", job, target)
|
||||
return
|
||||
_opened(run_id, job)
|
||||
try:
|
||||
_ad_hoc_work(job, fn, run_id)
|
||||
finally:
|
||||
db.close()
|
||||
scheduler = BackgroundScheduler(timezone="UTC")
|
||||
for job_id, fn in JOB_FUNCS.items():
|
||||
scheduler.add_job(fn, "interval", minutes=iv[job_id], id=job_id)
|
||||
_closed(run_id)
|
||||
|
||||
|
||||
def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int) -> None:
|
||||
with logs.bind(run_id=run_id):
|
||||
# As in _run: only the work is inside the try.
|
||||
try:
|
||||
result = fn()
|
||||
except Exception as exc:
|
||||
log.exception("job %s failed", job, extra=logs.event("scheduler.job.failed", job=job))
|
||||
err, failure = _failure(exc)
|
||||
_close(run_id, RunStatus.ERROR, error=err, failure=failure)
|
||||
return
|
||||
_finish_ok(job, run_id, result)
|
||||
|
||||
|
||||
# Room for every job at once (a "run all") beside a few ad-hoc runs: APScheduler's default pool of
|
||||
# 10 would queue a scheduled fire behind long manual runs.
|
||||
_POOL_SIZE = len(JOB_FUNCS) + 8
|
||||
|
||||
|
||||
def start_scheduler() -> None:
|
||||
"""Start this process's scheduler. It always starts, so "run now" works on every install; only
|
||||
with SCHEDULER_ENABLED does it also register the recurring interval jobs. Before anything can
|
||||
run, every run a previous process left open is closed as interrupted — one API process serves a
|
||||
database (app/jobruns.py), so an open run here is a dead process's."""
|
||||
global _scheduler
|
||||
if _scheduler is not None:
|
||||
return
|
||||
left = jobruns.interrupt_leftovers()
|
||||
if left:
|
||||
log.warning("%s run(s) left running by the previous process marked interrupted", left)
|
||||
scheduler = BackgroundScheduler(
|
||||
timezone="UTC", executors={"default": ThreadPoolExecutor(_POOL_SIZE)}
|
||||
)
|
||||
if settings.scheduler_enabled:
|
||||
# Effective intervals = env defaults overlaid with any admin overrides from the DB.
|
||||
db = SessionLocal()
|
||||
try:
|
||||
iv = load_intervals(db)
|
||||
finally:
|
||||
db.close()
|
||||
for job_id, fn in JOB_FUNCS.items():
|
||||
scheduler.add_job(fn, "interval", minutes=iv[job_id], id=job_id)
|
||||
scheduler.start()
|
||||
_scheduler = scheduler
|
||||
log.info("scheduler started")
|
||||
log.info("scheduler started (interval jobs %s)", "on" if settings.scheduler_enabled else "off")
|
||||
|
||||
|
||||
def shutdown_scheduler() -> None:
|
||||
|
||||
@@ -46,6 +46,12 @@ _fernet: Fernet | None = (
|
||||
)
|
||||
|
||||
|
||||
def can_encrypt() -> bool:
|
||||
"""Whether a value can be stored encrypted here: TOKEN_ENCRYPTION_KEY is set. Production
|
||||
refuses to start without it (`config`); a local box may run without one."""
|
||||
return _fernet is not None
|
||||
|
||||
|
||||
def _require_fernet() -> Fernet:
|
||||
if _fernet is None:
|
||||
raise RuntimeError("TOKEN_ENCRYPTION_KEY is not configured")
|
||||
|
||||
+22
-21
@@ -437,7 +437,7 @@ def run_deep_backfill(db: Session, max_channels: int = 10, max_pages: int = 5) -
|
||||
|
||||
|
||||
# Channels being deep-backfilled RIGHT NOW by EITHER path — the interactive backfill_channel_now (a
|
||||
# FastAPI BackgroundTasks threadpool thread) OR the scheduler's run_deep_backfill (its own thread).
|
||||
# one-shot run on the scheduler's pool, see scheduler.run_ad_hoc) OR the scheduler's run_deep_backfill.
|
||||
# BOTH register the channel they're paging (under the lock) so the other skips it: no two runs page
|
||||
# the same channel's `backfill_cursor` concurrently, whichever started first. `_immediate_running`
|
||||
# counts only the interactive path, capped so one user's burst can't drain the shared budget (the
|
||||
@@ -448,9 +448,9 @@ _backfill_lock = threading.Lock()
|
||||
_MAX_CONCURRENT_IMMEDIATE = 2
|
||||
|
||||
|
||||
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> None:
|
||||
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, in a
|
||||
background task with its OWN session, so the full catalogue lands in seconds instead of over the
|
||||
def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) -> dict:
|
||||
"""User-initiated "get full history": deep-backfill ONE channel to completion right now, as its
|
||||
own recorded run with its OWN session, so the full catalogue lands in seconds instead of over the
|
||||
paced scheduler's 10-minute ticks (max_pages=5/channel/tick). max_pages is high enough to finish
|
||||
any realistic channel in a single run (400 pages = 20 000 videos; e.g. BBC Earth's ~3400 = 69
|
||||
pages — the old 60-page cap stopped one page short of large channels, so their tail dribbled in
|
||||
@@ -459,34 +459,35 @@ def backfill_channel_now(channel_id: str, user_id: int, max_pages: int = 400) ->
|
||||
Quota-guarded, capped to _MAX_CONCURRENT_IMMEDIATE interactive runs so one user can't drain the
|
||||
shared backfill budget by opting many channels in at once (the overflow, and any channel too large
|
||||
to finish under the reserve, is finished by the scheduler since deep_requested stays set), and
|
||||
mutually excluded per-channel with the scheduler via _backfill_in_progress. Best-effort — logs and
|
||||
swallows failures."""
|
||||
mutually excluded per-channel with the scheduler via _backfill_in_progress.
|
||||
|
||||
Returns what the run did, as a job result: the counts, or `{"skipped": <why>}` when it left the
|
||||
channel to the scheduler. A failure raises — the run that wraps this records it."""
|
||||
global _immediate_running
|
||||
with _backfill_lock:
|
||||
if channel_id in _backfill_in_progress or _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
|
||||
return # already backfilling this channel, or too many interactive runs — leave it to the scheduler
|
||||
if channel_id in _backfill_in_progress:
|
||||
return {"skipped": "channel busy"}
|
||||
if _immediate_running >= _MAX_CONCURRENT_IMMEDIATE:
|
||||
return {"skipped": "at capacity"}
|
||||
_backfill_in_progress.add(channel_id)
|
||||
_immediate_running += 1
|
||||
try:
|
||||
with SessionLocal() as db:
|
||||
user = get_service_user(db)
|
||||
if user is None:
|
||||
return
|
||||
return {"skipped": "no service user"}
|
||||
channel = db.get(Channel, channel_id)
|
||||
if channel is None or channel.backfill_done or not channel.uploads_playlist_id:
|
||||
return
|
||||
return {"skipped": "nothing to do"}
|
||||
if quota.backfill_reserve_reached():
|
||||
return
|
||||
try:
|
||||
# Attribute the spend to the requesting user under the full-backfill action, so it
|
||||
# shows up correctly in the usage breakdown (not the default OTHER/system bucket).
|
||||
with quota.attribute(user_id, quota.QuotaAction.VIDEOS_BACKFILL_FULL):
|
||||
with YouTubeClient(db, user) as yt:
|
||||
backfill_channel_deep(db, yt, channel, max_pages)
|
||||
reconcile_full_history(db)
|
||||
except Exception:
|
||||
db.rollback()
|
||||
log.exception("Immediate backfill failed for channel %s", channel_id)
|
||||
return {"skipped": "low quota"}
|
||||
# Attribute the spend to the requesting user under the full-backfill action, so it
|
||||
# shows up correctly in the usage breakdown (not the default OTHER/system bucket).
|
||||
with quota.attribute(user_id, quota.QuotaAction.VIDEOS_BACKFILL_FULL):
|
||||
with YouTubeClient(db, user) as yt:
|
||||
added = backfill_channel_deep(db, yt, channel, max_pages)
|
||||
reconciled = reconcile_full_history(db)
|
||||
return {"videos_added": added, "reconciled_full": reconciled}
|
||||
finally:
|
||||
with _backfill_lock:
|
||||
_backfill_in_progress.discard(channel_id)
|
||||
|
||||
+251
-61
@@ -8,7 +8,8 @@ stored Fernet-encrypted at rest (requires TOKEN_ENCRYPTION_KEY; without it, secr
|
||||
stored in the DB and stay env-only).
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Literal
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -27,6 +28,16 @@ _decrypt_warned: set[str] = set()
|
||||
_coerce_warned: set[tuple[str, str]] = set()
|
||||
|
||||
|
||||
# When a changed value takes effect. "live": the next read obeys it — every consumer reads the key
|
||||
# where it uses it, never once at import (test-held: a read outside a function fails). A value read
|
||||
# once at startup is live only when its writer also applies it (the job intervals reschedule, the log
|
||||
# level is set as it is saved). "restart": the process has to start
|
||||
# again. No key needs a restart today, and the Configuration page has no way to say so yet, so a
|
||||
# "restart" key is refused until the page can (test-held).
|
||||
Applies = Literal["live", "restart"]
|
||||
LIVE: Applies = "live"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ConfigSpec:
|
||||
key: str
|
||||
@@ -37,6 +48,8 @@ class ConfigSpec:
|
||||
min: int | None = None
|
||||
max: int | None = None
|
||||
allow_empty: bool = False # str keys: a stored "" is an explicit empty value, not "unset"
|
||||
# Required, with no default: each key states it, so a new one cannot be live by omission.
|
||||
applies: Applies = field(kw_only=True)
|
||||
|
||||
@property
|
||||
def choices(self) -> tuple[str, ...]:
|
||||
@@ -45,30 +58,78 @@ class ConfigSpec:
|
||||
return choices_of(Settings.model_fields[self.default_attr].annotation)
|
||||
|
||||
|
||||
# Each interval job of the scheduler and the `settings` attribute holding its default period
|
||||
# (minutes). Its override is the `job_interval_<job>` key, set from the Scheduler page.
|
||||
JOB_INTERVAL_DEFAULTS: dict[str, str] = {
|
||||
"rss_poll": "rss_poll_minutes",
|
||||
"enrich": "enrich_interval_minutes",
|
||||
"backfill": "backfill_interval_minutes",
|
||||
"autotag": "autotag_interval_minutes",
|
||||
"shorts": "shorts_probe_interval_minutes",
|
||||
"subscriptions": "subscriptions_resync_minutes",
|
||||
"playlist_sync": "playlist_sync_minutes",
|
||||
"maintenance": "maintenance_interval_minutes",
|
||||
"explore_cleanup": "explore_cleanup_minutes",
|
||||
"download_gc": "download_gc_minutes",
|
||||
"plex_sync": "plex_sync_interval_min",
|
||||
"plex_watch_sync": "plex_watch_sync_interval_min",
|
||||
"plex_watch_reconcile": "plex_watch_reconcile_interval_min",
|
||||
"tmdb_enrich": "tmdb_enrich_interval_min",
|
||||
"people_enrich": "people_enrich_interval_min",
|
||||
"audit_gc": "audit_gc_minutes",
|
||||
"log_gc": "log_gc_minutes",
|
||||
}
|
||||
# Sane bounds for an admin-set interval (minutes).
|
||||
MIN_INTERVAL = 1
|
||||
MAX_INTERVAL = 1440 # one day
|
||||
|
||||
|
||||
def interval_key(job: str) -> str:
|
||||
return f"job_interval_{job}"
|
||||
|
||||
|
||||
# Registry of DB-overridable keys. Add a key here + wire its read through get_*() to move it
|
||||
# off env into the admin UI; the admin page renders it automatically from this list. A group's keys
|
||||
# stay together, in the order its section lists them; a NEW group also needs its place in SECTIONS.
|
||||
SPECS: tuple[ConfigSpec, ...] = (
|
||||
# --- Access / registration ---
|
||||
ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
|
||||
ConfigSpec("allow_registration", "bool", "access", "allow_registration", applies=LIVE),
|
||||
# --- Email / SMTP (one logical group; the password is the only secret) ---
|
||||
ConfigSpec("smtp_host", "str", "email", "smtp_host"),
|
||||
ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535),
|
||||
ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True),
|
||||
ConfigSpec("smtp_from", "str", "email", "smtp_from"),
|
||||
ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True),
|
||||
ConfigSpec("smtp_host", "str", "email", "smtp_host", applies=LIVE),
|
||||
ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535, applies=LIVE),
|
||||
ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True, applies=LIVE),
|
||||
ConfigSpec("smtp_from", "str", "email", "smtp_from", applies=LIVE),
|
||||
ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True, applies=LIVE),
|
||||
# --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
|
||||
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
|
||||
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True, applies=LIVE),
|
||||
# Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
|
||||
ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
|
||||
ConfigSpec(
|
||||
"youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True, applies=LIVE
|
||||
),
|
||||
# --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
|
||||
# install wizard / Configuration page so no restart is needed when the creds change. ---
|
||||
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
|
||||
ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
|
||||
# --- Quota (shared YouTube Data API daily budget) ---
|
||||
ConfigSpec("quota_daily_budget", "int", "quota", "quota_daily_budget", min=1, max=1_000_000),
|
||||
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True, applies=LIVE),
|
||||
ConfigSpec(
|
||||
"backfill_quota_reserve", "int", "quota", "backfill_quota_reserve", min=0, max=1_000_000
|
||||
"google_client_secret", "str", "google", "google_client_secret", secret=True, applies=LIVE
|
||||
),
|
||||
# --- Quota (shared YouTube Data API daily budget) ---
|
||||
ConfigSpec(
|
||||
"quota_daily_budget",
|
||||
"int",
|
||||
"quota",
|
||||
"quota_daily_budget",
|
||||
min=1,
|
||||
max=1_000_000,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"backfill_quota_reserve",
|
||||
"int",
|
||||
"quota",
|
||||
"backfill_quota_reserve",
|
||||
min=0,
|
||||
max=1_000_000,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"search_daily_limit_per_user",
|
||||
@@ -77,10 +138,19 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
"search_daily_limit_per_user",
|
||||
min=0,
|
||||
max=100_000,
|
||||
applies=LIVE,
|
||||
),
|
||||
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
|
||||
ConfigSpec("search_source", "choice", "quota", "search_source"),
|
||||
ConfigSpec("quota_retention_days", "int", "quota", "quota_retention_days", min=0, max=3650),
|
||||
ConfigSpec("search_source", "choice", "quota", "search_source", applies=LIVE),
|
||||
ConfigSpec(
|
||||
"quota_retention_days",
|
||||
"int",
|
||||
"quota",
|
||||
"quota_retention_days",
|
||||
min=0,
|
||||
max=3650,
|
||||
applies=LIVE,
|
||||
),
|
||||
# --- Backfill (recent-first first pass per channel) ---
|
||||
ConfigSpec(
|
||||
"backfill_recent_max_videos",
|
||||
@@ -89,19 +159,46 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
"backfill_recent_max_videos",
|
||||
min=1,
|
||||
max=100_000,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"backfill_recent_max_days", "int", "backfill", "backfill_recent_max_days", min=1, max=36_500
|
||||
"backfill_recent_max_days",
|
||||
"int",
|
||||
"backfill",
|
||||
"backfill_recent_max_days",
|
||||
min=1,
|
||||
max=36_500,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"rss_fallback_hours", "int", "backfill", "rss_fallback_hours", min=0, max=168, applies=LIVE
|
||||
),
|
||||
ConfigSpec("rss_fallback_hours", "int", "backfill", "rss_fallback_hours", min=0, max=168),
|
||||
# --- Shorts probe ---
|
||||
ConfigSpec(
|
||||
"shorts_probe_max_seconds", "int", "shorts", "shorts_probe_max_seconds", min=1, max=3_600
|
||||
"shorts_probe_max_seconds",
|
||||
"int",
|
||||
"shorts",
|
||||
"shorts_probe_max_seconds",
|
||||
min=1,
|
||||
max=3_600,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"shorts_probe_batch", "int", "shorts", "shorts_probe_batch", min=1, max=10_000, applies=LIVE
|
||||
),
|
||||
ConfigSpec("shorts_probe_batch", "int", "shorts", "shorts_probe_batch", min=1, max=10_000),
|
||||
# --- Batch sizes ---
|
||||
ConfigSpec("enrich_batch_size", "int", "batch", "enrich_batch_size", min=1, max=50),
|
||||
ConfigSpec("autotag_title_sample", "int", "batch", "autotag_title_sample", min=1, max=1_000),
|
||||
ConfigSpec(
|
||||
"enrich_batch_size", "int", "batch", "enrich_batch_size", min=1, max=50, applies=LIVE
|
||||
),
|
||||
ConfigSpec(
|
||||
"autotag_title_sample",
|
||||
"int",
|
||||
"batch",
|
||||
"autotag_title_sample",
|
||||
min=1,
|
||||
max=1_000,
|
||||
applies=LIVE,
|
||||
),
|
||||
# --- Maintenance (rolling re-validation of already-known videos) ---
|
||||
# Videos re-checked per maintenance run. Folded here from app_state (B-core-11) so every admin
|
||||
# knob lives in one store; read by sync/maintenance.py, tuned from the Scheduler dashboard.
|
||||
@@ -112,16 +209,33 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
"maintenance_revalidate_batch",
|
||||
min=100,
|
||||
max=100_000,
|
||||
applies=LIVE,
|
||||
),
|
||||
# --- Discovery cleanup (search + explore ephemeral content) ---
|
||||
# Days an explored-but-unsubscribed channel is kept before the cleanup job purges it.
|
||||
ConfigSpec("explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650),
|
||||
# Days an un-kept live-search result video is kept before the cleanup job removes it.
|
||||
ConfigSpec("search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650),
|
||||
# --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
|
||||
ConfigSpec("download_total_max_bytes", "int", "downloads", "download_total_max_bytes", min=0),
|
||||
ConfigSpec(
|
||||
"download_default_max_bytes", "int", "downloads", "download_default_max_bytes", min=0
|
||||
"explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650, applies=LIVE
|
||||
),
|
||||
# Days an un-kept live-search result video is kept before the cleanup job removes it.
|
||||
ConfigSpec(
|
||||
"search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650, applies=LIVE
|
||||
),
|
||||
# --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
|
||||
ConfigSpec(
|
||||
"download_total_max_bytes",
|
||||
"int",
|
||||
"downloads",
|
||||
"download_total_max_bytes",
|
||||
min=0,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"download_default_max_bytes",
|
||||
"int",
|
||||
"downloads",
|
||||
"download_default_max_bytes",
|
||||
min=0,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"download_default_max_concurrent",
|
||||
@@ -130,6 +244,7 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
"download_default_max_concurrent",
|
||||
min=1,
|
||||
max=20,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"download_default_max_jobs",
|
||||
@@ -138,6 +253,7 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
"download_default_max_jobs",
|
||||
min=1,
|
||||
max=100_000,
|
||||
applies=LIVE,
|
||||
),
|
||||
# min=1, NOT 0: a stored 0 used to read as "no per-file cap" (the cap was enforced as
|
||||
# `if cap and size > cap`, so a falsy value disabled it). Now the route passes this number
|
||||
@@ -161,14 +277,27 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
"download_upload_max_bytes",
|
||||
min=1,
|
||||
max=settings.download_upload_max_bytes,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"download_retention_days", "int", "downloads", "download_retention_days", min=0, max=3_650
|
||||
"download_retention_days",
|
||||
"int",
|
||||
"downloads",
|
||||
"download_retention_days",
|
||||
min=0,
|
||||
max=3_650,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec(
|
||||
"download_gc_grace_days", "int", "downloads", "download_gc_grace_days", min=0, max=3_650
|
||||
"download_gc_grace_days",
|
||||
"int",
|
||||
"downloads",
|
||||
"download_gc_grace_days",
|
||||
min=0,
|
||||
max=3_650,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec("download_layout", "choice", "downloads", "download_layout"),
|
||||
ConfigSpec("download_layout", "choice", "downloads", "download_layout", applies=LIVE),
|
||||
ConfigSpec(
|
||||
"download_worker_concurrency",
|
||||
"int",
|
||||
@@ -176,60 +305,112 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
"download_worker_concurrency",
|
||||
min=1,
|
||||
max=16,
|
||||
applies=LIVE,
|
||||
),
|
||||
ConfigSpec("sponsorblock_default", "bool", "downloads", "sponsorblock_default"),
|
||||
# --- Audit log ---
|
||||
# Days to keep admin audit-log rows before the audit_gc job prunes them (0 = keep forever).
|
||||
ConfigSpec("audit_retention_days", "int", "audit", "audit_retention_days", min=0, max=3650),
|
||||
ConfigSpec(
|
||||
"audit_retention_days",
|
||||
"int",
|
||||
"audit",
|
||||
"audit_retention_days",
|
||||
min=0,
|
||||
max=3650,
|
||||
applies=LIVE,
|
||||
),
|
||||
# Days the job_runs table keeps a finished background run before the audit_gc job prunes it.
|
||||
ConfigSpec(
|
||||
"job_run_retention_days",
|
||||
"int",
|
||||
"audit",
|
||||
"job_run_retention_days",
|
||||
min=0,
|
||||
max=3650,
|
||||
applies=LIVE,
|
||||
),
|
||||
# --- Diagnostics ---
|
||||
# Runtime log level for the API process. Applied without a restart — see routes/config.py and
|
||||
# main.py's lifespan.
|
||||
ConfigSpec("log_level", "choice", "diagnostics", "log_level"),
|
||||
ConfigSpec("log_level", "choice", "diagnostics", "log_level", applies=LIVE),
|
||||
# Days the log_events table keeps a row (the log_gc job prunes). No "0 = forever": stored logs
|
||||
# carry account-linked request history, and the privacy policy names a limit.
|
||||
ConfigSpec("log_retention_days", "int", "diagnostics", "log_retention_days", min=1, max=90),
|
||||
ConfigSpec(
|
||||
"log_retention_days",
|
||||
"int",
|
||||
"diagnostics",
|
||||
"log_retention_days",
|
||||
min=1,
|
||||
max=90,
|
||||
applies=LIVE,
|
||||
),
|
||||
# --- Plex integration (optional; local-file playback + Plex metadata) ---
|
||||
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled"),
|
||||
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url"),
|
||||
ConfigSpec("plex_server_token", "str", "plex", "plex_server_token", secret=True),
|
||||
ConfigSpec("plex_path_map", "str", "plex", "plex_path_map"),
|
||||
ConfigSpec("plex_libraries", "str", "plex", "plex_libraries"),
|
||||
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
|
||||
# (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
|
||||
ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
|
||||
ConfigSpec("plex_max_sessions", "int", "plex", "plex_max_sessions", min=1, max=64),
|
||||
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled", applies=LIVE),
|
||||
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url", applies=LIVE),
|
||||
ConfigSpec("plex_server_token", "str", "plex", "plex_server_token", secret=True, applies=LIVE),
|
||||
ConfigSpec("plex_path_map", "str", "plex", "plex_path_map", applies=LIVE),
|
||||
ConfigSpec("plex_libraries", "str", "plex", "plex_libraries", applies=LIVE),
|
||||
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map", applies=LIVE),
|
||||
ConfigSpec(
|
||||
"plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16, applies=LIVE
|
||||
),
|
||||
ConfigSpec(
|
||||
"plex_max_sessions", "int", "plex", "plex_max_sessions", min=1, max=64, applies=LIVE
|
||||
),
|
||||
# Transcode re-encode tuning. The libx264 knobs apply only to the software fallback.
|
||||
# crf 0–51 (x264 range); max_height 0 = source.
|
||||
ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw"),
|
||||
ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset"),
|
||||
ConfigSpec("plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51),
|
||||
ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw", applies=LIVE),
|
||||
ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset", applies=LIVE),
|
||||
ConfigSpec(
|
||||
"plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
|
||||
"plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51, applies=LIVE
|
||||
),
|
||||
ConfigSpec(
|
||||
"plex_transcode_max_height",
|
||||
"int",
|
||||
"plex",
|
||||
"plex_transcode_max_height",
|
||||
min=0,
|
||||
max=4320,
|
||||
applies=LIVE,
|
||||
),
|
||||
# Audio loudness normalisation on the HLS path (remux + transcode alike, not direct play).
|
||||
ConfigSpec("plex_audio_loudnorm", "bool", "plex", "plex_audio_loudnorm"),
|
||||
# NOTE: plex_watch_sync_interval_min is a config.py default read by the watch-sync scheduler job
|
||||
# (scheduler.py); it is intentionally NOT a runtime admin knob, so it stays out of SPECS.
|
||||
ConfigSpec("plex_audio_loudnorm", "bool", "plex", "plex_audio_loudnorm", applies=LIVE),
|
||||
# --- TMDB enrichment (optional) — the API key is the only secret ---
|
||||
ConfigSpec("tmdb_api_key", "str", "tmdb", "tmdb_api_key", secret=True),
|
||||
ConfigSpec("tmdb_language", "str", "tmdb", "tmdb_language"),
|
||||
ConfigSpec("tmdb_api_key", "str", "tmdb", "tmdb_api_key", secret=True, applies=LIVE),
|
||||
ConfigSpec("tmdb_language", "str", "tmdb", "tmdb_language", applies=LIVE),
|
||||
# --- Bug reports: the Gitea issue tracker a debug session's issue goes to ---
|
||||
ConfigSpec("issues_gitea_url", "str", "issues", "issues_gitea_url"),
|
||||
ConfigSpec("issues_gitea_repo", "str", "issues", "issues_gitea_repo"),
|
||||
ConfigSpec("issues_gitea_token", "str", "issues", "issues_gitea_token", secret=True),
|
||||
# (tmdb_enrich_interval_min is a scheduler job, tuned from the Scheduler dashboard; tmdb_enrich_batch
|
||||
# is a config.py default — both stay out of SPECS, like plex_sync_interval_min.)
|
||||
ConfigSpec("issues_gitea_url", "str", "issues", "issues_gitea_url", applies=LIVE),
|
||||
ConfigSpec("issues_gitea_repo", "str", "issues", "issues_gitea_repo", applies=LIVE),
|
||||
ConfigSpec(
|
||||
"issues_gitea_token", "str", "issues", "issues_gitea_token", secret=True, applies=LIVE
|
||||
),
|
||||
# --- Scheduler: how often each interval job runs (minutes), one key per job ---
|
||||
*(
|
||||
ConfigSpec(
|
||||
interval_key(job),
|
||||
"int",
|
||||
"scheduler",
|
||||
attr,
|
||||
min=MIN_INTERVAL,
|
||||
max=MAX_INTERVAL,
|
||||
applies=LIVE,
|
||||
)
|
||||
for job, attr in JOB_INTERVAL_DEFAULTS.items()
|
||||
),
|
||||
)
|
||||
|
||||
# The admin page's sections (the owner's pick): sixteen groups as tabs wrapped to three rows at 133%
|
||||
# zoom, so they are a vertical list under four headings, in this order. Every group of SPECS sits in
|
||||
# exactly one section (test-held), so a new group cannot fall off the page.
|
||||
# exactly one section, or is edited on its own page (test-held), so a new group cannot fall off.
|
||||
SECTIONS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
("access", ("access", "google", "email")),
|
||||
("youtube", ("youtube", "quota", "backfill", "shorts", "batch", "explore")),
|
||||
("media", ("downloads", "plex", "tmdb")),
|
||||
("system", ("maintenance", "audit", "diagnostics", "issues")),
|
||||
)
|
||||
# Groups another admin page edits, so the Configuration page neither lists nor writes them: an
|
||||
# interval is changed on the Scheduler page, which also reschedules the running job — a write that
|
||||
# skipped it would leave the job on its old period while the page showed the new one.
|
||||
OWN_PAGE_GROUPS: tuple[str, ...] = ("scheduler",)
|
||||
|
||||
_BY_KEY: dict[str, ConfigSpec] = {s.key: s for s in SPECS}
|
||||
|
||||
@@ -238,6 +419,12 @@ def spec(key: str) -> ConfigSpec | None:
|
||||
return _BY_KEY.get(key)
|
||||
|
||||
|
||||
def config_page_spec(key: str) -> ConfigSpec | None:
|
||||
"""`spec`, for the Configuration page: None for a key edited on its own page as well."""
|
||||
s = _BY_KEY.get(key)
|
||||
return None if s is None or s.group in OWN_PAGE_GROUPS else s
|
||||
|
||||
|
||||
def secrets_manageable() -> bool:
|
||||
"""Secret values can only be stored when Fernet is configured (TOKEN_ENCRYPTION_KEY)."""
|
||||
return bool(settings.token_encryption_key)
|
||||
@@ -426,13 +613,16 @@ def reset(db: Session, key: str) -> None:
|
||||
|
||||
|
||||
def describe(db: Session) -> dict:
|
||||
"""Admin-UI metadata + current values, grouped. Secret values are never echoed —
|
||||
only whether an override is set and whether an env default exists."""
|
||||
"""Admin-UI metadata + current values, grouped — the keys of the Configuration page, not those
|
||||
edited on their own page. Secret values are never echoed — only whether an override is set and
|
||||
whether an env default exists."""
|
||||
groups: dict[str, list[dict]] = {}
|
||||
# Every override in one query: a db.get per key was one round trip each (two for a key with no
|
||||
# row, which the session cannot cache), ~40 ms of the page's ~55 on prod.
|
||||
rows = {r.key: r for r in db.scalars(select(SystemConfig))}
|
||||
for s in SPECS:
|
||||
if s.group in OWN_PAGE_GROUPS:
|
||||
continue
|
||||
row = rows.get(s.key)
|
||||
is_set = row is not None
|
||||
item: dict = {
|
||||
|
||||
@@ -16,6 +16,9 @@ services:
|
||||
POSTGRES_DB: siftlode_test
|
||||
tmpfs:
|
||||
- /var/lib/postgresql/data # ephemeral + fast; the schema is rebuilt per run
|
||||
# One database per pytest-xdist worker, each through the app's own pool (5 + 10 overflow): the default 100
|
||||
# connections would run out at 8 workers.
|
||||
command: ["postgres", "-c", "max_connections=300"]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U siftlode -d siftlode_test"]
|
||||
interval: 1s
|
||||
@@ -37,4 +40,4 @@ services:
|
||||
- ./:/app
|
||||
working_dir: /app
|
||||
# Coverage is on demand, not in the gate: see docs/development.md.
|
||||
command: ["python", "-m", "pytest"]
|
||||
command: ["python", "-m", "pytest", "-n", "auto"]
|
||||
|
||||
+6
-1
@@ -7,4 +7,9 @@
|
||||
# not an unknown marker that quietly does nothing.
|
||||
testpaths = tests
|
||||
pythonpath = .
|
||||
addopts = -q -rs --strict-markers
|
||||
# `-p tests.db_per_worker`: under pytest-xdist (`-n`) every worker gets its own database — DATABASE_URL + `_gw<N>`,
|
||||
# created on first use — because the `db` fixture's TRUNCATE-per-test isolation assumes one process per database.
|
||||
# It is a plugin so it runs before conftest imports the app (whose settings read DATABASE_URL once). Serial: no-op.
|
||||
# The file is a verbatim copy of a shared tool (see its docstring); change the source, not the copy.
|
||||
addopts = -q -rs --strict-markers -p tests.db_per_worker
|
||||
db_per_worker_env = DATABASE_URL
|
||||
|
||||
@@ -16,3 +16,6 @@ pytest-cov==7.1.0
|
||||
ruff==0.15.21
|
||||
httpx2==2.13.0
|
||||
pip-audit==2.10.1
|
||||
# pytest-xdist: the DB lane runs on every core (`-n auto` in docker-compose.test.yml); tests/db_per_worker.py gives
|
||||
# each worker its own database. Same tests, same strictness — only side by side.
|
||||
pytest-xdist==3.8.0
|
||||
|
||||
@@ -27,7 +27,7 @@ from starlette.requests import HTTPConnection
|
||||
|
||||
import app.models # register every model on Base.metadata
|
||||
import app.state as app_state
|
||||
from app import security
|
||||
from app import apitokens, security
|
||||
from app.auth import current_user
|
||||
from app.config import settings
|
||||
from app.db import Base, SessionLocal, engine, get_db
|
||||
@@ -242,8 +242,16 @@ def as_user(db, override_dep, _configured):
|
||||
raise HTTPException(status_code=401, detail="Not authenticated")
|
||||
return registry[uid]
|
||||
|
||||
def _session_or_token(request: Request) -> User | None:
|
||||
# The Files routes' session half (`apitokens.session_caller`): a request carrying a token is
|
||||
# left to the real token check, exactly as in production.
|
||||
if apitokens.bearer_token(request) is not None:
|
||||
return None
|
||||
return _resolver(request)
|
||||
|
||||
override_dep(get_db, db.info["request"])
|
||||
override_dep(current_user, _resolver)
|
||||
override_dep(apitokens.session_caller, _session_or_token)
|
||||
|
||||
def _as(user: User) -> TestClient:
|
||||
registry[user.id] = user
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Give every pytest-xdist worker its own database, so a suite that shares one test database can run in parallel.
|
||||
|
||||
Generic (source of truth: the memory repo's tools/pytest-parallel/db_per_worker.py); a project carries a verbatim copy
|
||||
next to its conftest — tests often run in a container that cannot see this repo — and
|
||||
`tools/pytest-parallel/sync-check.sh <copy>` reports drift. Keep the copy byte-identical; change it here.
|
||||
|
||||
Use, as a pytest plugin (it then runs before any conftest, i.e. before the app reads its settings):
|
||||
pytest.ini: addopts = -p tests.db_per_worker (the copy's import path; `python -m pytest` puts cwd on sys.path)
|
||||
db_per_worker_env = DATABASE_URL (the environment variable that holds the test database)
|
||||
or call `apply("<ENV VAR>")` yourself before the app is imported. The variable may hold:
|
||||
- a Postgres URL (`postgresql://…/name`, `postgresql+psycopg://…`): the worker gets `name_gw<N>`, created from the
|
||||
base database's server if missing (empty — the project's own schema fixture migrates it, as for one process);
|
||||
- a SQLite URL (`sqlite:///path.db`) or a plain file path: the worker gets `path-gw<N>.db`.
|
||||
Without xdist (no PYTEST_XDIST_WORKER) or with the variable unset it changes nothing, so a serial run is unaffected.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
import pytest
|
||||
|
||||
_PG = re.compile(r"^postgres(?:ql)?(?:\+\w+)?://", re.I)
|
||||
|
||||
|
||||
def pytest_addoption(parser: pytest.Parser) -> None:
|
||||
parser.addini(
|
||||
"db_per_worker_env",
|
||||
"env var holding the test database (a Postgres URL, SQLite URL or file path)",
|
||||
)
|
||||
|
||||
|
||||
@pytest.hookimpl(tryfirst=True)
|
||||
def pytest_load_initial_conftests(early_config: pytest.Config) -> None:
|
||||
var = early_config.getini("db_per_worker_env")
|
||||
if var:
|
||||
apply(str(var))
|
||||
|
||||
|
||||
def apply(var: str) -> str | None:
|
||||
worker = os.environ.get("PYTEST_XDIST_WORKER")
|
||||
value = os.environ.get(var)
|
||||
if not worker or not value:
|
||||
return value
|
||||
if _PG.match(value):
|
||||
new = _postgres(value, worker)
|
||||
else:
|
||||
new = _file(value, worker)
|
||||
os.environ[var] = new
|
||||
return new
|
||||
|
||||
|
||||
def _postgres(url: str, worker: str) -> str:
|
||||
parts = urlsplit(url)
|
||||
base = parts.path.lstrip("/")
|
||||
name = f"{base}_{worker}"
|
||||
if not re.fullmatch(r"\w+", name):
|
||||
raise ValueError(f"db_per_worker: refusing an unusual database name {name!r}")
|
||||
# The server, through the BASE database (it exists; `postgres` may not be reachable for this role).
|
||||
plain = urlunsplit(
|
||||
(re.sub(r"\+\w+$", "", parts.scheme), parts.netloc, parts.path, parts.query, "")
|
||||
)
|
||||
_create_if_missing(plain, name)
|
||||
return urlunsplit((parts.scheme, parts.netloc, f"/{name}", parts.query, parts.fragment))
|
||||
|
||||
|
||||
def _create_if_missing(dsn: str, name: str) -> None:
|
||||
try:
|
||||
import psycopg # psycopg 3
|
||||
|
||||
conn = psycopg.connect(dsn, autocommit=True)
|
||||
except ImportError:
|
||||
import psycopg2 # fallback for projects on psycopg2
|
||||
|
||||
conn = psycopg2.connect(dsn)
|
||||
conn.autocommit = True
|
||||
try:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT 1 FROM pg_database WHERE datname = %s", (name,))
|
||||
if cur.fetchone() is None:
|
||||
cur.execute(f'CREATE DATABASE "{name}"') # name checked to be \w+ above
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _file(value: str, worker: str) -> str:
|
||||
prefix = ""
|
||||
m = re.match(r"^(sqlite(?:\+\w+)?:///)(.*)$", value, re.I)
|
||||
if m:
|
||||
prefix, value = m.group(1), m.group(2)
|
||||
root, ext = os.path.splitext(value)
|
||||
return f"{prefix}{root}-{worker}{ext}"
|
||||
@@ -0,0 +1,205 @@
|
||||
"""The activity page's reads: the audit log and the background runs, filtered on the server and
|
||||
paged by a keyset, and the quota breakdown over a time range and one account.
|
||||
|
||||
Each list is asked for the rows a filter must keep AND must drop, and a page boundary is crossed
|
||||
with rows that share a timestamp — the case an offset or a timestamp-only cursor gets wrong.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from app.auth import admin_user
|
||||
from app.main import app
|
||||
from app.models import AuditLog, Channel, JobRun, LogEvent, QuotaEvent
|
||||
from tests.test_body_limit import _route_objects
|
||||
|
||||
T0 = datetime(2026, 10, 1, 12, 0, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _audit(db, action, actor=None, at=T0, summary=None, request_id=None) -> int:
|
||||
row = AuditLog(
|
||||
actor_id=actor, action=action, summary=summary, created_at=at, request_id=request_id
|
||||
)
|
||||
db.add(row)
|
||||
db.flush()
|
||||
return row.id
|
||||
|
||||
|
||||
def _run(db, job, status="ok", actor=None, at=T0, target=None, trigger="scheduled") -> int:
|
||||
row = JobRun(
|
||||
job=job,
|
||||
trigger=trigger,
|
||||
actor_id=actor,
|
||||
target=target,
|
||||
status=status,
|
||||
started_at=at,
|
||||
finished_at=None if status == "running" else at + timedelta(seconds=4),
|
||||
quota_units=0,
|
||||
)
|
||||
db.add(row)
|
||||
db.flush()
|
||||
return row.id
|
||||
|
||||
|
||||
def _ids(body) -> list[int]:
|
||||
return [r["id"] for r in body["items"]]
|
||||
|
||||
|
||||
def test_the_audit_list_filters_on_the_server_and_pages_across_a_shared_timestamp(
|
||||
db, make_user, as_user
|
||||
):
|
||||
admin = make_user(role="admin")
|
||||
a = _audit(db, "config_set", admin.id, T0, "quota_daily_budget = 9000")
|
||||
b = _audit(db, "config_set", admin.id, T0, "log_level = DEBUG")
|
||||
c = _audit(db, "config_set", admin.id, T0, "smtp_host = mail")
|
||||
bg = _audit(db, "user_purged", None, T0 - timedelta(days=3))
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
|
||||
first = client.get("/api/admin/audit", params={"limit": 2, "action": "config_set"}).json()
|
||||
assert _ids(first) == [c, b]
|
||||
second = client.get(
|
||||
"/api/admin/audit",
|
||||
params={"limit": 2, "action": "config_set", "before": first["next_cursor"]},
|
||||
).json()
|
||||
assert (_ids(second), second["next_cursor"]) == ([a], None)
|
||||
# A page that ends exactly at the last row says there is nothing older.
|
||||
whole = client.get("/api/admin/audit", params={"limit": 3, "action": "config_set"}).json()
|
||||
assert (_ids(whole), whole["next_cursor"]) == ([c, b, a], None)
|
||||
assert first["accounts"] == {str(admin.id): admin.email}
|
||||
|
||||
def ids(**params) -> list[int]:
|
||||
return _ids(client.get("/api/admin/audit", params=params).json())
|
||||
|
||||
assert ids(user="none") == [bg]
|
||||
assert ids(user=str(admin.id)) == [c, b, a]
|
||||
assert ids(q="DEBUG") == [b]
|
||||
assert ids(since=(T0 - timedelta(days=1)).isoformat()) == [c, b, a]
|
||||
assert ids(until=(T0 - timedelta(days=1)).isoformat()) == [bg]
|
||||
# The action menu counts every action under the other filters, its own pick included.
|
||||
facets = client.get("/api/admin/audit/facets", params={"action": "config_set"}).json()
|
||||
assert facets["actions"] == [
|
||||
{"value": "config_set", "count": 3},
|
||||
{"value": "user_purged", "count": 1},
|
||||
]
|
||||
|
||||
|
||||
def test_an_admin_action_keeps_the_request_it_was_made_in(db, make_user, as_user):
|
||||
client = as_user(make_user(role="admin"))
|
||||
r = client.patch("/api/admin/config/quota_daily_budget", json={"value": 9500})
|
||||
assert r.status_code == 200, r.text
|
||||
(row,) = client.get("/api/admin/audit", params={"action": "config_set"}).json()["items"]
|
||||
assert row["request_id"] == r.headers["x-request-id"]
|
||||
|
||||
|
||||
def test_an_admin_action_says_how_many_of_its_requests_lines_are_still_kept(db, make_user, as_user):
|
||||
"""A request's lines are kept days, its audit row months: a link to lines gone would open an
|
||||
empty list, so the row says how many are left, as a run does."""
|
||||
admin = make_user(role="admin")
|
||||
kept = _audit(db, "config_set", actor=admin.id, request_id="req-kept-0001")
|
||||
gone = _audit(db, "config_set", actor=admin.id, request_id="req-gone-0001")
|
||||
db.add_all(
|
||||
[
|
||||
LogEvent(
|
||||
ts=T0,
|
||||
level="INFO",
|
||||
sev=20,
|
||||
source="api",
|
||||
logger="t",
|
||||
msg="m",
|
||||
request_id="req-kept-0001",
|
||||
)
|
||||
for _ in range(3)
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
items = as_user(admin).get("/api/admin/audit").json()["items"]
|
||||
lines = {r["id"]: r["log_lines"] for r in items}
|
||||
assert (lines[kept], lines[gone]) == (3, 0)
|
||||
|
||||
|
||||
def test_the_run_list_filters_pages_and_names_the_channel_and_the_log_lines(db, make_user, as_user):
|
||||
admin = make_user(role="admin")
|
||||
db.add(Channel(id="UCchan", title="Aileen Wang"))
|
||||
ok1 = _run(db, "rss_poll", at=T0)
|
||||
ok2 = _run(db, "rss_poll", at=T0)
|
||||
bad = _run(db, "backfill", "error", admin.id, T0, trigger="manual")
|
||||
adhoc = _run(
|
||||
db, "channel_backfill", "ok", admin.id, T0 - timedelta(hours=1), "UCchan", "ad_hoc"
|
||||
)
|
||||
for _ in range(2):
|
||||
db.add(
|
||||
LogEvent(ts=T0, level="WARNING", sev=30, source="api", logger="t", msg="m", run_id=bad)
|
||||
)
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
|
||||
first = client.get("/api/admin/runs", params={"limit": 2}).json()
|
||||
rest = client.get("/api/admin/runs", params={"limit": 2, "before": first["next_cursor"]}).json()
|
||||
assert _ids(first) + _ids(rest) == [bad, ok2, ok1, adhoc]
|
||||
assert rest["next_cursor"] is None
|
||||
rows = {r["id"]: r for r in first["items"] + rest["items"]}
|
||||
assert (rows[bad]["log_lines"], rows[ok2]["log_lines"]) == (2, 0)
|
||||
assert (rows[adhoc]["target_title"], rows[ok1]["target_title"]) == ("Aileen Wang", None)
|
||||
|
||||
def ids(**params) -> list[int]:
|
||||
return _ids(client.get("/api/admin/runs", params=params).json())
|
||||
|
||||
assert ids(status="error") == [bad]
|
||||
assert ids(job="rss_poll") == [ok2, ok1]
|
||||
assert ids(user="none") == [ok2, ok1]
|
||||
assert ids(user=str(admin.id)) == [bad, adhoc]
|
||||
assert ids(q="UCchan") == [adhoc]
|
||||
assert ids(since=(T0 - timedelta(minutes=30)).isoformat()) == [bad, ok2, ok1]
|
||||
assert client.get("/api/admin/runs", params={"status": "finished"}).status_code == 422
|
||||
|
||||
facets = client.get("/api/admin/runs/facets", params={"status": "error"}).json()
|
||||
assert facets["statuses"] == [{"value": "ok", "count": 3}, {"value": "error", "count": 1}]
|
||||
assert facets["jobs"] == [{"value": "backfill", "count": 1}]
|
||||
assert facets["total"] == 4
|
||||
|
||||
|
||||
def test_the_quota_breakdown_covers_its_range_and_one_account(db, make_user, as_user):
|
||||
admin = make_user(role="admin")
|
||||
now = datetime.now(timezone.utc)
|
||||
db.add_all(
|
||||
[
|
||||
QuotaEvent(user_id=admin.id, action="search", units=100, created_at=now),
|
||||
QuotaEvent(user_id=None, action="backfill", units=7, created_at=now),
|
||||
QuotaEvent(
|
||||
user_id=admin.id, action="search", units=50, created_at=now - timedelta(days=10)
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
client = as_user(admin)
|
||||
|
||||
def totals(**params) -> dict:
|
||||
body = client.get("/api/quota/admin", params=params).json()
|
||||
# The chart is the same filter's: its days add up to the table's total.
|
||||
assert sum(d["total"] for d in body["daily"]) == sum(r["total"] for r in body["rows"])
|
||||
return {r["user_id"]: r["total"] for r in body["rows"]}
|
||||
|
||||
assert totals() == {admin.id: 150, None: 7} # the last 30 days by default
|
||||
assert totals(since=(now - timedelta(days=1)).isoformat()) == {admin.id: 100, None: 7}
|
||||
assert totals(until=(now - timedelta(days=1)).isoformat()) == {admin.id: 50}
|
||||
assert totals(user="none") == {None: 7}
|
||||
assert totals(user=str(admin.id)) == {admin.id: 150}
|
||||
|
||||
|
||||
def test_every_admin_read_and_write_is_admin_only():
|
||||
"""The route table, not a list kept by hand: every /api/admin route and the quota breakdown
|
||||
resolve `admin_user`, so a dropped dependency cannot hand an account's runs, errors and e-mail
|
||||
addresses to any signed-in user."""
|
||||
routes = _route_objects(app)
|
||||
admin = {k: r for k, r in routes.items() if k[1].startswith(("/api/admin", "/api/quota/admin"))}
|
||||
assert ("GET", "/api/admin/runs") in admin and ("GET", "/api/quota/admin") in admin
|
||||
open_ = []
|
||||
for key, route in admin.items():
|
||||
stack, calls = [route.dependant], set()
|
||||
while stack:
|
||||
d = stack.pop()
|
||||
calls.add(d.call)
|
||||
stack.extend(d.dependencies)
|
||||
if admin_user not in calls:
|
||||
open_.append(key)
|
||||
assert open_ == []
|
||||
@@ -24,6 +24,7 @@ def _row(**kw) -> dict:
|
||||
"request_id": None,
|
||||
"user_id": None,
|
||||
"job_id": None,
|
||||
"run_id": None,
|
||||
"version": "0.90.0",
|
||||
}
|
||||
row.update(kw)
|
||||
@@ -80,6 +81,7 @@ def test_newest_first_and_every_row_once_across_pages_even_on_equal_timestamps(d
|
||||
({"event": "none"}, {"warn"}),
|
||||
({"request_id": "req-abcdef12"}, {"mine"}),
|
||||
({"job_id": 42}, {"err"}),
|
||||
({"run_id": 7}, {"warn"}), # a background run's lines, not a download job's
|
||||
({"q": "FORBIDDEN"}, {"err"}), # in attrs, any case
|
||||
({"q": "KeyError"}, {"warn"}), # in the stack
|
||||
({"q": "100%"}, {"plain"}), # a literal, not a wildcard
|
||||
@@ -102,7 +104,14 @@ def test_each_filter_selects_exactly_its_rows(db, admin, make_user, params, expe
|
||||
job_id=42,
|
||||
attrs={"code": "forbidden"},
|
||||
),
|
||||
_row(msg="warn", level="WARNING", sev=13, event=None, exc="Traceback …\nKeyError: 'x'"),
|
||||
_row(
|
||||
msg="warn",
|
||||
level="WARNING",
|
||||
sev=13,
|
||||
event=None,
|
||||
exc="Traceback …\nKeyError: 'x'",
|
||||
run_id=7,
|
||||
),
|
||||
_row(msg="plain", ts=T0 - timedelta(hours=2), attrs={"note": "100% done"}),
|
||||
# Matched by no filter above but `until`: another account, and `100%` read as a pattern.
|
||||
_row(msg="decoy", ts=T0 - timedelta(hours=3), attrs={"note": "1000 done"}, user_id=99),
|
||||
|
||||
@@ -0,0 +1,648 @@
|
||||
"""API tokens for the Files library (`app.apitokens`, `routes.tokens`), DB-lane.
|
||||
|
||||
A token is made, listed and revoked by its signed-in user only; its value is in the answer that made
|
||||
it and nowhere after. With it a machine reaches `/api/files` and nothing else, and there only the
|
||||
folders its grants name, with the rights they give — every right checked in a folder it has and in
|
||||
one it has not. A token that is malformed, unknown, expired, revoked, or whose account is suspended
|
||||
holds nothing, and a client trying tokens is slowed down.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends, Request
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from app import apitokens, audit
|
||||
from app.apitokens import files_caller
|
||||
from app.auth import current_user
|
||||
from app.downloads import folders, service
|
||||
from app.downloads import gc as gcmod
|
||||
from app.downloads import upload as uploadmod
|
||||
from app.downloads.status import JobStatus
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal, get_db
|
||||
from app.main import app
|
||||
from app.notifications import create_notification
|
||||
from app.models import (
|
||||
ApiToken,
|
||||
ApiTokenGrant,
|
||||
AuditLog,
|
||||
DownloadJob,
|
||||
FileFolder,
|
||||
MediaAsset,
|
||||
Notification,
|
||||
User,
|
||||
)
|
||||
from app.ratelimit import RateLimiter
|
||||
from app.routes import tokens as tokens_routes
|
||||
from app.routes.auth import bump_session_epoch
|
||||
from app.security import hash_token
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _no_probe(monkeypatch):
|
||||
# A file ffprobe cannot read: a generic file, done at once (no conversion).
|
||||
monkeypatch.setattr(uploadmod.probe, "probe_json", lambda _p: None)
|
||||
|
||||
|
||||
def _own(db, user_id, name):
|
||||
folder = FileFolder(user_id=user_id, kind=folders.FolderKind.OWN, name=name)
|
||||
db.add(folder)
|
||||
db.commit()
|
||||
return folder
|
||||
|
||||
|
||||
def _file_in(db, root, user_id, folder, name):
|
||||
"""A done file of `user_id`'s with bytes on disk, sitting in `folder`."""
|
||||
asset = MediaAsset(
|
||||
source_kind="upload",
|
||||
source_ref=name,
|
||||
format_sig=f"sig-{name}",
|
||||
status="ready",
|
||||
rel_path=f"{name}.bin",
|
||||
size_bytes=4,
|
||||
ref_count=0,
|
||||
title=name,
|
||||
media_kind="file",
|
||||
container="bin",
|
||||
)
|
||||
db.add(asset)
|
||||
db.flush()
|
||||
job = DownloadJob(
|
||||
user_id=user_id,
|
||||
asset_id=asset.id,
|
||||
source_kind="upload",
|
||||
source_ref=name,
|
||||
profile_snapshot={},
|
||||
status=JobStatus.DONE,
|
||||
)
|
||||
db.add(job)
|
||||
db.flush()
|
||||
folders.place(db, job)
|
||||
job.folder_id = folder.id
|
||||
folders.stamp_done(db, job)
|
||||
service.incr_ref_count(db, asset.id)
|
||||
db.commit()
|
||||
(root / f"{name}.bin").write_bytes(b"data")
|
||||
return job
|
||||
|
||||
|
||||
def _bearer(raw: str) -> TestClient:
|
||||
return TestClient(app, headers={"authorization": f"Bearer {raw}"})
|
||||
|
||||
|
||||
def _make(c, **body) -> dict:
|
||||
r = c.post("/api/tokens", json={"name": "a machine", **body})
|
||||
assert r.status_code == 200, r.text
|
||||
return r.json()
|
||||
|
||||
|
||||
def _audits(db, action):
|
||||
return db.execute(select(AuditLog).where(AuditLog.action == action)).scalars().all()
|
||||
|
||||
|
||||
# --- the token itself ----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_token_has_its_shape_and_a_checksum_that_refuses_a_typo():
|
||||
raw = apitokens.mint()
|
||||
assert raw.startswith("sfl_") and len(raw) == apitokens.TOKEN_LEN
|
||||
assert apitokens.well_formed(raw)
|
||||
assert apitokens.mint() != raw
|
||||
flipped = raw[:10] + ("A" if raw[10] != "A" else "B") + raw[11:]
|
||||
assert not apitokens.well_formed(flipped)
|
||||
assert not apitokens.well_formed(raw[:-1])
|
||||
assert not apitokens.well_formed("xxx_" + raw[4:])
|
||||
# A header byte past ASCII arrives as one latin-1 character: refused, never a 500.
|
||||
assert not apitokens.well_formed(raw[:-1] + "é")
|
||||
|
||||
|
||||
def test_a_non_ascii_token_is_refused_as_any_bad_one(db, client):
|
||||
# The test client re-sends a high byte as two UTF-8 bytes, which arrive as two latin-1
|
||||
# characters: one "A" fewer keeps the header a token's length, so it is the checksum that meets
|
||||
# them, not the length check.
|
||||
raw = ("Bearer sfl_" + "A" * 47 + "é").encode("latin-1")
|
||||
r = TestClient(app, headers={"authorization": raw}).get("/api/files/folders")
|
||||
assert r.status_code == 401 and r.json()["detail"]["reason"] == "not_authenticated"
|
||||
|
||||
|
||||
def test_the_hub_preset_makes_one_hub_folder_with_every_right_and_shows_the_token_once(
|
||||
db, make_user, as_user
|
||||
):
|
||||
alice = make_user()
|
||||
c = as_user(alice)
|
||||
made = _make(c, preset="hub")
|
||||
raw = made["token"]
|
||||
assert apitokens.well_formed(raw) and made["prefix"] == raw[: apitokens.SHOWN_LEN]
|
||||
[grant] = made["grants"]
|
||||
assert grant["kind"] == "hub" and grant["rights"] == ["delete", "list", "read", "write"]
|
||||
# Only the hash is kept, and the value is not in the list.
|
||||
stored = db.execute(select(ApiToken)).scalar_one()
|
||||
assert stored.token_hash != raw and raw not in str(c.get("/api/tokens").json())
|
||||
# A second hub token reuses the one Hub folder.
|
||||
again = _make(c, preset="hub")
|
||||
assert again["grants"][0]["folder_id"] == grant["folder_id"]
|
||||
hubs = db.execute(select(FileFolder).where(FileFolder.kind == "hub")).scalars().all()
|
||||
assert len(hubs) == 1
|
||||
assert len(_audits(db, "api_token_create")) == 2
|
||||
|
||||
|
||||
def test_a_token_is_granted_only_the_users_own_folders_and_writes_only_where_uploads_land(
|
||||
db, make_user, as_user
|
||||
):
|
||||
alice, bob = make_user(), make_user()
|
||||
theirs = _own(db, bob.id, "Bob's")
|
||||
c = as_user(alice)
|
||||
r = c.post(
|
||||
"/api/tokens",
|
||||
json={"name": "x", "grants": [{"folder_id": theirs.id, "rights": ["read"]}]},
|
||||
)
|
||||
assert r.status_code == 409 and r.json()["detail"]["reason"] == "file_folder_gone"
|
||||
youtube = folders.source_folder(db, alice.id, folders.SourceKey.YOUTUBE)
|
||||
db.commit()
|
||||
r = c.post(
|
||||
"/api/tokens",
|
||||
json={"name": "x", "grants": [{"folder_id": youtube.id, "rights": ["write"]}]},
|
||||
)
|
||||
assert r.json()["detail"]["reason"] == "file_folder_not_its_source"
|
||||
assert c.post("/api/tokens", json={"name": "x"}).json()["detail"]["reason"] == (
|
||||
"api_token_no_grants"
|
||||
)
|
||||
assert db.execute(select(ApiToken)).all() == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("right", ["read", "write"])
|
||||
def test_a_folder_granted_is_held_until_the_token_is_in(db, make_user, right):
|
||||
"""A delete of the folder at the same moment waits and takes the new grant with it, rather than
|
||||
failing the grant's foreign key."""
|
||||
import threading
|
||||
|
||||
alice = make_user()
|
||||
fid = _own(db, alice.id, "Mine").id
|
||||
body = tokens_routes.TokenIn(
|
||||
name="x", grants=[tokens_routes.GrantIn(folder_id=fid, rights=[right])]
|
||||
)
|
||||
tokens_routes._wanted_grants(db, alice, body)
|
||||
outcome: list[object] = []
|
||||
|
||||
def delete_it():
|
||||
other = SessionLocal()
|
||||
try:
|
||||
outcome.append(folders.delete_own(other, other.get(FileFolder, fid)))
|
||||
other.commit()
|
||||
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
|
||||
outcome.append(e)
|
||||
finally:
|
||||
other.close()
|
||||
|
||||
t = threading.Thread(target=delete_it)
|
||||
t.start()
|
||||
t.join(timeout=1)
|
||||
assert t.is_alive() # the delete waits for the token on its way in
|
||||
db.commit()
|
||||
t.join(timeout=10)
|
||||
assert not isinstance(outcome[0], Exception), outcome
|
||||
|
||||
|
||||
def test_a_user_has_at_most_the_ceiling_of_tokens(db, make_user, as_user, monkeypatch):
|
||||
monkeypatch.setattr(tokens_routes, "MAX_TOKENS", 2)
|
||||
c = as_user(make_user())
|
||||
_make(c, preset="hub")
|
||||
_make(c, preset="hub")
|
||||
r = c.post("/api/tokens", json={"name": "third", "preset": "hub"})
|
||||
assert r.status_code == 409 and r.json()["detail"]["reason"] == "api_token_limit"
|
||||
|
||||
|
||||
# --- what a token reaches ------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("right", ["list", "read", "write", "delete"])
|
||||
def test_each_right_holds_in_its_folder_only(
|
||||
db, make_user, as_user, _root, _no_probe, upload_headers, right
|
||||
):
|
||||
"""One right in folder A; folder B has none. Every Files route a token may use, in both."""
|
||||
alice = make_user()
|
||||
a, b = _own(db, alice.id, "A"), _own(db, alice.id, "B")
|
||||
in_a, in_b = _file_in(db, _root, alice.id, a, "fa"), _file_in(db, _root, alice.id, b, "fb")
|
||||
raw = _make(as_user(alice), grants=[{"folder_id": a.id, "rights": [right]}])["token"]
|
||||
m = _bearer(raw)
|
||||
|
||||
folders_seen = [f["id"] for f in m.get("/api/files/folders").json()]
|
||||
assert folders_seen == ([a.id] if right == "list" else [])
|
||||
listed = m.get("/api/files", params={"folder_id": a.id})
|
||||
assert listed.status_code == (200 if right == "list" else 404)
|
||||
if right == "list":
|
||||
assert [f["id"] for f in listed.json()] == [in_a.id]
|
||||
assert [f["id"] for f in m.get("/api/files").json()] == [in_a.id]
|
||||
else:
|
||||
assert m.get("/api/files").json() == []
|
||||
assert m.get("/api/files", params={"folder_id": b.id}).status_code == 404
|
||||
|
||||
read_a = m.get(f"/api/files/{in_a.id}/file")
|
||||
assert read_a.status_code == (200 if right == "read" else 404)
|
||||
assert m.get(f"/api/files/{in_a.id}/sha256").status_code == (200 if right == "read" else 404)
|
||||
assert m.get(f"/api/files/{in_b.id}/file").status_code == 404
|
||||
|
||||
up_a = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt", folder=a.id))
|
||||
assert up_a.status_code == (200 if right == "write" else 404)
|
||||
if right == "write":
|
||||
assert up_a.json()["folder_id"] == a.id
|
||||
up_b = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt", folder=b.id))
|
||||
assert up_b.status_code == 404
|
||||
no_folder = m.post("/api/files/upload", content=b"hi", headers=upload_headers("n.txt"))
|
||||
assert no_folder.status_code == 400
|
||||
assert no_folder.json()["detail"]["reason"] == "api_token_folder_required"
|
||||
|
||||
assert m.delete(f"/api/files/{in_b.id}").status_code == 404
|
||||
assert db.get(DownloadJob, in_b.id) is not None
|
||||
del_a = m.delete(f"/api/files/{in_a.id}")
|
||||
assert del_a.status_code == (200 if right == "delete" else 404)
|
||||
db.expire_all()
|
||||
assert (db.get(DownloadJob, in_a.id) is None) == (right == "delete")
|
||||
|
||||
|
||||
def test_a_token_never_reaches_another_users_files_or_the_shared_ones(
|
||||
db, make_user, as_user, _root
|
||||
):
|
||||
alice, bob = make_user(), make_user()
|
||||
hub_raw = _make(as_user(alice), preset="hub")["token"]
|
||||
bobs = _file_in(db, _root, bob.id, _own(db, bob.id, "B"), "bf")
|
||||
m = _bearer(hub_raw)
|
||||
assert m.get(f"/api/files/{bobs.id}/file").status_code == 404
|
||||
assert m.delete(f"/api/files/{bobs.id}").status_code == 404
|
||||
shared = m.get("/api/files", params={"scope": "shared"})
|
||||
assert shared.status_code == 403 and shared.json()["detail"]["reason"] == "api_token_scope"
|
||||
|
||||
|
||||
def test_a_token_is_judged_by_its_grants_even_beside_a_signed_in_cookie(db, make_user, client):
|
||||
"""Through the real session stack (`client`, a real sign-in): the `as_user` fixture stands in
|
||||
for the very dependency this is about."""
|
||||
alice = make_user(email="alice@example.com")
|
||||
a = _own(db, alice.id, "A")
|
||||
c = client
|
||||
signed_in = c.post(
|
||||
"/auth/password-login",
|
||||
json={"email": "alice@example.com", "password": "correct horse battery"},
|
||||
)
|
||||
assert signed_in.status_code == 200
|
||||
raw = _make(c, grants=[{"folder_id": a.id, "rights": ["read"]}])["token"]
|
||||
assert len(c.get("/api/files/folders").json()) == 1 # the user sees every folder
|
||||
both = c.get("/api/files/folders", headers={"authorization": f"Bearer {raw}"})
|
||||
assert both.json() == [] # the token may not list: its grants decide
|
||||
|
||||
|
||||
def _signed_in(client, email="alice@example.com") -> TestClient:
|
||||
r = client.post(
|
||||
"/auth/password-login", json={"email": email, "password": "correct horse battery"}
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
return client
|
||||
|
||||
|
||||
def test_a_token_works_on_the_files_routes_only(db, make_user, client):
|
||||
"""Through the real session stack: under `as_user` every route without its header answers 401
|
||||
whatever `current_user` does with a token, so these refusals would prove nothing there."""
|
||||
make_user(email="alice@example.com")
|
||||
raw = _make(_signed_in(client), preset="hub")["token"]
|
||||
m = _bearer(raw)
|
||||
assert m.get("/api/files/folders").status_code == 200
|
||||
# Making a wider token, or reading the account, takes the user's session.
|
||||
assert m.get("/api/me").json() == {"authenticated": False} # the token is no one here
|
||||
assert m.get("/api/tokens").status_code == 401
|
||||
assert m.post("/api/tokens", json={"name": "wider", "preset": "hub"}).status_code == 401
|
||||
assert m.get("/api/files/usage").status_code == 401
|
||||
assert m.patch("/api/files/1", json={"name": "x"}).status_code == 401
|
||||
|
||||
|
||||
def test_only_the_files_routes_take_a_token():
|
||||
"""The route table, not a list kept by hand: every route that resolves `files_caller` is under
|
||||
/api/files, and these are all of them. The walk recurses into included routers (see
|
||||
`test_body_limit._declared_routes` for why `app.routes` alone sees none of ours)."""
|
||||
takers, nodes, routes = set(), [app], []
|
||||
while nodes:
|
||||
for route in getattr(nodes.pop(), "routes", ()) or ():
|
||||
inner = getattr(route, "original_router", None)
|
||||
if inner is not None:
|
||||
nodes.append(inner)
|
||||
else:
|
||||
routes.append(route)
|
||||
assert len(routes) > 100 # the walk found the app's routes, not just the shell's
|
||||
for route in routes:
|
||||
dependant = getattr(route, "dependant", None)
|
||||
if dependant is None:
|
||||
continue
|
||||
stack, calls = [dependant], set()
|
||||
while stack:
|
||||
d = stack.pop()
|
||||
calls.add(d.call)
|
||||
stack.extend(d.dependencies)
|
||||
if files_caller in calls:
|
||||
for method in route.methods:
|
||||
takers.add((method, route.path))
|
||||
assert all(path.startswith("/api/files") for _, path in takers)
|
||||
assert takers == {
|
||||
("GET", "/api/files/folders"),
|
||||
("GET", "/api/files"),
|
||||
("GET", "/api/files/{job_id}/file"),
|
||||
("GET", "/api/files/{job_id}/sha256"),
|
||||
("POST", "/api/files/upload"),
|
||||
("DELETE", "/api/files/{job_id}"),
|
||||
}
|
||||
|
||||
|
||||
def _reset_password(db, client, alice):
|
||||
from app.routes.auth import _issue_token
|
||||
|
||||
link = _issue_token(db, alice, "reset", timedelta(hours=1))
|
||||
r = client.post(
|
||||
"/auth/password-reset/confirm", json={"token": link, "password": "a brand new password"}
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
|
||||
|
||||
def _change_password(db, client, alice):
|
||||
r = client.post(
|
||||
"/auth/set-password",
|
||||
json={"password": "a brand new password", "current_password": "correct horse battery"},
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
|
||||
|
||||
def _log_out_others(db, client, alice):
|
||||
assert client.post("/auth/logout-others").status_code == 200
|
||||
|
||||
|
||||
def _revoke_untrusted_password(db, client, alice):
|
||||
from app.routes.auth import _revoke_password
|
||||
|
||||
_revoke_password(db, alice)
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"end_sessions",
|
||||
[_reset_password, _change_password, _log_out_others, _revoke_untrusted_password],
|
||||
)
|
||||
def test_whatever_ends_every_session_ends_every_token_too(db, make_user, client, end_sessions):
|
||||
"""A token minted from a stolen session must not outlive the owner's answer to the theft."""
|
||||
alice = make_user(email="alice@example.com")
|
||||
bob = make_user(email="bob@example.com")
|
||||
raw = _make(_signed_in(client), preset="hub")["token"]
|
||||
bobs = apitokens.mint()
|
||||
db.add(ApiToken(user_id=bob.id, name="b", prefix=bobs[:12], token_hash=hash_token(bobs)))
|
||||
db.commit()
|
||||
assert _bearer(raw).get("/api/files/folders").status_code == 200
|
||||
end_sessions(db, client, alice)
|
||||
assert _bearer(raw).get("/api/files/folders").status_code == 401
|
||||
assert db.execute(select(ApiToken.user_id)).scalars().all() == [bob.id] # only hers went
|
||||
[revoked] = _audits(db, "api_token_revoke")
|
||||
assert revoked.actor_id == alice.id and "every session" in (revoked.summary or "")
|
||||
|
||||
|
||||
def test_a_token_asked_for_as_every_session_ends_is_refused(db, make_user, client, override_dep):
|
||||
"""The session was still good when its request came in, and ended before the token was made."""
|
||||
alice = make_user(email="alice@example.com")
|
||||
c = _signed_in(client)
|
||||
|
||||
def ended_meanwhile(request: Request, session=Depends(get_db)):
|
||||
user = current_user(request, session)
|
||||
other = SessionLocal()
|
||||
try:
|
||||
bump_session_epoch(other, other.get(User, alice.id))
|
||||
other.commit()
|
||||
finally:
|
||||
other.close()
|
||||
return user
|
||||
|
||||
override_dep(current_user, ended_meanwhile)
|
||||
r = c.post("/api/tokens", json={"name": "late", "preset": "hub"})
|
||||
assert r.status_code == 401
|
||||
assert db.execute(select(ApiToken.id)).scalars().all() == []
|
||||
|
||||
|
||||
def _meanwhile(work):
|
||||
"""Run `work(other_session)` in another thread and session, committed; (thread, outcome)."""
|
||||
import threading
|
||||
|
||||
outcome: list[object] = []
|
||||
|
||||
def run():
|
||||
other = SessionLocal()
|
||||
try:
|
||||
outcome.append(work(other))
|
||||
other.commit()
|
||||
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
|
||||
outcome.append(e)
|
||||
finally:
|
||||
other.close()
|
||||
|
||||
t = threading.Thread(target=run)
|
||||
t.start()
|
||||
return t, outcome
|
||||
|
||||
|
||||
def test_ending_every_session_never_deadlocks_with_a_tokens_expiry_notice(db, make_user):
|
||||
"""The notice's sweep holds the token's row and then writes a row naming the user; the sessions'
|
||||
end holds the user and then deletes the token. A lock on the user that a mere reference to it
|
||||
waits on would make that a deadlock — one of the two failing."""
|
||||
alice = make_user()
|
||||
raw = apitokens.mint()
|
||||
soon = datetime.now(timezone.utc) + timedelta(days=1)
|
||||
tok = ApiToken(
|
||||
user_id=alice.id, name="t", prefix=raw[:12], token_hash=hash_token(raw), expires_at=soon
|
||||
)
|
||||
db.add(tok)
|
||||
db.commit()
|
||||
# `warn_expiring`, its two halves held apart: the token row taken…
|
||||
db.execute(update(ApiToken).where(ApiToken.id == tok.id).values(expiry_warned=True))
|
||||
t, outcome = _meanwhile(lambda other: bump_session_epoch(other, other.get(User, alice.id)))
|
||||
t.join(timeout=1)
|
||||
assert t.is_alive() # the sessions' end waits for the token row
|
||||
# …and then its notice, naming the user.
|
||||
create_notification(db, alice.id, "api_token_expiring", title="t", commit=False)
|
||||
db.commit()
|
||||
t.join(timeout=10)
|
||||
assert outcome == [None]
|
||||
db.expire_all()
|
||||
assert db.execute(select(ApiToken.id)).scalars().all() == []
|
||||
|
||||
|
||||
def test_a_token_granted_a_folder_being_deleted_never_deadlocks(db, make_user):
|
||||
"""The delete holds the folder and then writes its audit line naming the user; the token holds
|
||||
the user and then waits for the folder."""
|
||||
alice = make_user()
|
||||
fid = _own(db, alice.id, "Mine").id
|
||||
folders.delete_own(db, db.get(FileFolder, fid)) # the folder held, as the delete route does
|
||||
body = tokens_routes.TokenIn(
|
||||
name="x", grants=[tokens_routes.GrantIn(folder_id=fid, rights=["read"])]
|
||||
)
|
||||
t, outcome = _meanwhile(
|
||||
lambda other: tokens_routes.create_token(body, other.get(User, alice.id), other)
|
||||
)
|
||||
t.join(timeout=1)
|
||||
assert t.is_alive() # the token waits for the folder
|
||||
audit.record(db, alice.id, "file_folder_delete", target_type="file_folder", target_id=fid)
|
||||
db.commit()
|
||||
t.join(timeout=10)
|
||||
[refusal] = outcome
|
||||
assert getattr(refusal, "status_code", None) == 409, refusal # the folder went first
|
||||
|
||||
|
||||
def test_a_token_made_as_every_session_ends_goes_with_them(db, make_user):
|
||||
"""The other order: the token is on its way in when the sessions end."""
|
||||
import threading
|
||||
|
||||
alice = make_user(email="alice@example.com")
|
||||
db.execute(select(User.id).where(User.id == alice.id).with_for_update()) # as create_token
|
||||
raw = apitokens.mint()
|
||||
db.add(ApiToken(user_id=alice.id, name="t", prefix=raw[:12], token_hash=hash_token(raw)))
|
||||
db.flush()
|
||||
outcome: list[object] = []
|
||||
|
||||
def end_sessions():
|
||||
other = SessionLocal()
|
||||
try:
|
||||
bump_session_epoch(other, other.get(User, alice.id))
|
||||
other.commit()
|
||||
outcome.append("done")
|
||||
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
|
||||
outcome.append(e)
|
||||
finally:
|
||||
other.close()
|
||||
|
||||
t = threading.Thread(target=end_sessions)
|
||||
t.start()
|
||||
t.join(timeout=1)
|
||||
assert t.is_alive() # it waits for the token being made
|
||||
db.commit()
|
||||
t.join(timeout=10)
|
||||
assert outcome == ["done"]
|
||||
db.expire_all()
|
||||
assert db.execute(select(ApiToken.id)).scalars().all() == []
|
||||
|
||||
|
||||
# --- a token that does not hold ------------------------------------------------------------------
|
||||
|
||||
|
||||
def _refused(m: TestClient) -> int:
|
||||
return m.get("/api/files/folders").status_code
|
||||
|
||||
|
||||
def test_a_malformed_unknown_expired_revoked_or_suspended_token_holds_nothing(
|
||||
db, make_user, as_user
|
||||
):
|
||||
alice = make_user()
|
||||
c = as_user(alice)
|
||||
raw = _make(c, preset="hub")["token"]
|
||||
assert _refused(_bearer(raw)) == 200
|
||||
assert _refused(_bearer("sfl_short")) == 401
|
||||
assert _refused(_bearer(apitokens.mint())) == 401 # well formed, never made
|
||||
|
||||
expiring = _make(c, preset="hub", expires_days=1)
|
||||
token = db.get(ApiToken, expiring["id"])
|
||||
token.expires_at = datetime.now(timezone.utc) - timedelta(seconds=1)
|
||||
db.commit()
|
||||
assert _refused(_bearer(expiring["token"])) == 401
|
||||
assert c.get("/api/tokens").json()[1]["expired"] is True
|
||||
|
||||
assert c.delete(f"/api/tokens/{expiring['id']}").json() == {
|
||||
"ok": True,
|
||||
"deleted": expiring["id"],
|
||||
}
|
||||
assert (
|
||||
db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == expiring["id"])).all()
|
||||
== []
|
||||
)
|
||||
assert len(_audits(db, "api_token_revoke")) == 1
|
||||
revoked = _make(c, preset="hub")
|
||||
assert c.delete(f"/api/tokens/{revoked['id']}").status_code == 200
|
||||
assert _refused(_bearer(revoked["token"])) == 401
|
||||
|
||||
alice.is_suspended = True
|
||||
db.commit()
|
||||
assert _refused(_bearer(raw)) == 401
|
||||
|
||||
|
||||
def test_a_token_about_to_run_out_warns_its_user_once_and_is_marked_in_the_list(
|
||||
db, make_user, as_user
|
||||
):
|
||||
alice, bob = make_user(), make_user()
|
||||
c = as_user(alice)
|
||||
soon = _make(c, preset="hub", expires_days=3)
|
||||
_make(c, preset="hub", expires_days=30) # outside the span
|
||||
_make(c, preset="hub") # never runs out
|
||||
gone = _make(c, preset="hub", expires_days=1)
|
||||
db.get(ApiToken, gone["id"]).expires_at = datetime.now(timezone.utc) - timedelta(seconds=1)
|
||||
_make(as_user(bob), preset="hub", expires_days=2)
|
||||
db.commit()
|
||||
assert [t["expiring"] for t in c.get("/api/tokens").json()] == [True, False, False, False]
|
||||
|
||||
assert apitokens.warn_expiring(db) == 2 # alice's and bob's; the run-out one is not "soon"
|
||||
notes = db.execute(select(Notification).where(Notification.user_id == alice.id)).scalars()
|
||||
[note] = notes.all()
|
||||
assert note.type == "api_token_expiring"
|
||||
assert note.data["token_id"] == soon["id"] and note.data["name"] == "a machine"
|
||||
assert note.data["expires_at"] == soon["expires_at"]
|
||||
assert apitokens.warn_expiring(db) == 0 # once per token
|
||||
|
||||
|
||||
def test_the_files_gc_runs_the_token_warning(db, make_user, as_user, _root):
|
||||
_make(as_user(make_user()), preset="hub", expires_days=3)
|
||||
assert gcmod.run_download_gc(db)["tokens_warned"] == 1
|
||||
|
||||
|
||||
def test_another_users_token_cannot_be_revoked(db, make_user, as_user):
|
||||
alice, bob = make_user(), make_user()
|
||||
made = _make(as_user(alice), preset="hub")
|
||||
assert as_user(bob).delete(f"/api/tokens/{made['id']}").status_code == 404
|
||||
assert db.get(ApiToken, made["id"]) is not None
|
||||
|
||||
|
||||
def test_a_client_trying_tokens_is_slowed_down(db, make_user, as_user, monkeypatch):
|
||||
alice = make_user()
|
||||
raw = _make(as_user(alice), preset="hub")["token"]
|
||||
monkeypatch.setattr(apitokens, "_bad", RateLimiter(3, 300))
|
||||
for _ in range(3):
|
||||
assert _refused(_bearer(apitokens.mint())) == 401
|
||||
slowed = _bearer(apitokens.mint()).get("/api/files/folders")
|
||||
assert slowed.status_code == 429
|
||||
assert slowed.json()["detail"]["reason"] == "api_token_rate_limited"
|
||||
# Even a good token from that client waits out the window: a guesser cannot tell a hit.
|
||||
assert _refused(_bearer(raw)) == 429
|
||||
|
||||
|
||||
def test_a_runaway_machine_is_held_back_per_token(db, make_user, as_user, monkeypatch):
|
||||
c = as_user(make_user())
|
||||
first, second = _make(c, preset="hub")["token"], _make(c, preset="hub")["token"]
|
||||
monkeypatch.setattr(apitokens, "_calls", RateLimiter(2, 60))
|
||||
assert [_refused(_bearer(first)) for _ in range(3)] == [200, 200, 429]
|
||||
assert _refused(_bearer(second)) == 200
|
||||
|
||||
|
||||
# --- what a token leaves behind ------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_tokens_use_is_stamped_and_its_uploads_and_deletes_name_it_in_the_audit(
|
||||
db, make_user, as_user, _root, _no_probe, upload_headers
|
||||
):
|
||||
alice = make_user()
|
||||
made = _make(as_user(alice), preset="hub")
|
||||
hub = made["grants"][0]["folder_id"]
|
||||
assert db.get(ApiToken, made["id"]).last_used_at is None
|
||||
m = _bearer(made["token"])
|
||||
up = m.post("/api/files/upload", content=b"hi", headers=upload_headers("h.txt", folder=hub))
|
||||
assert up.status_code == 200
|
||||
db.expire_all()
|
||||
assert db.get(ApiToken, made["id"]).last_used_at is not None
|
||||
assert m.delete(f"/api/files/{up.json()['id']}").status_code == 200
|
||||
via = f"api token {made['prefix']}…"
|
||||
assert [a.summary for a in _audits(db, "file_upload")] == [via]
|
||||
assert [a.summary for a in _audits(db, "file_delete")] == [via]
|
||||
@@ -1,6 +1,6 @@
|
||||
"""An admin action that counts something keeps the count as data in its audit row's `after`, under
|
||||
the key the audit log names in the reader's language (`scheduler.result.<key>`, frontend
|
||||
auditColumns.tsx) — not only in the English `summary`. A renamed key would fall back to the raw word.
|
||||
lib/jobResult.ts, called from lib/auditText.ts) — not only in the English `summary`. A renamed key would fall back to the raw word.
|
||||
DB lane (Postgres).
|
||||
"""
|
||||
|
||||
|
||||
@@ -493,7 +493,7 @@ def _revoke_as_google_adoption(db, user) -> None:
|
||||
Google callback adopting an account whose email was never verified."""
|
||||
from app.routes.auth import revoke_untrusted_preset_password
|
||||
|
||||
revoke_untrusted_preset_password(user)
|
||||
revoke_untrusted_preset_password(db, user)
|
||||
db.commit()
|
||||
|
||||
|
||||
|
||||
@@ -107,25 +107,48 @@ def _setup_channel_and_sub(db, *, email):
|
||||
|
||||
|
||||
def _patch_deep_requested(db, monkeypatch, override_dep, user):
|
||||
submitted: list[tuple] = []
|
||||
calls: list[tuple] = []
|
||||
# background.add_task runs after the response in TestClient; record what would be scheduled.
|
||||
# Record the run the route submits, and what its work would call, instead of running it.
|
||||
monkeypatch.setattr(
|
||||
channels_route.scheduler,
|
||||
"run_ad_hoc",
|
||||
lambda job, fn, **kw: submitted.append((job, kw)) or fn(),
|
||||
)
|
||||
monkeypatch.setattr(channels_route, "backfill_channel_now", lambda *a, **k: calls.append(a))
|
||||
monkeypatch.setattr(state, "_configured_cache", True)
|
||||
override_dep(get_db, db.info["request"])
|
||||
override_dep(current_user, lambda: user)
|
||||
resp = TestClient(app).patch("/api/channels/UC_wire", json={"deep_requested": True})
|
||||
return resp, calls
|
||||
return resp, submitted, calls
|
||||
|
||||
|
||||
def test_deep_request_schedules_immediate_backfill(db, monkeypatch, override_dep):
|
||||
def test_deep_request_starts_the_backfill_as_the_users_run_of_that_channel(
|
||||
db, monkeypatch, override_dep
|
||||
):
|
||||
"""It was a BackgroundTask: what it did, spent or failed with reached only the log."""
|
||||
ch, me = _setup_channel_and_sub(db, email="wire@example.com")
|
||||
resp, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
|
||||
resp, submitted, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json()["deep_requested"] is True
|
||||
# scheduled exactly once, with (channel_id, user_id) — the args backfill_channel_now needs.
|
||||
assert submitted == [("channel_backfill", {"actor_id": me.id, "target": ch.id})]
|
||||
# its work runs exactly once, with (channel_id, user_id) — the args backfill_channel_now needs.
|
||||
assert calls == [(ch.id, me.id)]
|
||||
|
||||
|
||||
def test_turning_full_history_on_for_a_finished_channel_records_no_run(
|
||||
db, monkeypatch, override_dep
|
||||
):
|
||||
"""Its whole history is in already: a run would only say "nothing to do", and switching the flag
|
||||
off and on again would otherwise write one such row each time, kept for months."""
|
||||
ch, me = _setup_channel_and_sub(db, email="done@example.com")
|
||||
ch.backfill_done = True
|
||||
db.commit()
|
||||
resp, submitted, calls = _patch_deep_requested(db, monkeypatch, override_dep, me)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert (submitted, calls) == ([], [])
|
||||
|
||||
|
||||
class TestImmediateBackfillGuard:
|
||||
"""backfill_channel_now coordinates with the paced scheduler through _backfill_in_progress (per-
|
||||
channel mutual exclusion, both directions) + _immediate_running (a cap on interactive runs so a
|
||||
@@ -148,15 +171,49 @@ class TestImmediateBackfillGuard:
|
||||
def test_a_second_call_for_an_in_flight_channel_is_a_noop(self, monkeypatch):
|
||||
runner._backfill_in_progress.add("UC_busy")
|
||||
monkeypatch.setattr(runner, "SessionLocal", self._boom_session)
|
||||
runner.backfill_channel_now("UC_busy", 1) # returns immediately; _boom_session never runs
|
||||
# returns at once — _boom_session never runs — and the run says why it did nothing.
|
||||
assert runner.backfill_channel_now("UC_busy", 1) == {"skipped": "channel busy"}
|
||||
assert runner._backfill_in_progress == {"UC_busy"} # the in-flight entry is untouched
|
||||
|
||||
def test_over_the_concurrency_cap_falls_back_to_the_scheduler(self, monkeypatch):
|
||||
runner._immediate_running = runner._MAX_CONCURRENT_IMMEDIATE
|
||||
monkeypatch.setattr(runner, "SessionLocal", self._boom_session)
|
||||
runner.backfill_channel_now("UC_new", 1) # over the cap → no-op, scheduler will handle it
|
||||
# over the cap → no-op, the scheduler will handle it
|
||||
assert runner.backfill_channel_now("UC_new", 1) == {"skipped": "at capacity"}
|
||||
assert "UC_new" not in runner._backfill_in_progress
|
||||
|
||||
@staticmethod
|
||||
def _one_channel(db, monkeypatch, cid):
|
||||
ch = Channel(id=cid, title=cid, uploads_playlist_id="UU" + cid, backfill_done=False)
|
||||
me = User(email=f"{cid}@example.com")
|
||||
db.add_all([ch, me])
|
||||
db.commit()
|
||||
monkeypatch.setattr(runner, "get_service_user", lambda _db: me)
|
||||
monkeypatch.setattr(runner.quota, "backfill_reserve_reached", lambda: False)
|
||||
monkeypatch.setattr(runner, "YouTubeClient", lambda *a, **k: _NullYt())
|
||||
return ch, me
|
||||
|
||||
def test_a_failure_reaches_its_run_and_still_frees_the_channel(self, db, monkeypatch):
|
||||
"""The failure was logged and swallowed, so the run that now wraps it would read "ok"."""
|
||||
ch, me = self._one_channel(db, monkeypatch, "UC_fail")
|
||||
|
||||
def boom(*_a):
|
||||
raise RuntimeError("playlistItems 500")
|
||||
|
||||
monkeypatch.setattr(runner, "backfill_channel_deep", boom)
|
||||
with pytest.raises(RuntimeError, match="playlistItems 500"):
|
||||
runner.backfill_channel_now(ch.id, me.id)
|
||||
assert runner._backfill_in_progress == set() and runner._immediate_running == 0
|
||||
|
||||
def test_a_finished_backfill_says_what_it_added(self, db, monkeypatch):
|
||||
ch, me = self._one_channel(db, monkeypatch, "UC_ok")
|
||||
monkeypatch.setattr(runner, "backfill_channel_deep", lambda *_a: 7)
|
||||
monkeypatch.setattr(runner, "reconcile_full_history", lambda _db: 1)
|
||||
assert runner.backfill_channel_now(ch.id, me.id) == {
|
||||
"videos_added": 7,
|
||||
"reconciled_full": 1,
|
||||
}
|
||||
|
||||
def test_scheduler_deep_pass_skips_an_in_flight_channel(self, db, monkeypatch):
|
||||
# A channel the scheduler's query WOULD pick (deep-requested, recent-synced, not done)…
|
||||
ch = Channel(
|
||||
|
||||
@@ -15,6 +15,7 @@ import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.apitokens import FilesCaller
|
||||
from app.db import SessionLocal
|
||||
from app.downloads import service
|
||||
from app.models import DownloadJob, MediaAsset, User
|
||||
@@ -50,11 +51,16 @@ def _asset_held_by_two_jobs(db, status="error"):
|
||||
return mine, asset, jobs[0]
|
||||
|
||||
|
||||
def delete_file_as(job_id, *, user, db):
|
||||
"""`delete_file` as the user's own request (it takes the Files caller, not the user)."""
|
||||
return delete_file(job_id, caller=FilesCaller(user=user), db=db)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("status", "transition", "route", "left"),
|
||||
[
|
||||
("error", service.cancel, cancel_download, "canceled"),
|
||||
("error", service.delete, delete_file, None),
|
||||
("error", service.delete, delete_file_as, None),
|
||||
("queued", service.cancel, pause_download, "canceled"),
|
||||
],
|
||||
)
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
"""A `/watch` link to one of its owner's own folders (routes/downloads.py, routes/public.py), DB-lane.
|
||||
|
||||
The link shows what is in the folder when it is opened: a file moved in appears, one moved out stops
|
||||
opening at its next request. Only an own folder can be shared — a source folder lets its files go on
|
||||
its own clock, the Hub is the channel to a machine. A password, an expiry and stream-only work as on
|
||||
a file link; a document is never served to open as a page.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import settings
|
||||
from app.db import engine
|
||||
from app.downloads import folders, og
|
||||
from app.downloads import probe as probemod
|
||||
from app.models import AuditLog, DownloadJob, DownloadLink, FileFolder, MediaAsset
|
||||
|
||||
_VIDEO = {
|
||||
"streams": [
|
||||
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
|
||||
{"codec_type": "audio", "codec_name": "aac"},
|
||||
],
|
||||
"format": {"format_name": "mov,mp4", "duration": "5"},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def probe(monkeypatch):
|
||||
"""What ffprobe says the next upload is: a phone-safe video, or (None) a file it cannot read."""
|
||||
said = {"data": _VIDEO}
|
||||
monkeypatch.setattr(probemod, "probe_json", lambda _p: said["data"])
|
||||
return said
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def shelf(as_user, make_user, tmp_path, monkeypatch, upload_headers, probe):
|
||||
"""An owner with an own folder "Friends" holding two videos, a third video in their Uploads, and
|
||||
a function to upload more."""
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
user = make_user()
|
||||
owner = as_user(user)
|
||||
folder = owner.post("/api/files/folders", json={"name": "Friends"}).json()
|
||||
|
||||
def upload(name: str, folder_id: int | None = None, body: bytes = b"fake-video") -> int:
|
||||
r = owner.post(
|
||||
"/api/files/upload", content=body, headers=upload_headers(name, folder=folder_id)
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
return r.json()["id"]
|
||||
|
||||
inside = [upload("a.mp4", folder["id"]), upload("b.mp4", folder["id"])]
|
||||
outside = upload("c.mp4")
|
||||
return {
|
||||
"user": user,
|
||||
"owner": owner,
|
||||
"folder": folder["id"],
|
||||
"inside": inside,
|
||||
"outside": outside,
|
||||
"upload": upload,
|
||||
}
|
||||
|
||||
|
||||
def _link(shelf, **body) -> dict:
|
||||
r = shelf["owner"].post(f"/api/downloads/folders/{shelf['folder']}/links", json=body)
|
||||
assert r.status_code == 200, r.text
|
||||
return r.json()
|
||||
|
||||
|
||||
def _ids(client, token: str) -> list[int]:
|
||||
meta = client.get(f"/api/public/watch/{token}").json()
|
||||
return [f["id"] for f in meta["files"]]
|
||||
|
||||
|
||||
def test_a_folder_link_lists_what_is_in_the_folder(shelf, client):
|
||||
token = _link(shelf)["token"]
|
||||
meta = client.get(f"/api/public/watch/{token}").json()
|
||||
assert meta["kind"] == "folder"
|
||||
assert meta["name"] == "Friends"
|
||||
assert sorted(f["id"] for f in meta["files"]) == sorted(shelf["inside"])
|
||||
# Each one plays and downloads through the link, under its own id.
|
||||
for f in meta["files"]:
|
||||
assert f["file_url"] == f"/api/public/watch/{token}/files/{f['id']}/file"
|
||||
served = client.get(f["file_url"])
|
||||
assert served.status_code == 200
|
||||
assert served.content == b"fake-video"
|
||||
|
||||
|
||||
def test_a_file_outside_the_folder_does_not_open_through_it(shelf, client):
|
||||
token = _link(shelf)["token"]
|
||||
r = client.get(f"/api/public/watch/{token}/files/{shelf['outside']}/file")
|
||||
assert r.status_code == 404
|
||||
# Nor does the folder link open as a file link.
|
||||
assert client.get(f"/api/public/watch/{token}/file").status_code == 404
|
||||
|
||||
|
||||
def test_the_link_follows_the_folder_as_files_move(shelf, client):
|
||||
token = _link(shelf)["token"]
|
||||
owner, moved_out = shelf["owner"], shelf["inside"][0]
|
||||
other = owner.post("/api/files/folders", json={"name": "Other"}).json()["id"]
|
||||
|
||||
def move(job_id: int, folder_id: int) -> None:
|
||||
r = owner.patch(f"/api/files/{job_id}", json={"folder_id": folder_id})
|
||||
assert r.status_code == 200, r.text
|
||||
|
||||
move(shelf["outside"], shelf["folder"])
|
||||
move(moved_out, other)
|
||||
|
||||
assert sorted(_ids(client, token)) == sorted([shelf["inside"][1], shelf["outside"]])
|
||||
assert client.get(f"/api/public/watch/{token}/files/{moved_out}/file").status_code == 404
|
||||
url = f"/api/public/watch/{token}/files/{shelf['outside']}/file"
|
||||
assert client.get(url).status_code == 200
|
||||
|
||||
|
||||
def test_only_a_folder_of_ones_own_can_be_shared(shelf, db, as_user, make_user):
|
||||
owner = shelf["owner"]
|
||||
source = db.execute(
|
||||
select(FileFolder.id).where(
|
||||
FileFolder.user_id == shelf["user"].id, FileFolder.kind == "source"
|
||||
)
|
||||
).scalar_one()
|
||||
r = owner.post(f"/api/downloads/folders/{source}/links", json={})
|
||||
assert r.status_code == 409
|
||||
assert r.json()["detail"]["reason"] == "share_folder_kind"
|
||||
# Nor the Hub: whatever a machine drops there would be public the moment it lands.
|
||||
hub = folders.hub_folder(db, shelf["user"].id).id
|
||||
db.commit()
|
||||
r = owner.post(f"/api/downloads/folders/{hub}/links", json={})
|
||||
assert r.status_code == 409
|
||||
assert r.json()["detail"]["reason"] == "share_folder_kind"
|
||||
|
||||
stranger = as_user(make_user())
|
||||
assert (
|
||||
stranger.post(f"/api/downloads/folders/{shelf['folder']}/links", json={}).status_code == 404
|
||||
)
|
||||
assert stranger.get(f"/api/downloads/folders/{shelf['folder']}/links").status_code == 404
|
||||
link_id = _link(shelf)["id"]
|
||||
patch = stranger.patch(f"/api/downloads/links/{link_id}", json={"allow_download": False})
|
||||
assert patch.status_code == 404
|
||||
assert stranger.delete(f"/api/downloads/links/{link_id}").status_code == 404
|
||||
|
||||
|
||||
def test_a_password_folder_link_shows_nothing_until_unlocked(shelf, client):
|
||||
token = _link(shelf, password="open sesame")["token"]
|
||||
assert client.get(f"/api/public/watch/{token}").json() == {
|
||||
"needs_password": True,
|
||||
"kind": "folder",
|
||||
}
|
||||
bare = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/file"
|
||||
assert client.get(bare).status_code == 403
|
||||
|
||||
unlocked = client.post(f"/api/public/watch/{token}/unlock", json={"password": "open sesame"})
|
||||
assert unlocked.status_code == 200
|
||||
files = unlocked.json()["files"]
|
||||
assert len(files) == 2
|
||||
assert all("?g=" in f["file_url"] for f in files)
|
||||
assert client.get(files[0]["file_url"]).status_code == 200
|
||||
|
||||
|
||||
def test_a_files_poster_opens_by_the_links_own_rules(shelf, db, client):
|
||||
"""The poster route on its own: a password link's poster needs the unlock grant, and a file
|
||||
outside the folder has none through it, grant or not."""
|
||||
for job_id in (shelf["inside"][0], shelf["outside"]):
|
||||
asset = db.get(MediaAsset, db.get(DownloadJob, job_id).asset_id)
|
||||
asset.poster_path = f"poster-{job_id}.jpg"
|
||||
(Path(settings.files_root) / asset.poster_path).write_bytes(b"a poster frame")
|
||||
db.commit()
|
||||
token = _link(shelf, password="open sesame")["token"]
|
||||
inside = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/poster.jpg"
|
||||
assert client.get(inside).status_code == 403
|
||||
|
||||
unlocked = client.post(f"/api/public/watch/{token}/unlock", json={"password": "open sesame"})
|
||||
[entry] = [f for f in unlocked.json()["files"] if f["id"] == shelf["inside"][0]]
|
||||
assert entry["poster_url"].startswith(inside + "?g=")
|
||||
got = client.get(entry["poster_url"])
|
||||
assert got.status_code == 200 and got.content == b"a poster frame"
|
||||
grant = entry["poster_url"].split("?g=", 1)[1]
|
||||
outside = f"/api/public/watch/{token}/files/{shelf['outside']}/poster.jpg?g={grant}"
|
||||
assert client.get(outside).status_code == 404
|
||||
|
||||
|
||||
def test_every_open_counts_as_a_view(shelf, client):
|
||||
made = _link(shelf)
|
||||
for _ in range(3):
|
||||
assert client.get(f"/api/public/watch/{made['token']}").status_code == 200
|
||||
[listed] = shelf["owner"].get(f"/api/downloads/folders/{shelf['folder']}/links").json()
|
||||
assert listed["view_count"] == 3
|
||||
|
||||
|
||||
def test_the_folder_list_is_capped_like_the_owners_files_list():
|
||||
from app.downloads import links as linksmod
|
||||
from app.routes.files import MAX_LIST
|
||||
|
||||
assert linksmod.FOLDER_MAX == MAX_LIST
|
||||
|
||||
|
||||
def test_a_document_in_a_shared_folder_only_downloads(shelf, client, probe):
|
||||
probe["data"] = None
|
||||
doc = shelf["upload"]("notes.html", shelf["folder"], b"<script>1</script>")
|
||||
token = _link(shelf, allow_download=False)["token"]
|
||||
assert doc in _ids(client, token)
|
||||
served = client.get(f"/api/public/watch/{token}/files/{doc}/file")
|
||||
assert served.status_code == 200
|
||||
assert served.headers["content-type"].startswith("application/octet-stream")
|
||||
assert "attachment" in served.headers["content-disposition"].lower()
|
||||
|
||||
|
||||
def test_a_stream_only_folder_link_plays_inline(shelf, client):
|
||||
token = _link(shelf, allow_download=False)["token"]
|
||||
meta = client.get(f"/api/public/watch/{token}").json()
|
||||
assert meta["allow_download"] is False
|
||||
served = client.get(meta["files"][0]["file_url"])
|
||||
assert "inline" in served.headers["content-disposition"].lower()
|
||||
|
||||
|
||||
def test_an_expired_or_deleted_folder_link_is_gone(shelf, client, db):
|
||||
made = _link(shelf)
|
||||
db.get(DownloadLink, made["id"]).expires_at = datetime.now(timezone.utc) - timedelta(minutes=1)
|
||||
db.commit()
|
||||
assert client.get(f"/api/public/watch/{made['token']}").status_code == 404
|
||||
|
||||
token = _link(shelf)["token"]
|
||||
assert shelf["owner"].delete(f"/api/files/folders/{shelf['folder']}").status_code == 200
|
||||
assert client.get(f"/api/public/watch/{token}").status_code == 404
|
||||
db.expire_all()
|
||||
assert db.execute(select(DownloadLink)).scalars().all() == []
|
||||
|
||||
|
||||
def test_the_folder_list_marks_a_folder_a_live_link_opens(shelf, db):
|
||||
owner = shelf["owner"]
|
||||
|
||||
def marked() -> dict[int, bool]:
|
||||
return {f["id"]: f["has_link"] for f in owner.get("/api/files/folders").json()}
|
||||
|
||||
assert marked()[shelf["folder"]] is False
|
||||
made = _link(shelf)
|
||||
assert marked()[shelf["folder"]] is True
|
||||
# A rename answers with the mark too, so the rail does not lose it.
|
||||
renamed = owner.patch(f"/api/files/folders/{shelf['folder']}", json={"name": "Pals"}).json()
|
||||
assert renamed["has_link"] is True
|
||||
|
||||
db.get(DownloadLink, made["id"]).expires_at = datetime.now(timezone.utc) - timedelta(minutes=1)
|
||||
db.commit()
|
||||
assert marked()[shelf["folder"]] is False
|
||||
live = _link(shelf)
|
||||
assert owner.delete(f"/api/downloads/links/{live['id']}").status_code == 200
|
||||
assert marked()[shelf["folder"]] is False
|
||||
assert not any(marked()[f] for f in marked() if f != shelf["folder"])
|
||||
|
||||
|
||||
def test_making_and_revoking_a_folder_link_is_audited(shelf, db):
|
||||
made = _link(shelf)
|
||||
assert shelf["owner"].delete(f"/api/downloads/links/{made['id']}").status_code == 200
|
||||
db.expire_all()
|
||||
rows = {
|
||||
a.action: a.after
|
||||
for a in db.execute(
|
||||
select(AuditLog).where(AuditLog.action.like("download_link_%"))
|
||||
).scalars()
|
||||
}
|
||||
assert rows["download_link_create"]["folder_id"] == shelf["folder"]
|
||||
assert "job_id" not in rows["download_link_create"]
|
||||
assert rows["download_link_revoke"] == {"folder_id": shelf["folder"]}
|
||||
|
||||
|
||||
def test_the_unfurl_names_the_folder_and_counts_its_files_only(shelf, db):
|
||||
token = _link(shelf)["token"]
|
||||
tags = og._watch_tags(token, db)
|
||||
assert "Friends" in tags
|
||||
assert "2 files" in tags
|
||||
names = db.execute(
|
||||
select(DownloadJob.display_name).where(DownloadJob.id.in_(shelf["inside"]))
|
||||
).scalars()
|
||||
for name in names:
|
||||
assert not name or name not in tags
|
||||
|
||||
# A file whose bytes are gone is off the page, and so off the card's count.
|
||||
gone = db.get(MediaAsset, db.get(DownloadJob, shelf["inside"][0]).asset_id)
|
||||
(Path(settings.files_root) / gone.rel_path).unlink()
|
||||
assert "1 file " in og._watch_tags(token, db)
|
||||
|
||||
locked = _link(shelf, password="pw")["token"]
|
||||
hidden = og._watch_tags(locked, db)
|
||||
assert "Friends" not in hidden and "Password-protected folder" in hidden
|
||||
assert 'content="website"' in hidden and "video.other" not in hidden
|
||||
|
||||
|
||||
def test_the_downgrade_drops_folder_links_and_keeps_file_links(shelf, db, client):
|
||||
_link(shelf)
|
||||
file_link = shelf["owner"].post(f"/api/downloads/{shelf['outside']}/links", json={}).json()
|
||||
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0107_link_token_at_rest")
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
left = conn.execute(text("SELECT job_id FROM download_links")).scalars().all()
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
assert left == [shelf["outside"]]
|
||||
assert client.get(f"/api/public/watch/{file_link['token']}").status_code == 200
|
||||
|
||||
|
||||
def test_a_password_file_link_says_it_opens_a_file(shelf, client):
|
||||
made = (
|
||||
shelf["owner"]
|
||||
.post(f"/api/downloads/{shelf['outside']}/links", json={"password": "pw"})
|
||||
.json()
|
||||
)
|
||||
meta = client.get(f"/api/public/watch/{made['token']}").json()
|
||||
assert meta == {"needs_password": True, "kind": "file"}
|
||||
|
||||
|
||||
def test_a_file_whose_bytes_are_gone_is_not_offered(shelf, client, db):
|
||||
from pathlib import Path
|
||||
|
||||
from app.models import MediaAsset
|
||||
|
||||
token = _link(shelf)["token"]
|
||||
gone = shelf["inside"][0]
|
||||
asset = db.get(MediaAsset, db.get(DownloadJob, gone).asset_id)
|
||||
(Path(settings.files_root) / asset.rel_path).unlink()
|
||||
assert _ids(client, token) == [shelf["inside"][1]]
|
||||
assert client.get(f"/api/public/watch/{token}/files/{gone}/file").status_code == 404
|
||||
|
||||
|
||||
def _meanwhile(work):
|
||||
"""Run `work(other_session)` in another thread and session, committed; (thread, outcome)."""
|
||||
import threading
|
||||
|
||||
from app.db import SessionLocal
|
||||
|
||||
outcome: list[object] = []
|
||||
|
||||
def run():
|
||||
other = SessionLocal()
|
||||
try:
|
||||
outcome.append(work(other))
|
||||
other.commit()
|
||||
except Exception as e: # noqa: BLE001 — the assertion below names what went wrong
|
||||
outcome.append(e)
|
||||
finally:
|
||||
other.close()
|
||||
|
||||
t = threading.Thread(target=run)
|
||||
t.start()
|
||||
return t, outcome
|
||||
|
||||
|
||||
def test_two_opens_at_once_count_as_two_views(shelf, db):
|
||||
from app.routes import public
|
||||
|
||||
made = _link(shelf)
|
||||
link = db.get(DownloadLink, made["id"])
|
||||
public._opened(db, link, made["token"]) # this open's count is not committed yet
|
||||
t, outcome = _meanwhile(
|
||||
lambda other: public._opened(other, other.get(DownloadLink, made["id"]), made["token"])
|
||||
)
|
||||
t.join(timeout=1)
|
||||
db.commit()
|
||||
t.join(timeout=10)
|
||||
assert not isinstance(outcome[0], Exception), outcome
|
||||
db.expire_all()
|
||||
assert db.get(DownloadLink, made["id"]).view_count == 2
|
||||
|
||||
|
||||
def test_a_file_shared_as_it_is_deleted_takes_its_new_link_with_it(shelf, db):
|
||||
from app.routes import downloads as routes
|
||||
|
||||
job_id = shelf["outside"]
|
||||
routes._shareable_job(db, shelf["user"], job_id) # held from the check until this commits
|
||||
t, outcome = _meanwhile(lambda other: other.delete(other.get(DownloadJob, job_id)))
|
||||
t.join(timeout=1)
|
||||
assert t.is_alive() # the delete waits for the link on its way in
|
||||
db.add(DownloadLink(job_id=job_id, token_hash="h" * 64, created_by=shelf["user"].id))
|
||||
db.commit()
|
||||
t.join(timeout=10)
|
||||
assert outcome == [None]
|
||||
assert db.execute(select(DownloadLink.id)).scalars().all() == []
|
||||
|
||||
|
||||
def test_a_folder_shared_as_it_is_deleted_takes_its_new_link_with_it(shelf, db):
|
||||
from app.routes import downloads as routes
|
||||
|
||||
fid = shelf["folder"]
|
||||
routes._shareable_folder(db, shelf["user"], fid) # held from the check until this commits
|
||||
t, outcome = _meanwhile(lambda other: folders.delete_own(other, other.get(FileFolder, fid)))
|
||||
t.join(timeout=1)
|
||||
assert t.is_alive() # the delete waits for the link on its way in
|
||||
db.add(DownloadLink(folder_id=fid, token_hash="h" * 64, created_by=shelf["user"].id))
|
||||
db.commit()
|
||||
t.join(timeout=10)
|
||||
assert not isinstance(outcome[0], Exception), outcome
|
||||
assert db.execute(select(DownloadLink.id)).scalars().all() == []
|
||||
@@ -0,0 +1,227 @@
|
||||
"""A `/watch` link's token at rest (downloads/links.py, migration 0107), DB-lane.
|
||||
|
||||
The token is the credential, so the database keeps only its SHA-256 (to find the link by) and a
|
||||
Fernet-encrypted copy (for its owner to copy the same link again). Without a copy this key can read,
|
||||
the link still opens and only its address cannot be shown again. Making, changing and revoking a
|
||||
link is audited, never with its token or password.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from cryptography.fernet import Fernet
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app import security
|
||||
from app.config import settings
|
||||
from app.db import engine
|
||||
from app.downloads import probe as probemod
|
||||
from app.models import AuditLog, DownloadLink
|
||||
|
||||
_VIDEO = {
|
||||
"streams": [
|
||||
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
|
||||
{"codec_type": "audio", "codec_name": "aac"},
|
||||
],
|
||||
"format": {"format_name": "mov,mp4", "duration": "5"},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def key(monkeypatch):
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def owner_job(as_user, make_user, tmp_path, monkeypatch, upload_headers):
|
||||
monkeypatch.setattr(settings, "files_root", str(tmp_path))
|
||||
monkeypatch.setattr(probemod, "probe_json", lambda _p: _VIDEO)
|
||||
owner = as_user(make_user())
|
||||
job = owner.post("/api/files/upload", content=b"fake-video", headers=upload_headers("clip.mp4"))
|
||||
assert job.status_code == 200, job.text
|
||||
return owner, job.json()["id"]
|
||||
|
||||
|
||||
def _row(db, link_id: int) -> dict:
|
||||
db.expire_all()
|
||||
row = db.execute(
|
||||
text("SELECT * FROM download_links WHERE id = :id"), {"id": link_id}
|
||||
).mappings()
|
||||
return dict(row.one())
|
||||
|
||||
|
||||
def _audits(db, action: str) -> list[AuditLog]:
|
||||
db.expire_all()
|
||||
return list(db.execute(select(AuditLog).where(AuditLog.action == action)).scalars())
|
||||
|
||||
|
||||
def test_the_token_is_kept_only_hashed_and_encrypted(db, key, owner_job, client):
|
||||
owner, job_id = owner_job
|
||||
made = owner.post(f"/api/downloads/{job_id}/links", json={}).json()
|
||||
token = made["token"]
|
||||
assert made["url"] == f"/watch/{token}"
|
||||
|
||||
row = _row(db, made["id"])
|
||||
assert "token" not in row
|
||||
assert token not in json.dumps(row, default=str)
|
||||
assert row["token_hash"] == security.hash_token(token)
|
||||
assert security.decrypt(row["token_enc"]) == token
|
||||
|
||||
# The address it was given opens it, and its owner can copy the same one again.
|
||||
assert client.get(f"/api/public/watch/{token}").status_code == 200
|
||||
listed = owner.get(f"/api/downloads/{job_id}/links").json()
|
||||
assert [entry["url"] for entry in listed] == [f"/watch/{token}"]
|
||||
|
||||
|
||||
def test_a_token_naming_no_link_finds_none(db, key, owner_job, client):
|
||||
owner, job_id = owner_job
|
||||
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
|
||||
other = token[:-1] + ("A" if token[-1] != "A" else "B")
|
||||
assert client.get(f"/api/public/watch/{other}").status_code == 404
|
||||
# Its hash is no key either: the lookup hashes what it is given.
|
||||
hashed = security.hash_token(token)
|
||||
assert client.get(f"/api/public/watch/{hashed}").status_code == 404
|
||||
|
||||
|
||||
def test_without_a_key_the_link_opens_but_its_address_is_not_kept(db, owner_job, client):
|
||||
assert not security.can_encrypt()
|
||||
owner, job_id = owner_job
|
||||
made = owner.post(f"/api/downloads/{job_id}/links", json={}).json()
|
||||
# The one who made it has it in hand…
|
||||
assert made["url"] == f"/watch/{made['token']}"
|
||||
assert _row(db, made["id"])["token_enc"] is None
|
||||
assert client.get(f"/api/public/watch/{made['token']}").status_code == 200
|
||||
# …but it cannot be shown again.
|
||||
listed = owner.get(f"/api/downloads/{job_id}/links").json()
|
||||
assert [(entry["token"], entry["url"]) for entry in listed] == [(None, None)]
|
||||
|
||||
|
||||
def test_a_changed_key_hides_the_address_and_the_link_still_opens(
|
||||
db, key, owner_job, client, monkeypatch
|
||||
):
|
||||
owner, job_id = owner_job
|
||||
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
|
||||
listed = owner.get(f"/api/downloads/{job_id}/links").json()
|
||||
assert [entry["url"] for entry in listed] == [None]
|
||||
assert client.get(f"/api/public/watch/{token}").status_code == 200
|
||||
|
||||
|
||||
def test_making_changing_and_revoking_a_link_is_audited_without_its_secrets(db, key, owner_job):
|
||||
owner, job_id = owner_job
|
||||
pw = "a secret word"
|
||||
made = owner.post(
|
||||
f"/api/downloads/{job_id}/links", json={"password": pw, "allow_download": False}
|
||||
).json()
|
||||
link_id = made["id"]
|
||||
patch = f"/api/downloads/links/{link_id}"
|
||||
assert owner.patch(patch, json={"allow_download": True}).status_code == 200
|
||||
assert owner.patch(patch, json={"password": "another one"}).status_code == 200
|
||||
# Nothing changed: nothing to record.
|
||||
assert owner.patch(patch, json={"allow_download": True}).status_code == 200
|
||||
gone = owner.delete(f"/api/downloads/links/{link_id}")
|
||||
assert gone.json() == {"ok": True, "deleted": link_id}
|
||||
|
||||
(create,) = _audits(db, "download_link_create")
|
||||
assert create.target_id == str(link_id)
|
||||
assert create.after == {
|
||||
"job_id": job_id,
|
||||
"allow_download": False,
|
||||
"has_password": True,
|
||||
"expires_at": None,
|
||||
}
|
||||
updates = _audits(db, "download_link_update")
|
||||
assert [(u.before["allow_download"], u.after["allow_download"]) for u in updates] == [
|
||||
(False, True),
|
||||
(True, True), # the password changed: a rotation is recorded though the flags are equal
|
||||
]
|
||||
(revoke,) = _audits(db, "download_link_revoke")
|
||||
assert revoke.target_id == str(link_id)
|
||||
assert revoke.after == {"job_id": job_id}
|
||||
|
||||
trail = json.dumps(
|
||||
[[a.summary, a.before, a.after] for a in [create, *updates, revoke]], default=str
|
||||
)
|
||||
for secret in (made["token"], pw, "another one"):
|
||||
assert secret not in trail
|
||||
|
||||
|
||||
def test_the_migration_keeps_every_address_already_sent(db, key, owner_job, client):
|
||||
owner, job_id = owner_job
|
||||
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0106_api_tokens")
|
||||
try:
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
text("INSERT INTO download_links (job_id, token) VALUES (:j, 'an-old-token-sent')"),
|
||||
{"j": job_id},
|
||||
)
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
(row,) = db.execute(text("SELECT token_hash, token_enc FROM download_links")).mappings().all()
|
||||
assert row["token_hash"] == security.hash_token("an-old-token-sent")
|
||||
assert security.decrypt(row["token_enc"]) == "an-old-token-sent"
|
||||
assert client.get("/api/public/watch/an-old-token-sent").status_code == 200
|
||||
listed = owner.get(f"/api/downloads/{job_id}/links").json()
|
||||
assert [entry["url"] for entry in listed] == ["/watch/an-old-token-sent"]
|
||||
|
||||
|
||||
def test_the_downgrade_writes_the_tokens_back(db, key, owner_job):
|
||||
owner, job_id = owner_job
|
||||
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
|
||||
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0106_api_tokens")
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
back = conn.execute(text("SELECT token FROM download_links")).scalars().all()
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
assert back == [token]
|
||||
assert db.execute(select(DownloadLink.token_hash)).scalars().all() == [
|
||||
security.hash_token(token)
|
||||
]
|
||||
|
||||
|
||||
def test_without_a_key_the_migration_keeps_the_address_opening_but_not_shown(db, owner_job, client):
|
||||
"""No key to seal with: the hash alone is kept, the link still opens, its address is gone."""
|
||||
assert not security.can_encrypt()
|
||||
owner, job_id = owner_job
|
||||
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0106_api_tokens")
|
||||
try:
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
text("INSERT INTO download_links (job_id, token) VALUES (:j, 'an-old-token-sent')"),
|
||||
{"j": job_id},
|
||||
)
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
(row,) = db.execute(text("SELECT token_hash, token_enc FROM download_links")).mappings().all()
|
||||
assert row["token_hash"] == security.hash_token("an-old-token-sent")
|
||||
assert row["token_enc"] is None
|
||||
assert client.get("/api/public/watch/an-old-token-sent").status_code == 200
|
||||
assert [e["url"] for e in owner.get(f"/api/downloads/{job_id}/links").json()] == [None]
|
||||
|
||||
|
||||
def test_a_downgrade_with_no_readable_copy_writes_a_fresh_token(db, key, owner_job, monkeypatch):
|
||||
"""A key since changed reads no copy: the old column gets a random token, so the link is
|
||||
revoked rather than left with no token (the column is NOT NULL there)."""
|
||||
owner, job_id = owner_job
|
||||
token = owner.post(f"/api/downloads/{job_id}/links", json={}).json()["token"]
|
||||
db.commit() # an open transaction's locks would hold the downgrade's ALTER TABLE forever
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0106_api_tokens")
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
back = conn.execute(text("SELECT token FROM download_links")).scalars().all()
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
assert len(back) == 1 and back[0] and back[0] != token
|
||||
@@ -53,6 +53,9 @@ class TestGrant:
|
||||
def test_malformed_grants_are_rejected_not_crashed(self):
|
||||
for bad in [None, "", "no-dot", "notanumber.sig"]:
|
||||
assert check_grant("tok", bad, 0) is False
|
||||
# The query string decodes as UTF-8: a signature past ASCII is just a wrong one.
|
||||
future = int(datetime.now(timezone.utc).timestamp()) + _GRANT_TTL
|
||||
assert check_grant("tok", f"{future}.é", 0) is False
|
||||
|
||||
|
||||
def _asset(**over):
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Migration 0106's downgrade data step, DB-lane: a Hub folder holding a file becomes an own folder
|
||||
that keeps the file — under a name its user's own folders do not already have (their names are
|
||||
unique per user, case-blind), and with its token gone with the tokens table."""
|
||||
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db import engine
|
||||
|
||||
|
||||
def test_a_hub_folder_with_a_file_comes_back_as_an_own_folder_beside_a_namesake(db, make_user):
|
||||
uid = make_user().id
|
||||
with engine.begin() as conn:
|
||||
hub = conn.execute(
|
||||
text("INSERT INTO file_folders (user_id, kind) VALUES (:u, 'hub') RETURNING id"),
|
||||
{"u": uid},
|
||||
).scalar_one()
|
||||
conn.execute(
|
||||
text("INSERT INTO file_folders (user_id, kind, name) VALUES (:u, 'own', :n)"),
|
||||
{"u": uid, "n": f"hub {hub}"}, # the name the downgrade would pick, in another case
|
||||
)
|
||||
held = conn.execute(
|
||||
text(
|
||||
"INSERT INTO download_jobs (user_id, source_kind, source_ref, profile_snapshot,"
|
||||
" status, folder_id) VALUES (:u, 'upload', 'x', '{}', 'queued', :f) RETURNING id"
|
||||
),
|
||||
{"u": uid, "f": hub},
|
||||
).scalar_one()
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO api_tokens (user_id, name, prefix, token_hash) VALUES"
|
||||
" (:u, 'hub', 'sfl_abcd1234', 'h') "
|
||||
),
|
||||
{"u": uid},
|
||||
)
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0105_job_source_key")
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
kind, name = conn.execute(
|
||||
text("SELECT kind, name FROM file_folders WHERE id = :i"), {"i": hub}
|
||||
).one()
|
||||
folder_of_job = conn.execute(
|
||||
text("SELECT folder_id FROM download_jobs WHERE id = :j"), {"j": held}
|
||||
).scalar_one()
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
assert (kind, name) == ("own", f"Hub {hub} (2)")
|
||||
assert folder_of_job == hub
|
||||
@@ -0,0 +1,165 @@
|
||||
"""Migration 0109's data step, DB-lane: the audit log's run rows move into `job_runs` with their
|
||||
outcome intact, and leave the audit log; an admin action stays where it was. The downgrade moves
|
||||
them back."""
|
||||
|
||||
import json
|
||||
from datetime import timedelta
|
||||
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db import engine
|
||||
|
||||
|
||||
def _audit(conn, actor, action, target, after):
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after,"
|
||||
" created_at) VALUES (:a, :act, 'job', :t, 's', CAST(:after AS json),"
|
||||
" CAST('2026-10-01T03:00:00+00:00' AS timestamptz))"
|
||||
),
|
||||
{"a": actor, "act": action, "t": target, "after": json.dumps(after)},
|
||||
)
|
||||
|
||||
|
||||
def test_the_audit_logs_runs_move_into_the_run_table_and_back(db, make_user):
|
||||
uid = make_user().id
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0108_folder_links")
|
||||
try:
|
||||
with engine.begin() as conn:
|
||||
_audit(
|
||||
conn,
|
||||
None,
|
||||
"job_run",
|
||||
"rss_poll",
|
||||
{"status": "ok", "summary": "new=3", "result": {"new": 3}},
|
||||
)
|
||||
_audit(
|
||||
conn,
|
||||
uid,
|
||||
"job_run",
|
||||
"rss_poll",
|
||||
{"status": "error", "summary": "boom", "result": {"failure": "rss_all_failed"}},
|
||||
)
|
||||
_audit(
|
||||
conn, uid, "job_run", "backfill", {"status": "skipped", "summary": "sync paused"}
|
||||
)
|
||||
_audit(conn, uid, "config_set", None, {"key": "x"})
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
with engine.connect() as conn:
|
||||
runs = conn.execute(
|
||||
text(
|
||||
"SELECT job, trigger, actor_id, status, summary, result, error, failure,"
|
||||
" to_char(started_at AT TIME ZONE 'UTC', 'MM-DD HH24'), finished_at = started_at"
|
||||
" FROM job_runs ORDER BY id"
|
||||
)
|
||||
).all()
|
||||
left = conn.execute(text("SELECT action FROM audit_log")).scalars().all()
|
||||
assert [tuple(r) for r in runs] == [
|
||||
("rss_poll", "scheduled", None, "ok", "new=3", {"new": 3}, None, None, "10-01 03", True),
|
||||
(
|
||||
"rss_poll",
|
||||
"manual",
|
||||
uid,
|
||||
"error",
|
||||
None,
|
||||
None,
|
||||
"boom",
|
||||
{"failure": "rss_all_failed"},
|
||||
"10-01 03",
|
||||
True,
|
||||
),
|
||||
("backfill", "manual", uid, "skipped", "sync paused", None, None, None, "10-01 03", True),
|
||||
]
|
||||
assert left == ["config_set"]
|
||||
|
||||
command.downgrade(cfg, "0108_folder_links")
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
back = conn.execute(
|
||||
text(
|
||||
"SELECT actor_id, target_id, after::jsonb FROM audit_log"
|
||||
" WHERE action = 'job_run' ORDER BY id"
|
||||
)
|
||||
).all()
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
# What the audit row has no column for rides along, so the next upgrade restores it.
|
||||
def run(trigger: str) -> dict:
|
||||
return {"trigger": trigger, "quota_units": 0, "started_at": "2026-10-01T03:00:00+00:00"}
|
||||
|
||||
assert [tuple(r) for r in back] == [
|
||||
(
|
||||
None,
|
||||
"rss_poll",
|
||||
{"status": "ok", "summary": "new=3", "result": {"new": 3}, **run("scheduled")},
|
||||
),
|
||||
(
|
||||
uid,
|
||||
"rss_poll",
|
||||
{
|
||||
"status": "error",
|
||||
"summary": "boom",
|
||||
"result": {"failure": "rss_all_failed"},
|
||||
**run("manual"),
|
||||
},
|
||||
),
|
||||
(uid, "backfill", {"status": "skipped", "summary": "sync paused", **run("manual")}),
|
||||
]
|
||||
|
||||
|
||||
def test_a_down_and_up_round_trip_keeps_every_run(db, make_user):
|
||||
"""A rollback then a redeploy: an interrupted run, a run caught mid-way and a user's run of one
|
||||
channel come back as runs — not stranded in the audit log under an action no code knows."""
|
||||
uid = make_user().id
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at,"
|
||||
" finished_at, quota_units) VALUES"
|
||||
" ('backfill', 'scheduled', NULL, NULL, 'interrupted', now(), now(), 0),"
|
||||
" ('maintenance', 'manual', :u, NULL, 'running', now(), NULL, 0),"
|
||||
" ('channel_backfill', 'ad_hoc', :u, 'UC_x', 'ok', now(), now(), 0),"
|
||||
# Its admin is gone (SET NULL): only the row itself still says it was manual.
|
||||
" ('enrich', 'manual', NULL, NULL, 'ok',"
|
||||
" '2026-10-01 10:00+00', '2026-10-01 10:30+00', 42)"
|
||||
),
|
||||
{"u": uid},
|
||||
)
|
||||
# What the run caught mid-way had spent: only its quota rows know it yet.
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO quota_events (action, units, job_run_id)"
|
||||
" SELECT 'videos.list', 7, id FROM job_runs WHERE job = 'maintenance'"
|
||||
)
|
||||
)
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0108_folder_links")
|
||||
command.upgrade(cfg, "head")
|
||||
with engine.connect() as conn:
|
||||
runs = conn.execute(
|
||||
text("SELECT job, trigger, actor_id, target, status FROM job_runs ORDER BY id")
|
||||
).all()
|
||||
kept = conn.execute(
|
||||
text("SELECT quota_units, finished_at - started_at FROM job_runs WHERE job = 'enrich'")
|
||||
).one()
|
||||
stopped = conn.execute(
|
||||
text("SELECT quota_units FROM job_runs WHERE job = 'maintenance'")
|
||||
).scalar()
|
||||
stranded = conn.execute(
|
||||
text("SELECT count(*) FROM audit_log WHERE action = 'job_run'")
|
||||
).scalar()
|
||||
assert [tuple(r) for r in runs] == [
|
||||
("backfill", "scheduled", None, None, "interrupted"),
|
||||
("maintenance", "manual", uid, None, "interrupted"), # the rollback stopped it
|
||||
("channel_backfill", "ad_hoc", uid, "UC_x", "ok"),
|
||||
("enrich", "manual", None, None, "ok"),
|
||||
]
|
||||
assert tuple(kept) == (42, timedelta(minutes=30))
|
||||
assert stopped == 7
|
||||
assert stranded == 0
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Migration 0110's data step, DB-lane: an admin's interval override moves from `scheduler_settings`
|
||||
into sysconfig and back, a row for a job that no longer exists is not carried over, and the unread
|
||||
SponsorBlock default leaves the store.
|
||||
|
||||
The schema half is checked by test_migrations.py; a value lost on the way is invisible there.
|
||||
"""
|
||||
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
|
||||
from app import scheduler
|
||||
from app.db import engine
|
||||
|
||||
|
||||
def _config(conn) -> dict[str, str]:
|
||||
return dict(conn.execute(text("SELECT key, value FROM system_config")).tuples().all())
|
||||
|
||||
|
||||
def test_an_interval_override_moves_into_sysconfig_and_back(db):
|
||||
cfg = Config("alembic.ini")
|
||||
command.downgrade(cfg, "0109_job_runs")
|
||||
try:
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
|
||||
" VALUES ('rss_poll', 25), ('log_gc', 90), ('retired_job', 5)"
|
||||
)
|
||||
)
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO system_config (key, value, encrypted) VALUES"
|
||||
" ('sponsorblock_default', 'true', false), ('quota_daily_budget', '5000', false)"
|
||||
)
|
||||
)
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
with engine.connect() as conn:
|
||||
assert _config(conn) == {
|
||||
"job_interval_rss_poll": "25",
|
||||
"job_interval_log_gc": "90",
|
||||
"quota_daily_budget": "5000",
|
||||
}
|
||||
assert conn.execute(text("SELECT to_regclass('scheduler_settings')")).scalar() is None
|
||||
assert scheduler.load_intervals(db)["rss_poll"] == 25
|
||||
|
||||
command.downgrade(cfg, "0109_job_runs")
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
back = conn.execute(text("SELECT job_id, interval_minutes FROM scheduler_settings"))
|
||||
assert dict(back.tuples().all()) == {"rss_poll": 25, "log_gc": 90}
|
||||
assert _config(conn) == {"quota_daily_budget": "5000"}
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
@@ -9,7 +9,6 @@ from datetime import datetime, timezone
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app import scheduler
|
||||
from app.models import PlexItem, PlexLibrary, PlexPerson, User
|
||||
from app.plex import imgcache
|
||||
from app.plex import people as plex_people
|
||||
@@ -340,12 +339,6 @@ def test_the_people_job_runs_its_halves_without_a_tmdb_key(db, cache_dir, plex_o
|
||||
}
|
||||
|
||||
|
||||
def test_a_phase_count_is_a_change_worth_an_audit_row():
|
||||
assert scheduler._run_changed({"bios": {"enriched": 3}, "portraits": {"cached": 0}})
|
||||
assert scheduler._run_changed({"recent": {"videos_added": 2}, "deep": {}})
|
||||
assert not scheduler._run_changed({"bios": {"skipped": "x"}, "portraits": {"cached": 0}})
|
||||
|
||||
|
||||
def test_the_person_page_says_a_bio_is_on_its_way(db, monkeypatch, override_dep):
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
"""The run words and the migrated database, DB-lane: the `job_runs` CHECKs are frozen in a migration,
|
||||
and the model's are built from the enums, so nothing else would notice the two drifting apart until a
|
||||
write of the new word failed in production."""
|
||||
|
||||
import re
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.runstatus import RunStatus, RunTrigger
|
||||
|
||||
|
||||
def _words(definition: str) -> set[str]:
|
||||
return set(re.findall(r"'([a-z_]+)'", definition))
|
||||
|
||||
|
||||
def test_the_databases_checks_allow_exactly_the_run_words(db):
|
||||
defs = dict(
|
||||
db.execute(
|
||||
text(
|
||||
"SELECT conname, pg_get_constraintdef(oid) FROM pg_constraint"
|
||||
" WHERE conname LIKE 'ck_job_runs_%'"
|
||||
)
|
||||
).all()
|
||||
)
|
||||
assert _words(defs["ck_job_runs_status"]) == set(RunStatus)
|
||||
assert _words(defs["ck_job_runs_trigger"]) == set(RunTrigger)
|
||||
assert _words(defs["ck_job_runs_finished"]) == {RunStatus.RUNNING}
|
||||
+534
-47
@@ -1,11 +1,21 @@
|
||||
"""_claim_running is the single-run guard that both the scheduled fire and the manual trigger go
|
||||
"""The scheduler's run path, DB-lane: the single-run guard, and every run as a durable `job_runs` row
|
||||
— its outcome as data, what it spent, the lines it logged — that outlives the process.
|
||||
|
||||
_claim_running is the single-run guard that both the scheduled fire and the manual trigger go
|
||||
through (at the top of _job): holding _activity_lock across the check-and-set means exactly one
|
||||
caller can hold a job's "running" claim at a time, and it's claimable again once released."""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import scheduler
|
||||
from app.models import Notification
|
||||
from app import jobruns, logs, logsink, quota, scheduler, state, sysconfig
|
||||
from app.models import AuditLog, Channel, JobRun, Notification, QuotaEvent
|
||||
from app.routes import scheduler as scheduler_routes
|
||||
from app.runstatus import RunStatus, RunTrigger
|
||||
|
||||
_JOB = "unit_test_claim" # not a real job id — exercises the primitive without touching live jobs
|
||||
|
||||
@@ -31,26 +41,38 @@ def test_claim_is_exclusive_until_released():
|
||||
# ---- the result as data --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _snapshot_job(job_id):
|
||||
return next(j for j in scheduler.scheduler_snapshot()["jobs"] if j["id"] == job_id)
|
||||
def _snapshot_job(db, job_id):
|
||||
return next(j for j in scheduler.scheduler_snapshot(db)["jobs"] if j["id"] == job_id)
|
||||
|
||||
|
||||
def _runs(db, job=None) -> list[JobRun]:
|
||||
db.expire_all()
|
||||
q = select(JobRun).order_by(JobRun.id)
|
||||
if job is not None:
|
||||
q = q.where(JobRun.job == job)
|
||||
return list(db.execute(q).scalars())
|
||||
|
||||
|
||||
def _manual(actor_id, name, fn):
|
||||
token = scheduler._manual_actor.set(actor_id)
|
||||
try:
|
||||
scheduler._job(name, fn)
|
||||
finally:
|
||||
scheduler._manual_actor.reset(token)
|
||||
|
||||
|
||||
def test_a_run_keeps_its_result_as_data_for_the_readers_language(db, make_user):
|
||||
"""The dashboard and the manual-run notice showed `_summarize`'s English one-liner ("new=3,
|
||||
failed=0", a Python repr for a nested phase). The result rides along as data, so the client names
|
||||
it; the one-liner stays as the fallback and the audit text."""
|
||||
it; the one-liner stays as the fallback."""
|
||||
admin = make_user("sched@example.com", role="admin")
|
||||
token = scheduler._manual_actor.set(admin.id)
|
||||
try:
|
||||
scheduler._job("rss_poll", lambda _db: {"new": 3, "failed": 0, "fallback": 1})
|
||||
scheduler._job("enrich", lambda _db: 5) # enrich returns a bare count
|
||||
finally:
|
||||
scheduler._manual_actor.reset(token)
|
||||
_manual(admin.id, "rss_poll", lambda _db: {"new": 3, "failed": 0, "fallback": 1})
|
||||
_manual(admin.id, "enrich", lambda _db: 5) # enrich returns a bare count
|
||||
|
||||
rss = _snapshot_job("rss_poll")
|
||||
rss = _snapshot_job(db, "rss_poll")
|
||||
assert rss["result"] == {"new": 3, "failed": 0, "fallback": 1}
|
||||
assert rss["last_result"] == "new=3, failed=0, fallback=1"
|
||||
assert _snapshot_job("enrich")["result"] == {"enriched": 5}
|
||||
assert _snapshot_job(db, "enrich")["result"] == {"enriched": 5}
|
||||
|
||||
notices = db.execute(
|
||||
select(Notification).where(Notification.user_id == admin.id).order_by(Notification.id)
|
||||
@@ -64,53 +86,518 @@ def test_a_run_keeps_its_result_as_data_for_the_readers_language(db, make_user):
|
||||
def test_a_failed_run_drops_the_previous_result(db):
|
||||
scheduler._job("rss_poll", lambda _db: {"new": 1})
|
||||
scheduler._job("rss_poll", lambda _db: (_ for _ in ()).throw(RuntimeError("boom")))
|
||||
rss = _snapshot_job("rss_poll")
|
||||
rss = _snapshot_job(db, "rss_poll")
|
||||
assert rss["status"] == "error"
|
||||
assert rss["last_error"] == "boom"
|
||||
assert rss["result"] is None
|
||||
|
||||
|
||||
def test_a_job_runs_audit_row_carries_its_result_as_data(db):
|
||||
"""The audit log names a run's result in the reader's language, as the inbox does —
|
||||
from data, not from the English one-liner (a nested phase there was a Python dict repr)."""
|
||||
from app.models import AuditLog, User
|
||||
|
||||
admin = User(email="auditrun@example.com")
|
||||
db.add(admin)
|
||||
db.commit()
|
||||
scheduler._audit_run(db, None, "people_enrich", "ok", "x", {"bios": {"enriched": 2}})
|
||||
scheduler._audit_run(db, admin.id, "rss_poll", "skipped", "sync paused", scheduler._PAUSED)
|
||||
scheduler._audit_run(db, admin.id, "rss_poll", "error", "boom", None)
|
||||
rows = db.execute(select(AuditLog).order_by(AuditLog.id)).scalars().all()
|
||||
assert [r.after for r in rows] == [
|
||||
{"status": "ok", "summary": "x", "result": {"bios": {"enriched": 2}}},
|
||||
{"status": "skipped", "summary": "sync paused", "result": {"skipped": "sync paused"}},
|
||||
{"status": "error", "summary": "boom"},
|
||||
]
|
||||
|
||||
|
||||
def test_a_failure_the_job_names_rides_along_as_data(db, make_user):
|
||||
"""A failed RSS poll reached the Hungarian audit log as the server's English sentence. A
|
||||
failure the job names (`progress.JobFailed`) is stored as data beside that sentence — on the
|
||||
dashboard, the manual-run notice and the audit row — so the client says it in the reader's
|
||||
language. Any other failure keeps only its own text (the test above)."""
|
||||
"""A failed RSS poll reached the Hungarian UI as the server's English sentence. A failure the job
|
||||
names (`progress.JobFailed`) is stored as data beside that sentence — on the run's row, the
|
||||
dashboard and the manual-run notice — so the client says it in the reader's language."""
|
||||
from app import progress
|
||||
from app.models import AuditLog
|
||||
|
||||
def outage(_db):
|
||||
raise progress.JobFailed("rss_all_failed", "RSS poll failed", total=2, cause="transport")
|
||||
|
||||
admin = make_user("failure@example.com", role="admin")
|
||||
token = scheduler._manual_actor.set(admin.id)
|
||||
try:
|
||||
scheduler._job("rss_poll", outage)
|
||||
finally:
|
||||
scheduler._manual_actor.reset(token)
|
||||
_manual(admin.id, "rss_poll", outage)
|
||||
|
||||
named = {"failure": "rss_all_failed", "total": 2, "cause": "transport"}
|
||||
rss = _snapshot_job("rss_poll")
|
||||
rss = _snapshot_job(db, "rss_poll")
|
||||
assert rss["last_error"] == "RSS poll failed"
|
||||
assert rss["result"] == named
|
||||
notice = db.execute(select(Notification).where(Notification.user_id == admin.id)).scalar_one()
|
||||
assert notice.data["result"] == named
|
||||
row = db.execute(select(AuditLog).order_by(AuditLog.id.desc())).scalars().first()
|
||||
assert row is not None and row.after["result"] == named
|
||||
(run,) = _runs(db, "rss_poll")
|
||||
assert (run.status, run.error, run.failure) == ("error", "RSS poll failed", named)
|
||||
|
||||
|
||||
# ---- the durable row -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_every_run_is_a_row_with_who_started_it_and_none_in_the_audit_log(db, make_user):
|
||||
"""A no-op scheduled poll was not written anywhere (the audit rule kept only changes), so "did
|
||||
it run at 03:00?" had no answer. Every run is a row now; the audit log keeps admin actions only."""
|
||||
admin = make_user("runs@example.com", role="admin")
|
||||
scheduler._job("rss_poll", lambda _db: {"new": 0})
|
||||
_manual(admin.id, "rss_poll", lambda _db: {"new": 2})
|
||||
|
||||
scheduled, manual = _runs(db, "rss_poll")
|
||||
assert (scheduled.trigger, scheduled.actor_id, scheduled.status) == ("scheduled", None, "ok")
|
||||
assert scheduled.result == {"new": 0}
|
||||
assert (manual.trigger, manual.actor_id, manual.summary) == ("manual", admin.id, "new=2")
|
||||
assert manual.finished_at is not None and manual.finished_at >= manual.started_at
|
||||
assert db.execute(select(AuditLog)).scalars().all() == []
|
||||
|
||||
|
||||
def test_the_last_run_survives_a_restart(db):
|
||||
"""The dashboard's "last run" lived in the process's memory: a deploy blanked every job."""
|
||||
scheduler._job("plex_sync", lambda _db: {"items": 4})
|
||||
with scheduler._activity_lock:
|
||||
scheduler._activity.clear() # what a new process starts with
|
||||
|
||||
plex = _snapshot_job(db, "plex_sync")
|
||||
assert (plex["status"], plex["result"], plex["last_trigger"]) == (
|
||||
"ok",
|
||||
{"items": 4},
|
||||
"scheduled",
|
||||
)
|
||||
assert plex["last_finished"] is not None
|
||||
|
||||
|
||||
def test_a_row_shows_its_last_three_runs_newest_first_an_interrupted_one_included(db):
|
||||
"""The row said only how the LAST run went; an interrupted run read as "no run recorded"."""
|
||||
oldest = jobruns.start("shorts", RunTrigger.SCHEDULED, None)
|
||||
jobruns.interrupt_leftovers()
|
||||
scheduler._job("shorts", lambda _db: {"probed": 1})
|
||||
scheduler._job("shorts", lambda _db: {"probed": 2})
|
||||
scheduler._job("shorts", lambda _db: 1 / 0)
|
||||
scheduler._job("rss_poll", lambda _db: {"new": 0}) # another job's run stays on its own row
|
||||
|
||||
recent = _snapshot_job(db, "shorts")["recent"]
|
||||
assert [r["status"] for r in recent] == ["error", "ok", "ok"]
|
||||
assert oldest not in [r["id"] for r in recent] # the fourth-newest is not shown
|
||||
assert recent[0]["trigger"] == "scheduled" and recent[0]["finished_at"] is not None
|
||||
assert [r["status"] for r in _snapshot_job(db, "rss_poll")["recent"]] == ["ok"]
|
||||
|
||||
jobruns.start("shorts", RunTrigger.SCHEDULED, None)
|
||||
jobruns.interrupt_leftovers()
|
||||
shorts = _snapshot_job(db, "shorts")
|
||||
assert shorts["status"] == "interrupted"
|
||||
assert [r["status"] for r in shorts["recent"]] == ["interrupted", "error", "ok"]
|
||||
|
||||
|
||||
def test_a_paused_sync_is_a_skipped_run(db):
|
||||
state.set_sync_paused(db, True)
|
||||
db.commit()
|
||||
scheduler._job("rss_poll", lambda _db: pytest.fail("a paused run must not do its work"))
|
||||
(run,) = _runs(db, "rss_poll")
|
||||
assert (run.status, run.result) == ("skipped", {"skipped": "sync paused"})
|
||||
|
||||
|
||||
def test_a_run_left_running_by_a_stopped_process_is_interrupted_and_stays_so(db):
|
||||
"""A deploy mid-run left nothing behind before; now the row would say `running` forever. The
|
||||
next startup closes it, and the stale thread's own finish cannot rewrite it."""
|
||||
run_id = jobruns.start("backfill", RunTrigger.SCHEDULED, None)
|
||||
assert jobruns.interrupt_leftovers() == 1
|
||||
|
||||
(run,) = _runs(db, "backfill")
|
||||
assert run.status == "interrupted" and run.finished_at is not None
|
||||
jobruns.finish(run_id, RunStatus.OK, summary="late")
|
||||
(run,) = _runs(db, "backfill")
|
||||
assert (run.status, run.summary) == ("interrupted", None)
|
||||
|
||||
|
||||
def test_a_run_knows_what_it_spent_and_a_request_spend_is_nobodys_run(db):
|
||||
def spends(_db):
|
||||
quota.record_usage(5)
|
||||
quota.record_usage(3)
|
||||
return {"enriched": 2}
|
||||
|
||||
scheduler._job("enrich", spends)
|
||||
quota.record_usage(11) # a request's own spend, outside any run
|
||||
|
||||
(run,) = _runs(db, "enrich")
|
||||
assert run.quota_units == 8
|
||||
events = db.execute(select(QuotaEvent.units, QuotaEvent.job_run_id)).all()
|
||||
assert sorted(events) == [(3, run.id), (5, run.id), (11, None)]
|
||||
assert _snapshot_job(db, "enrich")["quota_units"] == 8
|
||||
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.addFilter(logs.ContextFilter())
|
||||
self.records: list[logging.LogRecord] = []
|
||||
|
||||
def emit(self, record):
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
def test_a_runs_log_lines_carry_its_id(db):
|
||||
"""So a run's row can open exactly the lines it logged (the log table's `run_id`)."""
|
||||
probe = logging.getLogger("siftlode.test.run")
|
||||
handler = _Collect()
|
||||
probe.addHandler(handler)
|
||||
probe.setLevel(logging.INFO)
|
||||
try:
|
||||
scheduler._job("autotag", lambda _db: probe.info("tagging") or {"tagged": 1})
|
||||
probe.info("after the run")
|
||||
finally:
|
||||
probe.removeHandler(handler)
|
||||
|
||||
(run,) = _runs(db, "autotag")
|
||||
inside, after = (logs.record_fields(r).get("run_id") for r in handler.records)
|
||||
assert (inside, after) == (run.id, None)
|
||||
# …and it reaches the stored row, whose columns are the model's.
|
||||
assert logsink.to_row(logs.record_fields(handler.records[0]))["run_id"] == run.id
|
||||
|
||||
|
||||
def test_retention_reaps_finished_runs_only(db):
|
||||
old = datetime.now(timezone.utc) - timedelta(days=40)
|
||||
db.add_all(
|
||||
[
|
||||
JobRun(
|
||||
job="rss_poll",
|
||||
trigger="scheduled",
|
||||
status="ok",
|
||||
started_at=old,
|
||||
finished_at=old,
|
||||
),
|
||||
JobRun(job="backfill", trigger="scheduled", status="running", started_at=old),
|
||||
JobRun(
|
||||
job="rss_poll",
|
||||
trigger="scheduled",
|
||||
status="ok",
|
||||
started_at=datetime.now(timezone.utc),
|
||||
finished_at=datetime.now(timezone.utc),
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
reaped = next(r for r in _runs(db) if r.status == "ok" and r.started_at < old + timedelta(1))
|
||||
with logs.bind(run_id=reaped.id):
|
||||
quota.record_usage(7)
|
||||
assert jobruns.gc(db, 0) == 0 # 0 = keep forever
|
||||
assert jobruns.gc(db, 30) == 1
|
||||
db.commit()
|
||||
assert sorted((r.job, r.status) for r in _runs(db)) == [
|
||||
("backfill", "running"),
|
||||
("rss_poll", "ok"),
|
||||
]
|
||||
# Its spend stays in the budget's history (kept on its own retention), no longer anyone's run.
|
||||
assert db.execute(select(QuotaEvent.units, QuotaEvent.job_run_id)).all() == [(7, None)]
|
||||
|
||||
|
||||
# ---- "run now" through the scheduler --------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def live_scheduler(db, monkeypatch):
|
||||
"""This process's scheduler, started as the API starts it (on the migrated schema `db` brings),
|
||||
with interval jobs off."""
|
||||
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", False)
|
||||
scheduler.start_scheduler()
|
||||
try:
|
||||
yield scheduler._scheduler
|
||||
finally:
|
||||
scheduler.shutdown_scheduler()
|
||||
|
||||
|
||||
def _wait_finished(db, job, timeout=10.0) -> JobRun:
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
runs = [r for r in _runs(db, job) if r.status != "running"]
|
||||
if runs:
|
||||
return runs[-1]
|
||||
time.sleep(0.05)
|
||||
raise AssertionError(f"{job} did not finish")
|
||||
|
||||
|
||||
def test_run_now_goes_through_the_scheduler_even_with_interval_jobs_off(
|
||||
db, make_user, live_scheduler, monkeypatch
|
||||
):
|
||||
admin = make_user("now@example.com", role="admin")
|
||||
monkeypatch.setitem(
|
||||
scheduler.JOB_FUNCS, "shorts", lambda: scheduler._job("shorts", lambda _db: {"probed": 3})
|
||||
)
|
||||
assert live_scheduler.get_jobs() == [] # SCHEDULER_ENABLED=false: no interval job
|
||||
live_scheduler.pause() # hold the one-shot so the second click finds it pending
|
||||
assert scheduler.trigger_job("shorts", admin.id) == "started"
|
||||
assert scheduler.trigger_job("shorts", admin.id) == "already_running"
|
||||
assert [j.id for j in live_scheduler.get_jobs()] == ["shorts@now"]
|
||||
live_scheduler.resume()
|
||||
|
||||
run = _wait_finished(db, "shorts")
|
||||
assert (run.trigger, run.actor_id, run.result) == ("manual", admin.id, {"probed": 3})
|
||||
|
||||
|
||||
def test_an_ad_hoc_run_is_the_users_run_of_its_target_with_its_spend_and_failure(
|
||||
db, make_user, live_scheduler
|
||||
):
|
||||
"""The one-channel deep backfill ran as a BackgroundTask nothing recorded: its spend counted as
|
||||
nobody's run and a failure lived only in the log."""
|
||||
user = make_user("adhoc@example.com")
|
||||
|
||||
def work():
|
||||
quota.record_usage(3)
|
||||
return {"videos_added": 50}
|
||||
|
||||
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, work, actor_id=user.id, target="UC_a")
|
||||
run = _wait_finished(db, "channel_backfill")
|
||||
assert (run.trigger, run.actor_id, run.target) == ("ad_hoc", user.id, "UC_a")
|
||||
assert (run.status, run.result, run.quota_units) == ("ok", {"videos_added": 50}, 3)
|
||||
|
||||
def fail():
|
||||
raise RuntimeError("quota exceeded")
|
||||
|
||||
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, fail, actor_id=user.id, target="UC_b")
|
||||
deadline = time.monotonic() + 10
|
||||
while time.monotonic() < deadline and len(_finished(db, "channel_backfill")) < 2:
|
||||
time.sleep(0.05)
|
||||
failed = next(r for r in _runs(db, "channel_backfill") if r.target == "UC_b")
|
||||
assert (failed.status, failed.error) == ("error", "quota exceeded")
|
||||
|
||||
|
||||
def test_the_page_shows_the_ad_hoc_runs_beside_the_jobs_with_their_channel(db, make_user):
|
||||
user = make_user("shown@example.com")
|
||||
db.add(Channel(id="UC_shown", title="BBC Earth"))
|
||||
db.commit()
|
||||
scheduler._ad_hoc("channel_backfill", lambda: {"skipped": "at capacity"}, user.id, "UC_none")
|
||||
scheduler._ad_hoc("channel_backfill", lambda: {"videos_added": 69}, user.id, "UC_shown")
|
||||
|
||||
(row,) = scheduler.scheduler_snapshot(db)["ad_hoc"]
|
||||
assert (row["id"], row["running"], row["target"], row["target_title"]) == (
|
||||
"channel_backfill",
|
||||
False,
|
||||
"UC_shown",
|
||||
"BBC Earth",
|
||||
)
|
||||
assert (row["status"], row["result"], row["last_trigger"]) == (
|
||||
"ok",
|
||||
{"videos_added": 69},
|
||||
"ad_hoc",
|
||||
)
|
||||
assert len(row["recent"]) == 2
|
||||
assert all(j["id"] != "channel_backfill" for j in scheduler.scheduler_snapshot(db)["jobs"])
|
||||
|
||||
seen = []
|
||||
scheduler._ad_hoc(
|
||||
"channel_backfill",
|
||||
lambda: seen.append(scheduler.scheduler_snapshot(db)["ad_hoc"][0]["running"]) or {},
|
||||
user.id,
|
||||
"UC_shown",
|
||||
)
|
||||
assert seen == [True]
|
||||
assert scheduler.scheduler_snapshot(db)["ad_hoc"][0]["running"] is False
|
||||
|
||||
|
||||
def _finished(db, job) -> list[JobRun]:
|
||||
return [r for r in _runs(db, job) if r.status != "running"]
|
||||
|
||||
|
||||
def test_an_ad_hoc_job_outside_the_named_list_is_refused(live_scheduler):
|
||||
"""The page names every ad-hoc job from AD_HOC_JOBS; one outside it would show as a bare id."""
|
||||
with pytest.raises(KeyError):
|
||||
scheduler.run_ad_hoc("unnamed_job", dict, actor_id=1, target="x")
|
||||
assert live_scheduler.get_jobs() == []
|
||||
|
||||
|
||||
def test_a_second_ad_hoc_request_before_the_first_starts_is_that_same_run(
|
||||
db, make_user, live_scheduler
|
||||
):
|
||||
user = make_user("twice@example.com")
|
||||
live_scheduler.pause() # hold the one-shots so the second request finds the first pending
|
||||
for target in ("UC_x", "UC_x", "UC_y"):
|
||||
scheduler.run_ad_hoc(scheduler.CHANNEL_BACKFILL, dict, actor_id=user.id, target=target)
|
||||
assert sorted(j.id for j in live_scheduler.get_jobs()) == [
|
||||
"channel_backfill@UC_x",
|
||||
"channel_backfill@UC_y",
|
||||
]
|
||||
live_scheduler.remove_all_jobs() # never run: a run finishing after the test would land in the next
|
||||
|
||||
|
||||
def test_a_manual_run_leaves_no_actor_behind_on_its_thread(db, make_user):
|
||||
"""The scheduler's pool threads are reused: an actor a manual run left set made the next
|
||||
scheduled run on that thread an admin's "manual" run, with a notice in their inbox."""
|
||||
admin = make_user("pool@example.com", role="admin")
|
||||
seen = []
|
||||
scheduler._as_actor(lambda: seen.append(scheduler._manual_actor.get()), admin.id)()
|
||||
assert seen == [admin.id]
|
||||
assert scheduler._manual_actor.get() is None # this same thread's next task runs as nobody
|
||||
|
||||
|
||||
def test_run_now_refuses_a_job_already_running(live_scheduler):
|
||||
assert scheduler._claim_running("maintenance") is True
|
||||
try:
|
||||
assert scheduler.trigger_job("maintenance") == "already_running"
|
||||
finally:
|
||||
scheduler._record("maintenance", running=False)
|
||||
|
||||
|
||||
def test_startup_closes_what_the_previous_process_left_running(db, monkeypatch):
|
||||
jobruns.start("plex_sync", RunTrigger.MANUAL, None)
|
||||
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", False)
|
||||
scheduler.start_scheduler()
|
||||
scheduler.shutdown_scheduler()
|
||||
(run,) = _runs(db, "plex_sync")
|
||||
assert run.status == "interrupted"
|
||||
|
||||
|
||||
def test_an_interrupted_run_keeps_what_it_spent(db):
|
||||
"""Its spend is in the quota history, tagged with its id: the sweep sums it as `finish` would."""
|
||||
run_id = jobruns.start("backfill", RunTrigger.SCHEDULED, None)
|
||||
with logs.bind(run_id=run_id):
|
||||
quota.record_usage(40)
|
||||
quota.record_usage(2)
|
||||
assert jobruns.interrupt_leftovers() == 1
|
||||
(run,) = _runs(db, "backfill")
|
||||
assert (run.status, run.quota_units) == ("interrupted", 42)
|
||||
|
||||
|
||||
def _flaky_finish(monkeypatch) -> list:
|
||||
"""`jobruns.finish` whose first write fails (the database blinked); the statuses it was asked for."""
|
||||
calls: list = []
|
||||
real = jobruns.finish
|
||||
|
||||
def finish(run_id, status, **outcome):
|
||||
calls.append(status)
|
||||
if len(calls) == 1:
|
||||
raise RuntimeError("db blip")
|
||||
real(run_id, status, **outcome)
|
||||
|
||||
monkeypatch.setattr(jobruns, "finish", finish)
|
||||
return calls
|
||||
|
||||
|
||||
def test_a_run_whose_outcome_cannot_be_written_is_not_called_failed(db, make_user, monkeypatch):
|
||||
"""The job succeeded; only its bookkeeping did not. Writing ERROR then would say the work failed."""
|
||||
calls = _flaky_finish(monkeypatch)
|
||||
scheduler._job("autotag", lambda _db: {"tagged": 1})
|
||||
assert calls == ["ok"]
|
||||
assert not scheduler._is_running("autotag")
|
||||
(run,) = _runs(db, "autotag")
|
||||
assert run.status == "running" # left for the next start's sweep, not rewritten as a failure
|
||||
# Nothing here runs it: the page reads it as the sweep will close it, not as a run going on.
|
||||
row = _row(db, "autotag")
|
||||
assert (row["running"], row["status"], row["recent"][0]["status"]) == (
|
||||
False,
|
||||
"interrupted",
|
||||
"interrupted",
|
||||
)
|
||||
assert row["last_finished"] is not None
|
||||
|
||||
user = make_user("blip@example.com")
|
||||
calls.clear()
|
||||
scheduler._ad_hoc("channel_backfill", lambda: {"videos_added": 1}, user.id, "UC_blip")
|
||||
assert calls == ["ok"]
|
||||
# Its row is still `running`, but nothing here is: the page does not call it going.
|
||||
(row,) = scheduler.scheduler_snapshot(db)["ad_hoc"]
|
||||
assert (row["running"], row["status"], row["recent"][0]["status"]) == (
|
||||
False,
|
||||
"interrupted",
|
||||
"interrupted",
|
||||
)
|
||||
|
||||
|
||||
def test_a_run_going_on_reads_as_running_on_its_row(db):
|
||||
seen = []
|
||||
scheduler._job("autotag", lambda _db: seen.append(_row(db, "autotag")) or {"tagged": 1})
|
||||
(row,) = seen
|
||||
assert (row["running"], row["status"], row["recent"][0]["status"]) == (True, None, "running")
|
||||
assert _row(db, "autotag")["recent"][0]["status"] == "ok"
|
||||
|
||||
|
||||
def _row(db, job_id: str) -> dict:
|
||||
return next(j for j in scheduler.scheduler_snapshot(db)["jobs"] if j["id"] == job_id)
|
||||
|
||||
|
||||
def test_a_run_that_cannot_be_recorded_does_not_wedge_its_job(db, monkeypatch):
|
||||
def broken(*_a, **_k):
|
||||
raise RuntimeError("db down")
|
||||
|
||||
monkeypatch.setattr(jobruns, "start", broken)
|
||||
scheduler._job("autotag", lambda _db: pytest.fail("no row, no run"))
|
||||
assert not scheduler._is_running("autotag")
|
||||
monkeypatch.undo()
|
||||
scheduler._job("autotag", lambda _db: {"tagged": 1})
|
||||
assert [r.status for r in _runs(db, "autotag")] == ["ok"]
|
||||
|
||||
|
||||
def test_the_page_says_where_the_interval_jobs_do_not_run(db, live_scheduler, monkeypatch):
|
||||
"""The scheduler starts on every install now; only SCHEDULER_ENABLED says the jobs run here."""
|
||||
assert scheduler.scheduler_snapshot(db)["running_here"] is False
|
||||
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", True)
|
||||
assert scheduler.scheduler_snapshot(db)["running_here"] is True
|
||||
|
||||
|
||||
def test_an_ad_hoc_run_still_going_shows_although_a_newer_one_ended(db, make_user):
|
||||
"""Two channels can backfill at once; a later run that ended at once must not hide the first."""
|
||||
user = make_user("two@example.com")
|
||||
db.add(Channel(id="UC_long", title="Long one"))
|
||||
db.commit()
|
||||
rows = []
|
||||
|
||||
def long_one():
|
||||
scheduler._ad_hoc(
|
||||
"channel_backfill", lambda: {"skipped": "at capacity"}, user.id, "UC_short"
|
||||
)
|
||||
rows.extend(scheduler.scheduler_snapshot(db)["ad_hoc"])
|
||||
return {}
|
||||
|
||||
scheduler._ad_hoc("channel_backfill", long_one, user.id, "UC_long")
|
||||
(row,) = rows
|
||||
assert (row["running"], row["target"], row["target_title"]) == (True, "UC_long", "Long one")
|
||||
# The outcome beside it is the long one's (none yet), never the short one's skip.
|
||||
assert (row["status"], row["result"], row["last_error"]) == (None, None, None)
|
||||
assert [r["status"] for r in row["recent"]] == ["running", "ok"]
|
||||
|
||||
|
||||
def test_an_interval_change_lands_with_its_audit_row_or_not_at_all(
|
||||
db, make_user, as_user, live_scheduler, monkeypatch
|
||||
):
|
||||
live_scheduler.add_job(lambda: None, "interval", minutes=10, id="backfill")
|
||||
admin = as_user(make_user(role="admin"))
|
||||
|
||||
def broken(*_a, **_k):
|
||||
raise RuntimeError("audit write failed")
|
||||
|
||||
monkeypatch.setattr(scheduler_routes.audit, "record", broken)
|
||||
with pytest.raises(RuntimeError): # the handler raises: there is no response to check
|
||||
_ = admin.patch("/api/admin/scheduler/jobs/backfill", json={"interval_minutes": 45})
|
||||
db.expire_all()
|
||||
assert sysconfig.get_int(db, "job_interval_backfill") == scheduler.JOB_INTERVALS["backfill"]
|
||||
assert live_scheduler.get_job("backfill").trigger.interval == timedelta(minutes=10)
|
||||
|
||||
|
||||
def test_every_job_has_an_interval_and_every_interval_a_job():
|
||||
assert set(scheduler.JOB_FUNCS) == set(scheduler.JOB_INTERVALS)
|
||||
|
||||
|
||||
def test_a_stored_interval_is_what_the_job_starts_with(db, monkeypatch):
|
||||
sysconfig.set_value(db, "job_interval_rss_poll", 25)
|
||||
db.commit()
|
||||
monkeypatch.setattr(scheduler.settings, "scheduler_enabled", True)
|
||||
scheduler.start_scheduler()
|
||||
try:
|
||||
sched = scheduler._scheduler
|
||||
assert sched is not None
|
||||
assert sched.get_job("rss_poll").trigger.interval == timedelta(minutes=25)
|
||||
default = scheduler.JOB_INTERVALS["enrich"]
|
||||
assert sched.get_job("enrich").trigger.interval == timedelta(minutes=default)
|
||||
finally:
|
||||
scheduler.shutdown_scheduler()
|
||||
|
||||
|
||||
def test_an_interval_set_on_the_scheduler_page_is_stored_and_reschedules_the_job(
|
||||
db, make_user, as_user, live_scheduler
|
||||
):
|
||||
live_scheduler.add_job(lambda: None, "interval", minutes=10, id="backfill")
|
||||
admin = as_user(make_user(role="admin"))
|
||||
r = admin.patch("/api/admin/scheduler/jobs/backfill", json={"interval_minutes": 45})
|
||||
assert r.status_code == 200, r.text
|
||||
assert live_scheduler.get_job("backfill").trigger.interval == timedelta(minutes=45)
|
||||
db.expire_all()
|
||||
assert sysconfig.get_int(db, "job_interval_backfill") == 45
|
||||
assert scheduler.load_intervals(db)["backfill"] == 45
|
||||
# The Configuration page neither lists an interval nor writes one past the reschedule.
|
||||
assert (
|
||||
admin.patch("/api/admin/config/job_interval_backfill", json={"value": 5}).status_code == 404
|
||||
)
|
||||
assert admin.delete("/api/admin/config/job_interval_backfill").status_code == 404
|
||||
assert "scheduler" not in admin.get("/api/admin/config").json()["groups"]
|
||||
db.expire_all()
|
||||
assert sysconfig.get_int(db, "job_interval_backfill") == 45
|
||||
|
||||
|
||||
def test_the_setup_wizard_does_not_write_an_interval(db):
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app.routes.setup import setup_config
|
||||
|
||||
with pytest.raises(HTTPException) as refused:
|
||||
setup_config({"job_interval_rss_poll": 5}, db)
|
||||
assert refused.value.status_code == 400
|
||||
assert setup_config({"quota_daily_budget": 5000}, db)["saved"] == ["quota_daily_budget"]
|
||||
|
||||
@@ -96,6 +96,9 @@ class TestDownloadFilename:
|
||||
def test_does_not_double_an_extension_already_present(self):
|
||||
assert download_filename("clip.mp4", "mp4", Path("/x/f.mp4")) == "clip.mp4"
|
||||
|
||||
def test_no_extension_is_served_without_a_trailing_dot(self):
|
||||
assert download_filename("Makefile", None, Path("/x/upload-7")) == "Makefile"
|
||||
|
||||
def test_falls_back_to_the_file_suffix_when_no_container(self):
|
||||
assert download_filename("song", None, Path("/x/f.m4a")) == "song.m4a"
|
||||
|
||||
|
||||
@@ -15,10 +15,14 @@ boot, sysconfig holds a stored override to it, and the admin page renders a pick
|
||||
The DB lane (`db` fixture) covers the store; the rest is pure logic.
|
||||
"""
|
||||
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app import sysconfig
|
||||
from app.models import SystemConfig
|
||||
from app.config import Settings, choices_of
|
||||
|
||||
CHOICE_KEYS = {s.key for s in sysconfig.SPECS if s.type == "choice"}
|
||||
@@ -37,7 +41,8 @@ def test_each_group_is_one_run_of_keys():
|
||||
def test_every_group_sits_in_exactly_one_section():
|
||||
listed = [g for _, members in sysconfig.SECTIONS for g in members]
|
||||
assert len(listed) == len(set(listed)), listed
|
||||
assert set(listed) == {s.group for s in sysconfig.SPECS}
|
||||
assert set(listed) | set(sysconfig.OWN_PAGE_GROUPS) == {s.group for s in sysconfig.SPECS}
|
||||
assert not set(listed) & set(sysconfig.OWN_PAGE_GROUPS)
|
||||
assert [sid for sid, _ in sysconfig.SECTIONS] == ["access", "youtube", "media", "system"]
|
||||
|
||||
|
||||
@@ -112,3 +117,78 @@ def test_set_value_stores_the_canonical_spelling_and_refuses_anything_else(db):
|
||||
with pytest.raises(ValueError, match="scrape, api"):
|
||||
sysconfig.set_value(db, "search_source", "bing")
|
||||
db.rollback()
|
||||
|
||||
|
||||
# --- when a changed value takes effect --------------------------------------------------------------
|
||||
|
||||
_APP = Path(__file__).resolve().parent.parent / "app"
|
||||
_READERS = {"get", "get_str", "get_int", "get_bool"}
|
||||
|
||||
|
||||
def _key_reads() -> list[tuple[str, str, bool]]:
|
||||
"""Every `sysconfig.get*(db, "<key>")` in the app outside the registry: (key, where, inside a
|
||||
function)."""
|
||||
found: list[tuple[str, str, bool]] = []
|
||||
|
||||
def walk(node: ast.AST, path: Path, in_fn: bool) -> None:
|
||||
for child in ast.iter_child_nodes(node):
|
||||
fn = in_fn or isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda))
|
||||
if (
|
||||
isinstance(child, ast.Call)
|
||||
and isinstance(child.func, ast.Attribute)
|
||||
and isinstance(child.func.value, ast.Name)
|
||||
and child.func.value.id == "sysconfig"
|
||||
and child.func.attr in _READERS
|
||||
and len(child.args) >= 2
|
||||
and isinstance(child.args[1], ast.Constant)
|
||||
and isinstance(child.args[1].value, str)
|
||||
):
|
||||
found.append((child.args[1].value, f"{path.name}:{child.lineno}", in_fn))
|
||||
walk(child, path, fn)
|
||||
|
||||
for path in _APP.rglob("*.py"):
|
||||
if path.name != "sysconfig.py":
|
||||
walk(ast.parse(path.read_text(encoding="utf-8")), path, False)
|
||||
return found
|
||||
|
||||
|
||||
def test_every_key_states_when_it_applies_and_none_needs_a_restart_yet():
|
||||
with pytest.raises(TypeError):
|
||||
sysconfig.ConfigSpec("k", "int", "quota", "quota_daily_budget") # pyright: ignore[reportCallIssue]
|
||||
# The Configuration page cannot tell the admin that a setting waits for a restart; a key that
|
||||
# does needs that first (a badge on it and a banner once it changed).
|
||||
assert {s.applies for s in sysconfig.SPECS} == {"live"}
|
||||
|
||||
|
||||
def test_a_live_key_is_read_where_it_is_used_never_at_import():
|
||||
reads = _key_reads()
|
||||
assert len(reads) > 50, len(
|
||||
reads
|
||||
) # the scan sees the reads (it would pass vacuously otherwise)
|
||||
assert [(k, at) for k, at, in_fn in reads if not in_fn] == []
|
||||
|
||||
|
||||
def test_every_key_on_the_configuration_page_has_a_reader():
|
||||
read = {k for k, _, _ in _key_reads()}
|
||||
page = {s.key for s in sysconfig.SPECS if s.group not in sysconfig.OWN_PAGE_GROUPS}
|
||||
assert sorted(page - read) == []
|
||||
|
||||
|
||||
def test_the_intervals_are_keys_of_the_scheduler_page_only():
|
||||
keys = {s.key for s in sysconfig.SPECS if s.group == "scheduler"}
|
||||
assert keys == {sysconfig.interval_key(j) for j in sysconfig.JOB_INTERVAL_DEFAULTS}
|
||||
assert all(len(k) <= 64 for k in keys) # system_config.key
|
||||
assert sysconfig.config_page_spec("job_interval_rss_poll") is None
|
||||
assert sysconfig.config_page_spec("quota_daily_budget") is not None
|
||||
assert "scheduler" not in sysconfig.describe(_FakeDb())["groups"]
|
||||
|
||||
|
||||
def test_saving_the_value_in_force_still_stores_the_override(db, make_user, as_user):
|
||||
client = as_user(make_user(role="admin"))
|
||||
default = sysconfig.get_int(db, "quota_daily_budget")
|
||||
r = client.patch("/api/admin/config/quota_daily_budget", json={"value": default})
|
||||
assert r.status_code == 200, r.text
|
||||
(item,) = [i for i in r.json()["groups"]["quota"] if i["key"] == "quota_daily_budget"]
|
||||
assert item["is_set"] is True
|
||||
db.expire_all()
|
||||
assert db.get(SystemConfig, "quota_daily_budget") is not None
|
||||
|
||||
+138
-126
@@ -44,6 +44,10 @@ runs this check before it is called done (E28 S3g):
|
||||
A rule that holds is not the same as a screen that works: the person page passed every rule above with
|
||||
half its panel empty.
|
||||
|
||||
A **copy-only** change — translated text and nothing else: no markup, class, component or logic — runs step 4
|
||||
alone: a screenshot of every state the changed text renders in, in Hungarian and in English (Hungarian runs about
|
||||
a fifth longer, so it is the one that wraps or clips). The key parity is the gate's.
|
||||
|
||||
## The design lab
|
||||
|
||||
`npm run dev`, then open **`http://localhost:5173/design-lab`**. The lab renders each open decision's variants
|
||||
@@ -126,7 +130,10 @@ style guide.
|
||||
to name one of the two. Black and white are the media scrims and their text. Five schemes × dark
|
||||
and light (B2): **Midnight** (blue), **Forest** (cyan), **Slate** (fuchsia), **Matrix** (moss) each earn their
|
||||
place with their own hue family; **Fjord** (Midnight's navy with a calm steel-blue accent) is the one kept
|
||||
exception, the calm twin of the vivid Midnight. A new scheme still needs its own hue family. Normal text ≥ 4.5:1 and a control's edge ≥ 3:1 in **every** scheme and mode.
|
||||
exception, the calm twin of the vivid Midnight. A new scheme still needs its own hue family. Normal text ≥ 4.5:1 and a control's edge ≥ 3:1 in **every** scheme and mode;
|
||||
`--muted` keeps ≥ 6:1 on the bare `--bg`, because it mostly sits on glass, which costs up to ~1.5 of the ratio.
|
||||
A tonal fill under text (a badge, the item count, the current rail row) is an opaque `bg-tint*` (its colour
|
||||
mixed into `--surface`), never a translucent `bg-fg/8`: the glass and what scrolls under it show through.
|
||||
[WCAG 1.4.3, 1.4.11] The input edge token is set under Controls (A4). The ≥ 3:1 edge is the field
|
||||
family's (a field, the form's dropdown trigger), whose edge is the only thing that says "type here". A
|
||||
labelled pill (a chip, a toolbar trigger, a tab, a segment) keeps the faint `--border`: its text names it,
|
||||
@@ -268,10 +275,9 @@ Also in force:
|
||||
- **Page chrome (B4):** a page's tabs and filters sit in ONE sticky band of the page colour, with the fade
|
||||
under it (below). Below ~600px of height the filter row folds into one "Filters · n" button. Every page has
|
||||
a visually hidden `h1`; the nav rail and the browser tab title are the visible identity. [WCAG 2.4.6]
|
||||
Built as: the band is `PageShell`'s toolbar slot (`PageToolbar`), above the one scroller; a console
|
||||
whose toolbar lives inside its frame (the server log) makes that toolbar `sticky` with `fade-below`
|
||||
instead. The band holds controls only: an intro paragraph scrolls with the content (in the band it
|
||||
ate three lines of a short window). The fold is `useFilterFold` + `FilterFoldButton`: the same row
|
||||
Built as: the band is `PageShell`'s toolbar slot (`PageToolbar`), above the one scroller. The band
|
||||
holds controls only: an intro paragraph scrolls with the content (in the band it ate three lines of a
|
||||
short window). The fold is `useFilterFold` + `FilterFoldButton`: the same row
|
||||
shown or hidden in place, never copied into a popover. The hidden `h1` is `PageShell`'s `title`; a
|
||||
view with its own visible title (a channel, a Plex show, a playlist) renders it as `PageHeading`
|
||||
and the hidden one steps aside, so a page has exactly one `h1` (a hand `<h1>` inside the shell is
|
||||
@@ -642,7 +648,7 @@ of a full-width panel and nothing on the right.
|
||||
`app/collation.py` on the server, so both sort alike.
|
||||
- **Hungarian words, not English ones** (E30 D3). Kept as Hungarian users say them: Feed, Shorts, Admin,
|
||||
Token, Port, and technical tokens (log levels, ffmpeg presets, remux). Everything else is Hungarian:
|
||||
közvetítés (stream), Prioritás, Feladat (job), Eseménynapló (audit), Demó, Automatikus, Infó,
|
||||
közvetítés (stream), Prioritás, Feladat (job), Napló (activity), Admin műveletek (audit), Demó, Automatikus, Infó,
|
||||
Felület / Szerver (frontend / backend), videók behúzása (backfill), átkódolás (transcode), e-mail.
|
||||
A longer Hungarian word gets its room — a column is sized to it, a pill truncates — never clipped.
|
||||
- A label says what happens ("Delete playlist"), not "OK"; a destructive confirm names the object.
|
||||
@@ -661,123 +667,129 @@ of a full-width panel and nothing on the right.
|
||||
|
||||
## Decision log
|
||||
|
||||
| Date | Decision | Chosen | Over | Why |
|
||||
| ---------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
|
||||
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
|
||||
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
|
||||
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
|
||||
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
|
||||
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
|
||||
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
|
||||
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
|
||||
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
|
||||
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
|
||||
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
|
||||
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
|
||||
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
|
||||
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
|
||||
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
|
||||
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
|
||||
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
|
||||
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
|
||||
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
|
||||
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
|
||||
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
|
||||
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
|
||||
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
|
||||
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
|
||||
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
|
||||
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
|
||||
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
|
||||
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
|
||||
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
|
||||
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
|
||||
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
|
||||
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
|
||||
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
|
||||
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
|
||||
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
|
||||
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
|
||||
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
|
||||
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
|
||||
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
|
||||
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
|
||||
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
|
||||
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
|
||||
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
|
||||
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
|
||||
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
|
||||
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
|
||||
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
|
||||
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
|
||||
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
|
||||
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
|
||||
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
|
||||
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
|
||||
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
|
||||
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
|
||||
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
|
||||
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
|
||||
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
|
||||
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
|
||||
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
|
||||
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
|
||||
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
|
||||
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
|
||||
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
|
||||
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
|
||||
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
|
||||
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
|
||||
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
|
||||
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
|
||||
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
|
||||
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
|
||||
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
|
||||
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
|
||||
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
|
||||
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
|
||||
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
|
||||
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
|
||||
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
|
||||
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
|
||||
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
|
||||
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
|
||||
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
|
||||
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
|
||||
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
|
||||
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
|
||||
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
|
||||
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
|
||||
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
|
||||
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
|
||||
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
|
||||
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
|
||||
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
|
||||
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
|
||||
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
|
||||
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
|
||||
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
|
||||
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
|
||||
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
|
||||
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
|
||||
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
|
||||
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
|
||||
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
|
||||
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
|
||||
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
|
||||
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
|
||||
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
|
||||
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
|
||||
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
|
||||
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
|
||||
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
|
||||
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
|
||||
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
|
||||
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
|
||||
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
|
||||
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
|
||||
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
|
||||
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
|
||||
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
|
||||
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
|
||||
| Date | Decision | Chosen | Over | Why |
|
||||
| ---------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
|
||||
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
|
||||
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
|
||||
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
|
||||
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
|
||||
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
|
||||
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
|
||||
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
|
||||
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
|
||||
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
|
||||
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
|
||||
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
|
||||
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
|
||||
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
|
||||
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
|
||||
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
|
||||
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
|
||||
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
|
||||
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
|
||||
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
|
||||
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
|
||||
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
|
||||
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
|
||||
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
|
||||
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
|
||||
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
|
||||
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
|
||||
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
|
||||
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
|
||||
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
|
||||
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
|
||||
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
|
||||
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
|
||||
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
|
||||
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
|
||||
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
|
||||
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
|
||||
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
|
||||
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
|
||||
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
|
||||
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
|
||||
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
|
||||
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
|
||||
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
|
||||
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
|
||||
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
|
||||
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
|
||||
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
|
||||
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
|
||||
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
|
||||
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
|
||||
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
|
||||
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
|
||||
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
|
||||
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
|
||||
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
|
||||
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
|
||||
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
|
||||
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
|
||||
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
|
||||
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
|
||||
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
|
||||
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
|
||||
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
|
||||
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
|
||||
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
|
||||
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
|
||||
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
|
||||
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
|
||||
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
|
||||
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
|
||||
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
|
||||
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
|
||||
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
|
||||
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
|
||||
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
|
||||
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
|
||||
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
|
||||
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
|
||||
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
|
||||
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
|
||||
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
|
||||
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
|
||||
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
|
||||
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
|
||||
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
|
||||
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
|
||||
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
|
||||
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
|
||||
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
|
||||
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
|
||||
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
|
||||
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
|
||||
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
|
||||
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
|
||||
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
|
||||
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
|
||||
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
|
||||
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
|
||||
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
|
||||
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
|
||||
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
|
||||
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
|
||||
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
|
||||
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
|
||||
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
|
||||
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
|
||||
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
|
||||
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
|
||||
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
|
||||
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
|
||||
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
|
||||
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
|
||||
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
|
||||
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
|
||||
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
|
||||
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
|
||||
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
|
||||
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
|
||||
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
|
||||
| 2026-10-17 | Copy-only changes (owner ★) | Step 4 of the Conformance check alone (not 1–3 or 5): every state the changed text renders in, seen in both languages | The whole five-step check; no screenshot at all (DOM test + key parity) | Steps 1–3 and 5 judge controls, patterns and composition, which a text change does not touch; the one risk left is a longer Hungarian string wrapping or clipping, and only a screenshot shows that |
|
||||
| 2026-10-17 | Text on glass (owner ★) | Light `--muted` 15% toward `--fg` in every scheme (≥ 6:1 on the bare page); the item count, the badges and the current Files row on opaque tints mixed into the surface | Only the two failing components; the item count's number in `--fg` | Pixel-measured over every scheme × mode × backdrop, the count fell to 3.8:1 (dark too: thumbnails scroll under the toolbar's glass) and Channels' tabs to 4.1:1; an `--fg` count outshouts the filters beside it |
|
||||
| 2026-10-18 | Scheduler run history (owner D4, E1, E2 ★) | A row's last three runs as dots, newest first (interrupted: a hollow warn ring, set apart from skipped and failed), and a history button that opens the job's runs on the Activity page; a user's ad-hoc runs in their own block under the jobs, with no interval and no run button | An inline expandable history per row; a dark-grey interrupted dot; ad-hoc runs only on the Activity page | Three dots answer "did it run, did it work" at a glance; the whole history needs filters, which the Activity page has |
|
||||
| 2026-10-18 | Activity page (owner D1, D2, G1–G3 ★) | One page, "Napló" / "Activity": four tabs (Runs, Admin actions, Server log, Quota) under one filter bar (text, a period of 1 h · 24 h · 7 d · 30 d · 90 d defaulting to 7 days, an account), the view in the URL; runs and admin actions as dense rows that open in place as a log line does, filtered and paged on the server; a run's or a request's log lines one pivot away, as a history entry | One merged timeline; a page per source; a default period per tab; card rows; the audit log's client-side table | Each source keeps its own table and retention, read in one place; one bar keeps what you narrowed across tabs; about 1.8k runs a day need a log's density and a server-side filter |
|
||||
|
||||
+13
-4
@@ -44,12 +44,11 @@ A change is done when this is green, from the repo root:
|
||||
bash scripts/check.sh # every lane
|
||||
bash scripts/check.sh lint # the static lanes only (no tests, build or audits) — the quick loop
|
||||
bash scripts/check.sh --brief # either mode: one line per lane, output only for a red one
|
||||
bash scripts/lint-changed.sh # ruff + eslint + prettier on the uncommitted files only
|
||||
bash scripts/check.sh --list # the lane names, one per line (runs nothing)
|
||||
bash scripts/check.sh --only "vitest" # one lane, by its exact name — never "the gate"
|
||||
```
|
||||
|
||||
It runs every lane even after one fails and names each red one at the end. `lint-changed.sh` is a
|
||||
convenience, not the gate; with `--hook` it doubles as a Claude Code Stop hook (silent when clean,
|
||||
the findings handed back otherwise).
|
||||
It runs every lane even after one fails and names each red one at the end.
|
||||
|
||||
The codebase's health numbers — size, complexity, test ratio, gate weight, comment noise — come from
|
||||
`python scripts/audit_metrics.py` (seconds; it reports, it never fails). `--speed` adds the speed
|
||||
@@ -85,6 +84,13 @@ docker compose -f backend/docker-compose.test.yml run --rm --no-deps test \
|
||||
pip-audit -r requirements.lock --no-deps --disable-pip
|
||||
```
|
||||
|
||||
**Shell scripts**
|
||||
|
||||
```bash
|
||||
# every tracked *.sh at shellcheck's default bar (info included); the version is pinned in scripts/check.sh
|
||||
git ls-files -z -- '*.sh' | xargs -0 shellcheck
|
||||
```
|
||||
|
||||
A missing database or ffmpeg is a hard error, not a skip — so the gate can't print green while the
|
||||
DB-backed tests were quietly not run (`SIFTLODE_TEST_REQUIRE_DB=0` is the explicit "pure-logic only"
|
||||
run for a machine without Postgres). Tests that need a database take the `db` fixture
|
||||
@@ -107,6 +113,9 @@ Coverage is on demand, never enforced and not in the gate (E35): `npm run test -
|
||||
`frontend/`, and for the backend
|
||||
`docker compose -f backend/docker-compose.test.yml run --rm test python -m pytest --cov=app` (then
|
||||
`… down -v`).
|
||||
The DB lane runs on every core (`pytest -n auto`, pytest-xdist): `tests/db_per_worker.py` gives each worker its own
|
||||
database (`siftlode_test_gw<N>`, created and migrated on first use), so the same tests run side by side. A serial
|
||||
run (no `-n`) still works and uses `siftlode_test` itself.
|
||||
The Playwright suite (`npm run e2e`) runs against the `:8080` stack built from your tree; the release
|
||||
requires it. Before an image is pushed, it is scanned with Trivy (HIGH/CRITICAL with a fix fails;
|
||||
exceptions in `.trivyignore.yaml`, each with a reason and an expiry).
|
||||
|
||||
@@ -44,7 +44,9 @@ The installer asks for the **public URL** where the instance will be reached (ju
|
||||
local `.env` file — keep that file private. **Do not change `TOKEN_ENCRYPTION_KEY` once the app has
|
||||
stored any secret** (Google credentials, API keys, SMTP password, Plex token): those values are
|
||||
encrypted with it, and a changed key makes them unreadable. If it is ever lost or rotated, clear the
|
||||
affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page,
|
||||
affected rows from the `system_config` table and re-enter the secrets in the admin Configuration page.
|
||||
Share links are kept with it too: after a key change every existing link still opens, but its
|
||||
address can no longer be shown — its owner makes a new link to share it again,
|
||||
- pulls the image and starts the app + database,
|
||||
- prints the **setup wizard URL**, which looks like `…/setup?token=…`.
|
||||
|
||||
@@ -163,7 +165,9 @@ then cannot read its own secrets.
|
||||
2. **`.env`** — in particular `TOKEN_ENCRYPTION_KEY`. Your Google client secret, SMTP password, Plex
|
||||
token and every stored refresh token are encrypted with it *inside* the database. Restore a dump
|
||||
against a different key and those values are lost — you would have to clear the affected rows from
|
||||
`system_config` and re-enter every secret by hand. Keep `.env` beside the dump, and just as private.
|
||||
`system_config` and re-enter every secret by hand — and no existing share link's address can be
|
||||
shown again (the links still open; their owners make new ones to share). Keep `.env` beside the
|
||||
dump, and just as private.
|
||||
3. **Downloaded media** *(only if you use Files)* — the `siftlode_files` volume, or
|
||||
whatever `FILES_HOST_PATH` points at. Not needed to bring the instance back; the jobs and their
|
||||
metadata are in the database, the files themselves are not.
|
||||
|
||||
@@ -45,6 +45,11 @@ login-free `/watch/<token>` share link).
|
||||
PW_BASE_URL=http://localhost:5173 siftlode e2e
|
||||
```
|
||||
|
||||
**Accessibility** — `a11y.spec.ts` runs axe-core's WCAG 2.x A + AA rules (`@axe-core/playwright`)
|
||||
over every rail page, every admin page and the player, in `chromium` only (the rules read the DOM,
|
||||
which the engines build alike). It does not judge colour contrast: axe files text on a gradient —
|
||||
every glass surface — as "incomplete", not as a pass or a hit.
|
||||
|
||||
## Account & fixture
|
||||
|
||||
Tests run as a dedicated **admin** account `e2e@siftlode.local` (never a real account;
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
import AxeBuilder from "@axe-core/playwright";
|
||||
import { test, expect, type Page } from "@playwright/test";
|
||||
import { gotoPage, openApp, openFirstVideo } from "./helpers";
|
||||
|
||||
// Accessibility, the machine-checkable half: axe-core's WCAG 2.x A + AA rules over every page the
|
||||
// owner reaches and the player. A hit fails with the rule, its impact and the elements.
|
||||
//
|
||||
// What it does NOT cover — colour contrast, mostly: axe cannot compute a contrast over a gradient,
|
||||
// and the glass surfaces are gradients, so it files most text as "incomplete" instead of judging it
|
||||
// (measured 2026-10-17: 110 nodes on the feed, 9-32 on the other pages). A green here says nothing
|
||||
// about contrast; that stays with the visual check.
|
||||
//
|
||||
// Chromium only (playwright.config.ts leaves it out of the other projects).
|
||||
const TAGS = ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"];
|
||||
|
||||
/** Wait until nothing is loading: no QueryState spinner and no skeleton bone left. A bone says it is
|
||||
* one (`data-skeleton`): a pulse alone is not loading — a running job's dot pulses as long as it runs. */
|
||||
async function settled(page: Page): Promise<void> {
|
||||
await expect(page.locator('[role="status"] .animate-spin, [data-skeleton]')).toHaveCount(0);
|
||||
}
|
||||
|
||||
async function violations(page: Page): Promise<string[]> {
|
||||
const result = await new AxeBuilder({ page }).withTags(TAGS).analyze();
|
||||
return result.violations.map((v) => {
|
||||
const shown = v.nodes.slice(0, 5).map((n) => n.target.join(" "));
|
||||
const more = v.nodes.length > 5 ? ` (+${v.nodes.length - 5})` : "";
|
||||
return `${v.id} (${v.impact ?? "?"}): ${v.help} — ${shown.join(" | ")}${more}`;
|
||||
});
|
||||
}
|
||||
|
||||
test.describe("accessibility (axe)", () => {
|
||||
const RAIL = ["feed", "channels", "playlists", "files", "stats", "settings"];
|
||||
const ADMIN = ["scheduler", "config", "users", "activity"];
|
||||
|
||||
for (const id of RAIL) {
|
||||
test(`page: ${id}`, async ({ page }) => {
|
||||
await openApp(page);
|
||||
await gotoPage(page, id);
|
||||
await settled(page);
|
||||
expect(await violations(page)).toEqual([]);
|
||||
});
|
||||
}
|
||||
|
||||
for (const id of ADMIN) {
|
||||
test(`admin page: ${id}`, async ({ page }) => {
|
||||
await openApp(page);
|
||||
await page.getByTestId("nav-admin").click();
|
||||
await page.getByTestId(`nav-admin-${id}`).click();
|
||||
await expect(page.getByTestId("nav-admin")).toHaveAttribute("aria-current", "page");
|
||||
await settled(page);
|
||||
expect(await violations(page)).toEqual([]);
|
||||
});
|
||||
}
|
||||
|
||||
test("player", async ({ page }) => {
|
||||
// hermetic: the YouTube iframe is not loaded
|
||||
await page.route(/youtube\.com/, (route) => route.abort());
|
||||
await openApp(page);
|
||||
await openFirstVideo(page);
|
||||
await settled(page);
|
||||
expect(await violations(page)).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -11,7 +11,7 @@ test.describe("page switching", () => {
|
||||
// and it failed on every run for a while without anyone seeing it, because e2e was in no
|
||||
// required gate (it is now part of `release`).
|
||||
const RAIL = ["channels", "playlists", "files", "stats", "settings", "feed"];
|
||||
const ADMIN = ["scheduler", "config", "users", "audit"];
|
||||
const ADMIN = ["scheduler", "config", "users", "activity"];
|
||||
|
||||
test("each rail item activates its page", async ({ page }) => {
|
||||
await openApp(page);
|
||||
|
||||
@@ -17,7 +17,7 @@ import { retryConnect } from "./retryConnect";
|
||||
|
||||
const RUNS = 5;
|
||||
const OUT = "e2e/.speed/latest.json";
|
||||
const CONTAINER = process.env.E2E_DB_CONTAINER || "siftlode-db-1";
|
||||
const CONTAINER = process.env.E2E_API_CONTAINER || "siftlode-api-1";
|
||||
|
||||
const median = (xs: number[]) => [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)] ?? NaN;
|
||||
|
||||
@@ -32,20 +32,33 @@ const scriptBytes = (page: Page) =>
|
||||
|
||||
/** Public share links on the local stack that could play: no password, not expired, asset ready.
|
||||
* The page itself is the judge of the last step (its file must be on THIS disk — a localdev DB
|
||||
* seeded from prod lists assets whose files never came along), so the caller probes them. */
|
||||
* seeded from prod lists assets whose files never came along), so the caller probes them. Read
|
||||
* through the API container: the database keeps a link's token only hashed and encrypted, and only
|
||||
* the app holds the key to read it back (`links.recover`). */
|
||||
const SHARE_TOKENS_PY = `
|
||||
from sqlalchemy import select
|
||||
from app.db import SessionLocal
|
||||
from app.downloads import links
|
||||
from app.models import DownloadJob, DownloadLink, MediaAsset
|
||||
with SessionLocal() as db:
|
||||
for link in db.execute(
|
||||
select(DownloadLink)
|
||||
.join(DownloadJob, DownloadJob.id == DownloadLink.job_id)
|
||||
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
|
||||
.where(DownloadLink.password_hash.is_(None), MediaAsset.status == "ready")
|
||||
.order_by(DownloadLink.id)
|
||||
).scalars():
|
||||
token = None if links.is_expired(link) else links.recover(link)
|
||||
if token:
|
||||
print(token)
|
||||
`;
|
||||
|
||||
function shareTokens(): string[] {
|
||||
const sql =
|
||||
"select l.token from download_links l join download_jobs j on j.id = l.job_id" +
|
||||
" join media_assets a on a.id = j.asset_id" +
|
||||
" where l.password_hash is null and (l.expires_at is null or l.expires_at > now())" +
|
||||
" and a.status = 'ready' order by l.id";
|
||||
// Not caught: a database this cannot reach (no docker, a renamed container) is a failed
|
||||
// Not caught: an API container this cannot reach (no docker, a renamed container) is a failed
|
||||
// measurement, not "no playable share link on the local stack".
|
||||
const out = execFileSync(
|
||||
"docker",
|
||||
["exec", CONTAINER, "psql", "-U", "siftlode", "-d", "siftlode", "-tAc", sql],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
const out = execFileSync("docker", ["exec", CONTAINER, "python", "-c", SHARE_TOKENS_PY], {
|
||||
encoding: "utf8",
|
||||
});
|
||||
return out
|
||||
.split("\n")
|
||||
.map((t) => t.trim())
|
||||
|
||||
@@ -627,6 +627,13 @@ export default tseslint.config(
|
||||
message:
|
||||
"No native <select>: use ui/SegmentedToggle for a short fixed set, ui/FilterSelect for a longer one or a user's own list.",
|
||||
},
|
||||
// A picture that failed to load left the browser's broken-image mark on a card: a bare <img>
|
||||
// sends a Referer Google's image CDNs refuse, and has no retry and no fallback.
|
||||
{
|
||||
selector: "JSXOpeningElement[name.name='img']",
|
||||
message:
|
||||
"No bare <img>: use ui/RemoteImage — no Referer, a retry past the browser's negative cache, and a fallback instead of the broken-image mark.",
|
||||
},
|
||||
// Keep the rem system from rotting. A px-frozen `text-[Npx]` font size does NOT
|
||||
// scale with the Settings --font-scale slider while its rem neighbours do; use
|
||||
// `text-2xs`/`text-3xs` or a `text-[Nrem]` arbitrary instead. Matches both plain strings and
|
||||
|
||||
Generated
+27
-3
@@ -23,6 +23,7 @@
|
||||
"react-i18next": "^15.7.4"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@axe-core/playwright": "4.13.0",
|
||||
"@babel/core": "7.29.7",
|
||||
"@eslint-community/eslint-plugin-eslint-comments": "4.8.1",
|
||||
"@eslint/js": "9.39.1",
|
||||
@@ -91,6 +92,19 @@
|
||||
"dev": true,
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/@axe-core/playwright": {
|
||||
"version": "4.13.0",
|
||||
"resolved": "https://registry.npmjs.org/@axe-core/playwright/-/playwright-4.13.0.tgz",
|
||||
"integrity": "sha512-6YLx+kxXu5GJceG4ozFg+33a2EMTdjYwWGloJ3sb9Kta5pp+ZNS53uxGVog5JetIY8s++P5UrtX+cri+u0VAVg==",
|
||||
"dev": true,
|
||||
"license": "MPL-2.0",
|
||||
"dependencies": {
|
||||
"axe-core": "~4.13.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"playwright-core": ">= 1.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/code-frame": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.7.tgz",
|
||||
@@ -3631,6 +3645,16 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/axe-core": {
|
||||
"version": "4.13.0",
|
||||
"resolved": "https://registry.npmjs.org/axe-core/-/axe-core-4.13.0.tgz",
|
||||
"integrity": "sha512-UzGt8zg7Ny8djbYMhxl2zuEevVa7r2gJjYY5Lwr1xM7+XU2nd6CkIWFTVcCIbAP63vSz71NaVyyuSk9lHKcy0A==",
|
||||
"dev": true,
|
||||
"license": "MPL-2.0",
|
||||
"engines": {
|
||||
"node": ">=4"
|
||||
}
|
||||
},
|
||||
"node_modules/babel-plugin-react-compiler": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/babel-plugin-react-compiler/-/babel-plugin-react-compiler-1.0.0.tgz",
|
||||
@@ -6163,9 +6187,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/source-map-js": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
|
||||
"integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
|
||||
"version": "1.2.2",
|
||||
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz",
|
||||
"integrity": "sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"engines": {
|
||||
|
||||
@@ -39,6 +39,7 @@
|
||||
"react-i18next": "^15.7.4"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@axe-core/playwright": "4.13.0",
|
||||
"@babel/core": "7.29.7",
|
||||
"@eslint-community/eslint-plugin-eslint-comments": "4.8.1",
|
||||
"@eslint/js": "9.39.1",
|
||||
|
||||
@@ -17,6 +17,9 @@ import { defineConfig, devices } from "@playwright/test";
|
||||
|
||||
const BASE_URL = process.env.PW_BASE_URL || "http://127.0.0.1:8080";
|
||||
const STORAGE_STATE = "e2e/.auth/state.json";
|
||||
// The axe scan reads the DOM and the accessibility tree, which the engines build alike: one engine
|
||||
// answers for all, so the other projects leave it out instead of tripling its time.
|
||||
const CHROMIUM_ONLY = /a11y\.spec\.ts/;
|
||||
|
||||
export default defineConfig({
|
||||
testDir: "./e2e",
|
||||
@@ -54,11 +57,13 @@ export default defineConfig({
|
||||
{
|
||||
name: "webkit",
|
||||
use: { ...devices["Desktop Safari"], storageState: STORAGE_STATE },
|
||||
testIgnore: CHROMIUM_ONLY,
|
||||
dependencies: ["setup"],
|
||||
},
|
||||
{
|
||||
name: "ipad",
|
||||
use: { ...devices["iPad (gen 7) landscape"], storageState: STORAGE_STATE },
|
||||
testIgnore: CHROMIUM_ONLY,
|
||||
dependencies: ["setup"],
|
||||
},
|
||||
// The speed baseline (e2e/speed.measure.ts): readings, not tests. Only when PW_SPEED is set —
|
||||
|
||||
@@ -16,7 +16,7 @@ import {
|
||||
type PanelLayout,
|
||||
} from "./lib/panelLayout";
|
||||
import { notify, removeByMetaKind } from "./lib/notifications";
|
||||
import { LS, setTabRaw } from "./lib/storage";
|
||||
import { LS } from "./lib/storage";
|
||||
import { usePref } from "./lib/usePref";
|
||||
import { useVerifyEntry } from "./lib/useVerifyEntry";
|
||||
import { useNavigation, useNavigationActions } from "./components/navigationContext";
|
||||
@@ -46,6 +46,7 @@ import ErrorDialog from "./components/ErrorDialog";
|
||||
import VersionBanner from "./components/VersionBanner";
|
||||
import VerifyBanner from "./components/VerifyBanner";
|
||||
import { focusAccessRequestsTab } from "./lib/adminUsersTab";
|
||||
import { focusSettingsTab } from "./lib/settingsTab";
|
||||
// The built app version, NOT imported from releaseNotes — importing anything from that module
|
||||
// pulls the release-notes loader into the eager App chunk and defeats the lazy ReleaseNotes
|
||||
// split. FRONTEND_VERSION equals the newest release note's version by release convention (both bump together).
|
||||
@@ -261,7 +262,7 @@ export default function App() {
|
||||
if (result === "ok") {
|
||||
notify({ level: "success", message: t("settings.account.googleLink.linked") });
|
||||
void refetchMe();
|
||||
setTabRaw(LS.settingsTab, "account");
|
||||
focusSettingsTab("account");
|
||||
setPage("settings");
|
||||
} else {
|
||||
const key = result === "conflict" || result === "mismatch" ? result : "error";
|
||||
|
||||
@@ -9,13 +9,13 @@ import { displayName } from "../lib/api/account";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { notifyActionError } from "../lib/actionErrors";
|
||||
import { copyText } from "../lib/clipboard";
|
||||
import { formatDateTime } from "../lib/format";
|
||||
import { invalidateUserAccess } from "../lib/adminUsers";
|
||||
import type { AdminUserAccess, AdminUserRow } from "../lib/api/admin";
|
||||
import { Button } from "./ui/Button";
|
||||
import { SURFACE } from "./ui/surfaces";
|
||||
import { SectionLabel } from "./ui/Headings";
|
||||
import { useCopyFeedback } from "./useCopyFeedback";
|
||||
|
||||
// The admin's answer to "why can this person not get in?", and the place every recovery action for
|
||||
// one account lives.
|
||||
@@ -59,7 +59,7 @@ export default function AccountPanel({
|
||||
const lang = i18n.language;
|
||||
const qc = useQueryClient();
|
||||
const [link, setLink] = useState<{ url: string; expires_at: string | null } | null>(null);
|
||||
const [copied, setCopied] = useState(false);
|
||||
const { copied, copy: copyValue } = useCopyFeedback();
|
||||
|
||||
const q = useQuery({
|
||||
queryKey: qk.adminUserAccess(user.id),
|
||||
@@ -88,19 +88,11 @@ export default function AccountPanel({
|
||||
onError: (err) => notifyActionError(err, t("users.pendingAccess.activateFailed")),
|
||||
});
|
||||
|
||||
const copy = async () => {
|
||||
if (!link) return;
|
||||
// Via the shared helper, whose synchronous `execCommand` path is what makes copying work on a
|
||||
// plain-http LAN origin — where `navigator.clipboard` is undefined, and which is exactly the
|
||||
// minimal self-host this no-email link was built for.
|
||||
if (await copyText(link.url)) {
|
||||
setCopied(true);
|
||||
window.setTimeout(() => setCopied(false), 2000);
|
||||
return;
|
||||
}
|
||||
// The link is on screen and selectable, so a failed copy is a convenience that failed — not the
|
||||
// action failing. Say so rather than leaving the button looking inert.
|
||||
notify({ level: "warning", message: t("users.account.copyFailed") });
|
||||
// Via the shared helper, whose synchronous `execCommand` path is what makes copying work on a
|
||||
// plain-http LAN origin — where `navigator.clipboard` is undefined, and which is exactly the
|
||||
// minimal self-host this no-email link was built for.
|
||||
const copy = () => {
|
||||
if (link) void copyValue(link.url);
|
||||
};
|
||||
|
||||
const a: AdminUserAccess | undefined = q.data;
|
||||
|
||||
@@ -0,0 +1,446 @@
|
||||
// @vitest-environment jsdom
|
||||
//
|
||||
// The activity page around its run and audit lists (the server-log tab has its own file): the view
|
||||
// is the URL — a link narrows the runs to one job, a status chip narrows them further; a run opens
|
||||
// to its log lines and an admin action to its request's, each a new history entry that Back undoes;
|
||||
// the account filter reaches the quota tab; a
|
||||
// failed read is an error with a retry, never an empty list.
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { act, cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
|
||||
import type { AuditQuery, AuditRow, HistoryPage, LogQuery, RunQuery, RunRow } from "../lib/api";
|
||||
|
||||
vi.mock("react-i18next", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("react-i18next")>();
|
||||
return {
|
||||
...actual,
|
||||
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
|
||||
};
|
||||
});
|
||||
|
||||
const runsMock = vi.fn<(q: RunQuery, before: string | null) => Promise<HistoryPage<RunRow>>>();
|
||||
const auditMock = vi.fn<(q: AuditQuery, before: string | null) => Promise<HistoryPage<AuditRow>>>();
|
||||
const logsMock = vi.fn<(q: LogQuery, before: string | null) => Promise<unknown>>();
|
||||
const quotaMock = vi.fn<(since: string, user: string | null) => Promise<unknown>>();
|
||||
const runFacetsMock = vi.fn<() => Promise<unknown>>();
|
||||
const usersMock = vi.fn<() => Promise<unknown>>();
|
||||
vi.mock("../lib/api", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../lib/api")>();
|
||||
return {
|
||||
...actual,
|
||||
api: {
|
||||
...actual.api,
|
||||
adminRuns: (q: RunQuery, before: string | null) => runsMock(q, before),
|
||||
adminRunFacets: () => runFacetsMock(),
|
||||
adminAudit: (q: AuditQuery, before: string | null) => auditMock(q, before),
|
||||
adminAuditFacets: () => Promise.resolve({ actions: [] }),
|
||||
adminLogs: (q: LogQuery, before: string | null) => logsMock(q, before),
|
||||
adminLogFacets: () =>
|
||||
Promise.resolve({ levels: [], sources: [], events: [], users: [], accounts: {}, total: 0 }),
|
||||
adminUsers: () => usersMock(),
|
||||
adminQuota: (since: string, user: string | null) => quotaMock(since, user),
|
||||
clearAudit: () => Promise.resolve({ ok: true, cleared: 1 }),
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import Activity from "./Activity";
|
||||
import { ChoiceContext } from "./confirmContext";
|
||||
import { PageToolbarSlot } from "./ui/pageToolbarSlot";
|
||||
|
||||
// The filter bar lives in the toolbar, which portals into the shell's band: give it one.
|
||||
const slot = document.body.appendChild(document.createElement("div"));
|
||||
|
||||
const RUN: RunRow = {
|
||||
id: 41,
|
||||
job: "backfill",
|
||||
trigger: "manual",
|
||||
actor_id: 1,
|
||||
target: null,
|
||||
target_title: null,
|
||||
status: "error",
|
||||
started_at: "2026-10-18T02:31:04Z",
|
||||
finished_at: "2026-10-18T02:31:45Z",
|
||||
summary: null,
|
||||
result: null,
|
||||
error: "quotaExceeded",
|
||||
failure: null,
|
||||
quota_units: 212,
|
||||
log_lines: 9,
|
||||
};
|
||||
|
||||
const ACTION: AuditRow = {
|
||||
id: 5,
|
||||
created_at: "2026-10-18T02:27:00Z",
|
||||
actor_id: 1,
|
||||
action: "scheduler_interval",
|
||||
target_type: "job",
|
||||
target_id: "rss_poll",
|
||||
summary: "rss_poll → 25 min/run",
|
||||
before: null,
|
||||
after: { interval_minutes: 25 },
|
||||
request_id: "0efdd960c9ad1977",
|
||||
log_lines: 4,
|
||||
};
|
||||
|
||||
const page = <T,>(items: T[]): HistoryPage<T> => ({
|
||||
items,
|
||||
accounts: { "1": "admin@example.com" },
|
||||
next_cursor: null,
|
||||
});
|
||||
|
||||
function renderPage(qc = new QueryClient({ defaultOptions: { queries: { retry: false } } })) {
|
||||
return render(
|
||||
<QueryClientProvider client={qc}>
|
||||
{/* Every confirmation is answered yes. */}
|
||||
<ChoiceContext.Provider value={() => Promise.resolve("confirm")}>
|
||||
<PageToolbarSlot.Provider value={slot}>
|
||||
<Activity />
|
||||
</PageToolbarSlot.Provider>
|
||||
</ChoiceContext.Provider>
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
}
|
||||
|
||||
const url = () => new URLSearchParams(window.location.search);
|
||||
const rowButton = (text: RegExp) =>
|
||||
screen
|
||||
.getAllByRole("button")
|
||||
.find((b) => b.hasAttribute("aria-expanded") && text.test(b.textContent ?? ""))!;
|
||||
|
||||
beforeEach(() => {
|
||||
runsMock.mockResolvedValue(page([RUN]));
|
||||
auditMock.mockResolvedValue(page([ACTION]));
|
||||
logsMock.mockResolvedValue({ items: [], accounts: {}, next_cursor: null });
|
||||
quotaMock.mockResolvedValue({ rows: [], daily: [] });
|
||||
runFacetsMock.mockResolvedValue({ jobs: [], statuses: [], total: 0 });
|
||||
usersMock.mockResolvedValue([{ id: 1, email: "admin@example.com" }]);
|
||||
});
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
describe("Activity", () => {
|
||||
it("a link narrows the runs to one job, and a status chip narrows them further", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity&rjob=backfill");
|
||||
renderPage();
|
||||
await waitFor(() =>
|
||||
expect(runsMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ job: "backfill", status: undefined, since: expect.any(String) }),
|
||||
null,
|
||||
),
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: /activity\.runs\.status\.error/ }));
|
||||
|
||||
await waitFor(() =>
|
||||
expect(runsMock).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ job: "backfill", status: "error" }),
|
||||
null,
|
||||
),
|
||||
);
|
||||
expect([url().get("rjob"), url().get("rst")]).toEqual(["backfill", "error"]);
|
||||
});
|
||||
|
||||
it("Older reads the next page from its cursor, and offers none past the last", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity");
|
||||
runsMock.mockImplementation((_q, before) =>
|
||||
Promise.resolve(
|
||||
before === null
|
||||
? { ...page([RUN]), next_cursor: "c1" }
|
||||
: page([{ ...RUN, id: 40, error: "the older one" }]),
|
||||
),
|
||||
);
|
||||
renderPage();
|
||||
fireEvent.click(await screen.findByRole("button", { name: "activity.older" }));
|
||||
await waitFor(() => expect(screen.getByText("the older one")).toBeTruthy());
|
||||
expect(runsMock).toHaveBeenLastCalledWith(expect.anything(), "c1");
|
||||
expect(screen.getByText("quotaExceeded")).toBeTruthy();
|
||||
expect(screen.queryByRole("button", { name: "activity.older" })).toBeNull();
|
||||
});
|
||||
|
||||
it("an older page that fails says so and keeps what was loaded", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity");
|
||||
runsMock.mockImplementation((_q, before) =>
|
||||
before === null
|
||||
? Promise.resolve({ ...page([RUN]), next_cursor: "c1" })
|
||||
: Promise.reject(new Error("500")),
|
||||
);
|
||||
renderPage();
|
||||
fireEvent.click(await screen.findByRole("button", { name: "activity.older" }));
|
||||
const alert = await screen.findByRole("alert");
|
||||
expect(alert.textContent).toContain("activity.olderFailed");
|
||||
expect(screen.getByText("quotaExceeded")).toBeTruthy();
|
||||
});
|
||||
|
||||
it("offers no older page while a filter change is still loading", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity");
|
||||
// The cursor on screen is the previous filter's: it would continue the wrong list.
|
||||
runsMock.mockImplementation((q) =>
|
||||
q.status ? new Promise(() => {}) : Promise.resolve({ ...page([RUN]), next_cursor: "c1" }),
|
||||
);
|
||||
renderPage();
|
||||
await screen.findByRole("button", { name: "activity.older" });
|
||||
fireEvent.click(screen.getByRole("button", { name: /activity\.runs\.status\.error/ }));
|
||||
await waitFor(() =>
|
||||
expect(screen.queryByRole("button", { name: "activity.older" })).toBeNull(),
|
||||
);
|
||||
expect(screen.getByText("quotaExceeded")).toBeTruthy(); // the old rows stay, dimmed
|
||||
});
|
||||
|
||||
it("a run opens to its log lines, and Back returns to the runs", async () => {
|
||||
window.history.replaceState({ sfPage: "activity" }, "", "/?page=activity&aw=24h&aq=quota");
|
||||
renderPage();
|
||||
await waitFor(() => expect(screen.getByText("quotaExceeded")).toBeTruthy());
|
||||
fireEvent.click(rowButton(/quotaExceeded/));
|
||||
fireEvent.click(screen.getByRole("button", { name: "activity.runs.logLines" }));
|
||||
|
||||
await waitFor(() =>
|
||||
expect(logsMock).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ run_id: 41, q: undefined }),
|
||||
null,
|
||||
),
|
||||
);
|
||||
// Every other filter is off, and the range is the log's whole retention.
|
||||
expect([url().get("at"), url().get("lrun"), url().get("aw"), url().get("aq")]).toEqual([
|
||||
"logs",
|
||||
"41",
|
||||
"90d",
|
||||
null,
|
||||
]);
|
||||
|
||||
await act(async () => {
|
||||
const landed = new Promise((resolve) =>
|
||||
window.addEventListener("popstate", resolve, { once: true }),
|
||||
);
|
||||
window.history.back();
|
||||
await landed;
|
||||
});
|
||||
await waitFor(() => expect(url().get("at")).toBeNull());
|
||||
expect([url().get("aw"), url().get("aq"), url().get("lrun")]).toEqual(["24h", "quota", null]);
|
||||
expect(screen.getByText("quotaExceeded")).toBeTruthy();
|
||||
});
|
||||
|
||||
it("a run's outcome is its result in the reader's words, its failure in the danger colour", async () => {
|
||||
runsMock.mockResolvedValue(
|
||||
page<RunRow>([
|
||||
{ ...RUN, id: 1, status: "ok", error: null, result: { new: 3 }, summary: "new=3" },
|
||||
// Moved over from the audit log before the result rode along as data: the one-liner only.
|
||||
{ ...RUN, id: 2, status: "ok", error: null, result: null, summary: "failed=2" },
|
||||
{ ...RUN, id: 3, status: "interrupted", error: null },
|
||||
{ ...RUN, id: 4 },
|
||||
]),
|
||||
);
|
||||
renderPage();
|
||||
await waitFor(() => expect(screen.getByText("scheduler.result.new: 3")).toBeTruthy());
|
||||
expect(screen.queryByText("new=3")).toBeNull();
|
||||
expect(screen.getByText("scheduler.result.failed: 2")).toBeTruthy();
|
||||
expect(screen.getByText("scheduler.interrupted").className).toContain("text-warn");
|
||||
expect(screen.getByText("quotaExceeded").className).toContain("text-danger");
|
||||
});
|
||||
|
||||
it("filter choices that failed to load say so beside the filters, with a retry", async () => {
|
||||
runFacetsMock.mockRejectedValueOnce(new Error("500"));
|
||||
usersMock.mockRejectedValueOnce(new Error("500"));
|
||||
renderPage();
|
||||
await waitFor(() =>
|
||||
expect(screen.getAllByRole("alert").map((a) => a.textContent)).toEqual([
|
||||
expect.stringContaining("activity.accountsFailed"),
|
||||
expect.stringContaining("activity.filtersFailed"),
|
||||
]),
|
||||
);
|
||||
for (const retry of screen.getAllByRole("button", { name: "common.retry" }))
|
||||
fireEvent.click(retry);
|
||||
await waitFor(() => expect(screen.queryAllByRole("alert")).toEqual([]));
|
||||
expect(runFacetsMock).toHaveBeenCalledTimes(2);
|
||||
expect(usersMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("accounts that could not be refreshed, still listed, are not called unloaded", async () => {
|
||||
const qc = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: 0 } } });
|
||||
qc.setQueryData(qk.adminUsers(), [{ id: 1, email: "admin@example.com" }]);
|
||||
usersMock.mockRejectedValue(new Error("502"));
|
||||
renderPage(qc);
|
||||
await waitFor(() => expect(usersMock).toHaveBeenCalled());
|
||||
await waitFor(() => expect(qc.getQueryState(qk.adminUsers())?.status).toBe("error"));
|
||||
expect(screen.queryByText("activity.accountsFailed")).toBeNull();
|
||||
});
|
||||
|
||||
it("clearing the admin actions reads the list again", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity&at=admin");
|
||||
renderPage();
|
||||
await waitFor(() => expect(screen.getByText("auditActions.scheduler_interval")).toBeTruthy());
|
||||
auditMock.mockResolvedValue(page<AuditRow>([]));
|
||||
fireEvent.click(screen.getByRole("button", { name: "audit.clear" }));
|
||||
await waitFor(() => expect(screen.getByText("audit.empty")).toBeTruthy());
|
||||
// An emptied list is announced, as the error and loading states are.
|
||||
expect(screen.getByText("audit.empty").closest('[role="status"]')).toBeTruthy();
|
||||
});
|
||||
|
||||
it("closing a dialog over the server log leaves Live on", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity&at=logs");
|
||||
renderPage();
|
||||
const live = await screen.findByRole("button", { name: "adminLogs.live" });
|
||||
fireEvent.click(live);
|
||||
expect(live.getAttribute("aria-pressed")).toBe("true");
|
||||
// A closing dialog steps back over its own history entry: a popstate onto the same address.
|
||||
act(() => {
|
||||
window.dispatchEvent(new PopStateEvent("popstate"));
|
||||
});
|
||||
const after = screen.getByRole("button", { name: "adminLogs.live" });
|
||||
expect(after.getAttribute("aria-pressed")).toBe("true");
|
||||
expect(after).toBe(live); // the same log, not a fresh one
|
||||
});
|
||||
|
||||
it("Back/Forward on the server log reads only the filters of the entry it lands on", async () => {
|
||||
// A browser runs the microtasks between two popstate listeners, so each listener's update
|
||||
// renders on its own; a test dispatch does not. Each listener runs in its own act here.
|
||||
const listeners: EventListener[] = [];
|
||||
const addOther = window.addEventListener.bind(window);
|
||||
const removeOther = window.removeEventListener.bind(window);
|
||||
const add = vi.spyOn(window, "addEventListener").mockImplementation((type, l, o) => {
|
||||
if (type === "popstate") listeners.push(l as EventListener);
|
||||
else addOther(type, l, o);
|
||||
});
|
||||
const remove = vi.spyOn(window, "removeEventListener").mockImplementation((type, l, o) => {
|
||||
if (type === "popstate") listeners.splice(listeners.indexOf(l as EventListener), 1);
|
||||
else removeOther(type, l, o);
|
||||
});
|
||||
window.history.replaceState(null, "", "/?page=activity&at=logs&aw=24h");
|
||||
renderPage();
|
||||
await waitFor(() => expect(logsMock).toHaveBeenCalled());
|
||||
logsMock.mockClear();
|
||||
// Forward onto a pivot's entry: one run's lines over the log's whole retention.
|
||||
window.history.pushState(null, "", "/?page=activity&at=logs&aw=90d&lrun=41");
|
||||
for (const l of [...listeners]) {
|
||||
await act(async () => l(new PopStateEvent("popstate")));
|
||||
}
|
||||
await waitFor(() =>
|
||||
expect(logsMock).toHaveBeenLastCalledWith(expect.objectContaining({ run_id: 41 }), null),
|
||||
);
|
||||
const days = (since?: string) => (Date.now() - Date.parse(since ?? "")) / 86_400_000;
|
||||
// Never one run's lines over a day, nor the whole log over 90 days.
|
||||
expect(logsMock.mock.calls.map(([q]) => [q.run_id ?? null, days(q.since) > 30])).toEqual(
|
||||
logsMock.mock.calls.map(() => [41, true]),
|
||||
);
|
||||
add.mockRestore();
|
||||
remove.mockRestore();
|
||||
});
|
||||
|
||||
it("an action whose request's lines are gone says so instead of linking to an empty list", async () => {
|
||||
auditMock.mockResolvedValue(page([{ ...ACTION, log_lines: 0 }]));
|
||||
window.history.replaceState(null, "", "/?page=activity&at=admin");
|
||||
renderPage();
|
||||
await waitFor(() => expect(screen.getByText("auditActions.scheduler_interval")).toBeTruthy());
|
||||
fireEvent.click(rowButton(/scheduler_interval/));
|
||||
expect(screen.getByText("activity.audit.noRequestLines")).toBeTruthy();
|
||||
expect(screen.queryByRole("button", { name: "activity.audit.requestLines" })).toBeNull();
|
||||
});
|
||||
|
||||
it("the admin tab lists the actions, and an action opens its request's lines", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity&at=admin");
|
||||
renderPage();
|
||||
await waitFor(() => expect(screen.getByText("auditActions.scheduler_interval")).toBeTruthy());
|
||||
expect(runsMock).not.toHaveBeenCalled();
|
||||
|
||||
fireEvent.click(rowButton(/auditActions\.scheduler_interval/));
|
||||
fireEvent.click(screen.getByRole("button", { name: "activity.audit.requestLines" }));
|
||||
|
||||
await waitFor(() =>
|
||||
expect(logsMock).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ request_id: "0efdd960c9ad1977" }),
|
||||
null,
|
||||
),
|
||||
);
|
||||
expect(url().get("lreq")).toBe("0efdd960c9ad1977");
|
||||
});
|
||||
|
||||
it("the account filter reaches the quota tab, over the page's range", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity&at=quota&au=none&aw=30d");
|
||||
renderPage();
|
||||
await waitFor(() => expect(quotaMock).toHaveBeenCalled());
|
||||
const [since, user] = quotaMock.mock.calls[0]!;
|
||||
expect(user).toBe("none");
|
||||
const days = (Date.now() - new Date(since).getTime()) / 86_400_000;
|
||||
expect(Math.round(days)).toBe(30);
|
||||
});
|
||||
|
||||
it("each day's spend is readable without a pointer", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity&at=quota");
|
||||
quotaMock.mockResolvedValue({
|
||||
rows: [{ user_id: null, email: "System", total: 9, by_action: {} }],
|
||||
daily: [
|
||||
{ day: "2026-10-16", total: 4 },
|
||||
{ day: "2026-10-17", total: 5 },
|
||||
],
|
||||
});
|
||||
renderPage();
|
||||
const chart = await screen.findByRole("list", { name: "activity.quota.daily" });
|
||||
expect(
|
||||
within(chart)
|
||||
.getAllByRole("listitem")
|
||||
.map((d) => d.textContent),
|
||||
).toEqual(["stats.dailyTooltip", "stats.dailyTooltip"]);
|
||||
});
|
||||
|
||||
// The runs moved over from the audit log kept one instant for start and end: their length is
|
||||
// unknown, and a "1 sec" would be invented.
|
||||
it("a run whose start and end are one instant has no length", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity");
|
||||
runsMock.mockResolvedValue(
|
||||
page([
|
||||
{ ...RUN, id: 1, status: "ok", error: null, finished_at: RUN.started_at, summary: "moved" },
|
||||
{ ...RUN, id: 2, status: "ok", error: null, summary: "timed" },
|
||||
]),
|
||||
);
|
||||
renderPage();
|
||||
await waitFor(() => expect(screen.getByText("moved")).toBeTruthy());
|
||||
const cells = (text: string) =>
|
||||
[...rowButton(new RegExp(text)).children].map((c) => c.textContent);
|
||||
expect(cells("moved")).toContain("—");
|
||||
expect(cells("timed")).not.toContain("—");
|
||||
});
|
||||
|
||||
// A bare "212" in a row reads as nothing in particular: the spend carries its unit, and a run that
|
||||
// spent none says nothing rather than "0".
|
||||
it("a run's quota spend reads with its unit, and none spent leaves the cell blank", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity");
|
||||
runsMock.mockResolvedValue(
|
||||
page([
|
||||
{ ...RUN, id: 1, status: "ok", error: null, summary: "spent" },
|
||||
{ ...RUN, id: 2, status: "ok", error: null, summary: "free", quota_units: 0 },
|
||||
]),
|
||||
);
|
||||
renderPage();
|
||||
await waitFor(() => expect(screen.getByText("spent")).toBeTruthy());
|
||||
const cells = (text: string) =>
|
||||
[...rowButton(new RegExp(text)).children].map((c) => c.textContent);
|
||||
expect(cells("spent")).toContain("activity.runs.units");
|
||||
expect(cells("spent")).not.toContain("212");
|
||||
expect(cells("free")).not.toContain("0");
|
||||
expect(cells("free")).not.toContain("activity.runs.units");
|
||||
});
|
||||
|
||||
// Most runs store no line at all: a link there would only lead to an empty list.
|
||||
it("a run that logged nothing says so instead of linking to an empty list", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity");
|
||||
runsMock.mockResolvedValue(page([{ ...RUN, log_lines: 0 }]));
|
||||
renderPage();
|
||||
await waitFor(() => expect(screen.getByText("quotaExceeded")).toBeTruthy());
|
||||
fireEvent.click(rowButton(/quotaExceeded/));
|
||||
expect(screen.getByText("activity.runs.noLogLines")).toBeTruthy();
|
||||
expect(screen.queryByRole("button", { name: "activity.runs.logLines" })).toBeNull();
|
||||
});
|
||||
|
||||
it("a failed read is an error with a retry, not an empty list", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity");
|
||||
runsMock.mockRejectedValue(new Error("502"));
|
||||
renderPage();
|
||||
|
||||
expect(await screen.findByRole("alert")).toBeTruthy();
|
||||
expect(screen.queryByText("activity.runs.empty")).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,214 @@
|
||||
import clsx from "clsx";
|
||||
import { useEffect, useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { Search } from "lucide-react";
|
||||
|
||||
import { api } from "../lib/api";
|
||||
import {
|
||||
ACTIVITY_TABS,
|
||||
ACTIVITY_WINDOWS,
|
||||
activityParamsToState,
|
||||
activityParamUpdates,
|
||||
type ActivityTab,
|
||||
type ActivityUrlState,
|
||||
type ActivityWindow,
|
||||
} from "../lib/activityUrlState";
|
||||
import { nextNavId, notifyNav, rememberNavPosition } from "../lib/history";
|
||||
import {
|
||||
DEFAULT_LOGS_STATE,
|
||||
logsParamsToState,
|
||||
logsParamUpdates,
|
||||
type LogsUrlState,
|
||||
} from "../lib/logsUrlState";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useDebounced } from "../lib/useDebounced";
|
||||
import { urlWithParams } from "../lib/urlState";
|
||||
import { useUrlMirror } from "../lib/useUrlMirror";
|
||||
import ActivityAudit from "./ActivityAudit";
|
||||
import ActivityQuota from "./ActivityQuota";
|
||||
import ActivityRuns from "./ActivityRuns";
|
||||
import AdminLogs, { type LogPivot } from "./AdminLogs";
|
||||
import FilterChipGroup from "./ui/FilterChipGroup";
|
||||
import FilterSelect from "./ui/FilterSelect";
|
||||
import { InnerInput } from "./ui/Input";
|
||||
import { fieldClass } from "./ui/fieldStyles";
|
||||
import { PageToolbar } from "./ui/PageShell";
|
||||
import { StateMessage } from "./ui/QueryState";
|
||||
import Tabs from "./ui/Tabs";
|
||||
import { useTabPanel } from "./ui/tabPanel";
|
||||
|
||||
// Activity ("Napló"): what happened, in one place — the background runs, the admins' actions, the
|
||||
// server's log and the quota spend, each its own table on the server, read here as four tabs under
|
||||
// one filter bar (time range, account, text). The view lives in the URL, so it is a link; a pivot to
|
||||
// one run's or one request's log lines is a new history entry, and Back returns to where it was made.
|
||||
|
||||
// A pivot shows everything the run or request logged: every other filter would only hide part of it,
|
||||
// and the longest range is the log's own retention.
|
||||
const PIVOT_SHARED: Pick<ActivityUrlState, "window" | "user" | "q"> = {
|
||||
window: "90d",
|
||||
user: null,
|
||||
q: "",
|
||||
};
|
||||
|
||||
export default function Activity() {
|
||||
const { t } = useTranslation();
|
||||
const [state, setState] = useState<ActivityUrlState>(() =>
|
||||
activityParamsToState(new URLSearchParams(window.location.search)),
|
||||
);
|
||||
// Bumped when the view changes by a jump (a pivot, Back/Forward) rather than by typing: the search
|
||||
// text then applies at once instead of after the typing debounce.
|
||||
const [jumps, setJumps] = useState(0);
|
||||
// Bumped by a pivot only: the server log, already open, takes the pivot's filters from the address
|
||||
// bar when it mounts again. Back/Forward it follows in place — a popstate is also a dialog closing,
|
||||
// onto the same address, and a fresh log would drop its Live and its focus.
|
||||
const [pivots, setPivots] = useState(0);
|
||||
// The log's half of the entry Back/Forward landed on, read here with this page's half: one listener,
|
||||
// so both change in one render. Two listeners rendered in between (a browser runs the microtasks
|
||||
// after each), and that render read the log with one entry's filters and the other's range.
|
||||
const [landed, setLanded] = useState<{ logs: LogsUrlState } | null>(null);
|
||||
const q = useDebounced(state.q, 300, jumps);
|
||||
const panel = useTabPanel();
|
||||
// `page` too: a link to a page this one replaced (audit, logs) is rewritten to this page's id.
|
||||
useUrlMirror({ page: "activity", ...activityParamUpdates(state) });
|
||||
useEffect(() => {
|
||||
const onPop = () => {
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
setJumps((n) => n + 1);
|
||||
setState(activityParamsToState(params));
|
||||
setLanded({ logs: logsParamsToState(params) });
|
||||
};
|
||||
window.addEventListener("popstate", onPop);
|
||||
return () => window.removeEventListener("popstate", onPop);
|
||||
}, []);
|
||||
const update = (patch: Partial<ActivityUrlState>) => setState((s) => ({ ...s, ...patch }));
|
||||
|
||||
const pivot = (target: LogPivot) => {
|
||||
const next: ActivityUrlState = { ...state, ...PIVOT_SHARED, tab: "logs" };
|
||||
window.history.pushState(
|
||||
{ ...rememberNavPosition(), sfScroll: nextNavId() },
|
||||
"",
|
||||
urlWithParams({
|
||||
page: "activity",
|
||||
...activityParamUpdates(next),
|
||||
...logsParamUpdates({ ...DEFAULT_LOGS_STATE, ...target }),
|
||||
}),
|
||||
);
|
||||
notifyNav();
|
||||
setJumps((n) => n + 1);
|
||||
setPivots((n) => n + 1);
|
||||
setState(next);
|
||||
};
|
||||
|
||||
const shared = { window: state.window, user: state.user, q };
|
||||
// The account menu's list (SharedBar reads the same query): without it the menu offers only "Any".
|
||||
const users = useQuery({ queryKey: qk.adminUsers(), queryFn: api.adminUsers });
|
||||
return (
|
||||
<div {...panel.panelProps(state.tab)} className="px-4 pb-4 pt-3 max-w-6xl w-full mx-auto">
|
||||
<PageToolbar>
|
||||
<div className="px-4 pt-2 max-w-6xl w-full mx-auto">
|
||||
<Tabs
|
||||
idBase={panel.idBase}
|
||||
active={state.tab}
|
||||
onChange={(tab) => update({ tab: tab as ActivityTab })}
|
||||
tabs={ACTIVITY_TABS.map((id) => ({ id, label: t(`activity.tabs.${id}`) }))}
|
||||
className="flex flex-wrap gap-1 pb-2"
|
||||
/>
|
||||
<SharedBar state={state} onChange={update} />
|
||||
</div>
|
||||
</PageToolbar>
|
||||
{/* Only when the menu really lacks them: a refresh that failed keeps the list it had. */}
|
||||
{users.isError && !users.data && (
|
||||
<StateMessage tone="error" compact onRetry={() => void users.refetch()}>
|
||||
{t("activity.accountsFailed")}
|
||||
</StateMessage>
|
||||
)}
|
||||
{/* The quota tab says what its numbers are itself, beside them. */}
|
||||
{state.tab !== "quota" && (
|
||||
<p className="mb-3 text-xs leading-relaxed text-muted">
|
||||
{t(`activity.intro.${state.tab}`)}
|
||||
</p>
|
||||
)}
|
||||
{state.tab === "runs" ? (
|
||||
<ActivityRuns
|
||||
shared={shared}
|
||||
job={state.job}
|
||||
status={state.status}
|
||||
onFilter={update}
|
||||
onLogs={(runId) => pivot({ requestId: null, jobId: null, runId })}
|
||||
/>
|
||||
) : state.tab === "admin" ? (
|
||||
<ActivityAudit
|
||||
shared={shared}
|
||||
action={state.action}
|
||||
onAction={(action) => update({ action })}
|
||||
onRequest={(requestId) => pivot({ requestId, jobId: null, runId: null })}
|
||||
/>
|
||||
) : state.tab === "logs" ? (
|
||||
// Keyed by the pivots: a pivot hands the log its filters through the address bar, which it
|
||||
// reads when it mounts.
|
||||
<AdminLogs key={pivots} shared={shared} landed={landed} onPivot={pivot} />
|
||||
) : (
|
||||
<ActivityQuota shared={shared} />
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/** The filters every tab shares: the text (not on the quota tab — it has nothing to match), the time
|
||||
* range, and the account. */
|
||||
function SharedBar({
|
||||
state,
|
||||
onChange,
|
||||
}: {
|
||||
state: ActivityUrlState;
|
||||
onChange: (patch: Partial<ActivityUrlState>) => void;
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
const users = useQuery({ queryKey: qk.adminUsers(), queryFn: api.adminUsers });
|
||||
const known = users.data ?? [];
|
||||
// The account in use stays offered even when the list does not have it (an erased account).
|
||||
const kept =
|
||||
state.user && state.user !== "none" && !known.some((u) => String(u.id) === state.user)
|
||||
? [{ value: state.user, label: `#${state.user}` }]
|
||||
: [];
|
||||
return (
|
||||
<div className="flex flex-wrap items-center gap-2 pb-2">
|
||||
{state.tab !== "quota" && (
|
||||
<label
|
||||
className={clsx(
|
||||
fieldClass({ size: "sm", pill: true }),
|
||||
"flex min-w-[12rem] flex-1 items-center gap-1.5",
|
||||
)}
|
||||
>
|
||||
<Search className="h-3.5 w-3.5 shrink-0 text-muted" aria-hidden />
|
||||
<InnerInput
|
||||
type="search"
|
||||
value={state.q}
|
||||
onChange={(e) => onChange({ q: e.target.value.slice(0, 200) })}
|
||||
placeholder={t("activity.search")}
|
||||
aria-label={t("activity.search")}
|
||||
/>
|
||||
</label>
|
||||
)}
|
||||
<FilterChipGroup
|
||||
ariaLabel={t("activity.windowLabel")}
|
||||
value={state.window}
|
||||
onChange={(window) => onChange({ window: window as ActivityWindow })}
|
||||
options={ACTIVITY_WINDOWS.map((w) => ({ value: w, label: t(`activity.windows.${w}`) }))}
|
||||
/>
|
||||
<FilterSelect
|
||||
label={t("activity.account")}
|
||||
options={[
|
||||
{ value: "", label: t("activity.any") },
|
||||
{ value: "none", label: t("activity.noAccount") },
|
||||
...known.map((u) => ({ value: String(u.id), label: u.email })),
|
||||
...kept,
|
||||
]}
|
||||
selected={[state.user ?? ""]}
|
||||
onChange={([user]) => onChange({ user: user || null })}
|
||||
testId="activity-user"
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
import { useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { keepPreviousData, useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Trash2 } from "lucide-react";
|
||||
|
||||
import { api, type AuditQuery, type AuditRow } from "../lib/api";
|
||||
import { notifyActionError } from "../lib/actionErrors";
|
||||
import { sinceOf } from "../lib/activityUrlState";
|
||||
import { auditActionLabel, auditText } from "../lib/auditText";
|
||||
import { formatDateTimeSec } from "../lib/format";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useHistoryList } from "../lib/useHistoryList";
|
||||
import type { Shared } from "./ActivityRuns";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import { StateMessage } from "./ui/QueryState";
|
||||
import HistoryList from "./HistoryList";
|
||||
import HistoryRow from "./HistoryRow";
|
||||
import { RelativeTime } from "./ui/RelativeTime";
|
||||
import { Button } from "./ui/Button";
|
||||
import FilterSelect from "./ui/FilterSelect";
|
||||
|
||||
// Activity → Admin actions: the append-only who/what/when trail, newest first, filtered on the
|
||||
// server. A row opens to its before/after and to the lines its request logged. Clear stays here,
|
||||
// leaving one row that says who cleared it.
|
||||
|
||||
const GRID =
|
||||
"grid-cols-[minmax(0,1fr)_auto] sm:grid-cols-[minmax(0,12rem)_minmax(0,12rem)_8rem_minmax(10rem,1fr)]";
|
||||
|
||||
export default function ActivityAudit({
|
||||
shared,
|
||||
action,
|
||||
onAction,
|
||||
onRequest,
|
||||
}: {
|
||||
shared: Shared;
|
||||
action: string | null;
|
||||
onAction: (action: string | null) => void;
|
||||
/** Open the server log at this request's lines. */
|
||||
onRequest: (requestId: string) => void;
|
||||
}) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const qc = useQueryClient();
|
||||
const confirm = useConfirm();
|
||||
const [open, setOpen] = useState<number | null>(null);
|
||||
const query: AuditQuery = {
|
||||
user: shared.user ?? undefined,
|
||||
q: shared.q.trim() || undefined,
|
||||
action: action ?? undefined,
|
||||
};
|
||||
const key = { ...query, window: shared.window };
|
||||
const state = useHistoryList(qk.adminAudit(key), (before) =>
|
||||
api.adminAudit({ ...query, since: sinceOf(shared.window) }, before),
|
||||
);
|
||||
const facets = useQuery({
|
||||
queryKey: qk.adminAuditFacets(key),
|
||||
queryFn: () => api.adminAuditFacets({ ...query, since: sinceOf(shared.window) }),
|
||||
placeholderData: keepPreviousData,
|
||||
});
|
||||
const actions = [
|
||||
...(facets.data?.actions ?? []),
|
||||
...(action && !facets.data?.actions.some((f) => f.value === action)
|
||||
? [{ value: action, count: 0 }]
|
||||
: []),
|
||||
];
|
||||
|
||||
const clear = useMutation({
|
||||
mutationFn: () => api.clearAudit(),
|
||||
onSuccess: (r) => {
|
||||
void qc.invalidateQueries({ queryKey: qk.adminAudit() });
|
||||
void qc.invalidateQueries({ queryKey: qk.adminAuditFacets() });
|
||||
notify({ level: "success", message: t("audit.cleared", { count: r.cleared }) });
|
||||
},
|
||||
onError: (e) => notifyActionError(e, t("audit.clearFailed")),
|
||||
});
|
||||
const onClear = async () => {
|
||||
if (
|
||||
await confirm({
|
||||
title: t("audit.clearTitle"),
|
||||
message: t("audit.clearConfirm"),
|
||||
confirmLabel: t("audit.clear"),
|
||||
danger: true,
|
||||
})
|
||||
)
|
||||
clear.mutate();
|
||||
};
|
||||
const actor = (row: AuditRow) =>
|
||||
row.actor_id === null ? null : (state.accounts[String(row.actor_id)] ?? `#${row.actor_id}`);
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-2">
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
<FilterSelect
|
||||
label={t("activity.audit.action")}
|
||||
options={[
|
||||
{ value: "", label: t("activity.any") },
|
||||
...actions.map((f) => ({
|
||||
value: f.value,
|
||||
label: auditActionLabel(t, f.value),
|
||||
count: f.count,
|
||||
})),
|
||||
]}
|
||||
selected={[action ?? ""]}
|
||||
onChange={([value]) => onAction(value || null)}
|
||||
testId="audit-action"
|
||||
/>
|
||||
<Button
|
||||
variant="outline-danger"
|
||||
size="xs"
|
||||
className="ml-auto"
|
||||
onClick={() => void onClear()}
|
||||
disabled={clear.isPending}
|
||||
icon={<Trash2 className="h-3.5 w-3.5" />}
|
||||
>
|
||||
{t("audit.clear")}
|
||||
</Button>
|
||||
</div>
|
||||
{/* The action menu above is built from these counts: without them it offers only "Any". */}
|
||||
{facets.isError && (
|
||||
<StateMessage tone="error" compact onRetry={() => void facets.refetch()}>
|
||||
{t("activity.filtersFailed")}
|
||||
</StateMessage>
|
||||
)}
|
||||
<HistoryList
|
||||
state={state}
|
||||
empty={t("audit.empty")}
|
||||
count={t("activity.audit.count", { count: state.rows.length })}
|
||||
>
|
||||
{state.rows.map((row, index) => {
|
||||
const { head, sub } = auditText(t, row, lang);
|
||||
const who = actor(row);
|
||||
return (
|
||||
<HistoryRow
|
||||
key={row.id}
|
||||
navIndex={index}
|
||||
navText={`${auditActionLabel(t, row.action)} ${head ?? ""}`}
|
||||
grid={GRID}
|
||||
open={open === row.id}
|
||||
onToggle={() => setOpen(open === row.id ? null : row.id)}
|
||||
title={formatDateTimeSec(row.created_at, lang)}
|
||||
cells={
|
||||
<>
|
||||
<span className="truncate font-medium">{auditActionLabel(t, row.action)}</span>
|
||||
<RelativeTime
|
||||
at={row.created_at}
|
||||
seconds
|
||||
className="tabular-nums text-muted sm:hidden"
|
||||
/>
|
||||
<span
|
||||
className={`hidden truncate sm:block ${who ? "text-muted" : "italic text-muted"}`}
|
||||
>
|
||||
{who ?? t("audit.system")}
|
||||
</span>
|
||||
<RelativeTime
|
||||
at={row.created_at}
|
||||
seconds
|
||||
className="hidden tabular-nums text-muted sm:block"
|
||||
/>
|
||||
<span className="col-span-2 truncate text-muted sm:col-span-1">
|
||||
{[head, sub].filter(Boolean).join(" · ")}
|
||||
</span>
|
||||
</>
|
||||
}
|
||||
>
|
||||
<AuditDetail row={row} who={who} onRequest={onRequest} />
|
||||
</HistoryRow>
|
||||
);
|
||||
})}
|
||||
</HistoryList>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function AuditDetail({
|
||||
row,
|
||||
who,
|
||||
onRequest,
|
||||
}: {
|
||||
row: AuditRow;
|
||||
who: string | null;
|
||||
onRequest: (requestId: string) => void;
|
||||
}) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const { head, sub } = auditText(t, row, lang);
|
||||
return (
|
||||
<div className="flex flex-col gap-2">
|
||||
<dl className="grid grid-cols-[max-content_minmax(0,1fr)] gap-x-3 gap-y-0.5">
|
||||
<dt>{t("audit.cols.actor")}</dt>
|
||||
<dd>{who ?? t("audit.system")}</dd>
|
||||
<dt>{t("audit.cols.when")}</dt>
|
||||
<dd className="tabular-nums">{formatDateTimeSec(row.created_at, lang)}</dd>
|
||||
{(head || sub) && (
|
||||
<>
|
||||
<dt>{t("audit.cols.details")}</dt>
|
||||
<dd className="wrap-break-word">{[head, sub].filter(Boolean).join(" · ")}</dd>
|
||||
</>
|
||||
)}
|
||||
{row.target_id && (
|
||||
<>
|
||||
<dt>{t("activity.audit.target")}</dt>
|
||||
<dd className="font-mono">{row.target_id}</dd>
|
||||
</>
|
||||
)}
|
||||
</dl>
|
||||
{(row.before || row.after) && (
|
||||
<pre className="overflow-x-auto whitespace-pre font-mono">
|
||||
{JSON.stringify({ before: row.before, after: row.after }, null, 2)}
|
||||
</pre>
|
||||
)}
|
||||
{/* A request's lines are kept days, its row months: a link to none would open an empty list. */}
|
||||
{row.request_id &&
|
||||
(row.log_lines > 0 ? (
|
||||
<div>
|
||||
<Button
|
||||
variant="link"
|
||||
className="underline decoration-dotted underline-offset-2 hover:decoration-solid"
|
||||
onClick={() => onRequest(row.request_id!)}
|
||||
>
|
||||
{t("activity.audit.requestLines")}
|
||||
</Button>
|
||||
</div>
|
||||
) : (
|
||||
<p>{t("activity.audit.noRequestLines")}</p>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
import { useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { keepPreviousData, useQuery } from "@tanstack/react-query";
|
||||
|
||||
import { api, type AdminQuotaRow } from "../lib/api";
|
||||
import { sinceOf } from "../lib/activityUrlState";
|
||||
import { formatNumber, quotaActionLabel } from "../lib/format";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import type { Shared } from "./ActivityRuns";
|
||||
import { Pressable } from "./ui/Pressable";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import { SURFACE } from "./ui/surfaces";
|
||||
|
||||
// Activity → Quota: the shared YouTube API budget's spend over the page's time range — per day
|
||||
// (Pacific days, as the budget resets), then per account, each opening to its actions. The
|
||||
// account filter narrows both to one account; the text filter has nothing to match here.
|
||||
|
||||
export default function ActivityQuota({ shared }: { shared: Shared }) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const q = useQuery({
|
||||
queryKey: qk.adminQuota({ window: shared.window, user: shared.user }),
|
||||
queryFn: () => api.adminQuota(sinceOf(shared.window), shared.user),
|
||||
placeholderData: keepPreviousData,
|
||||
});
|
||||
const data = q.data;
|
||||
const maxDaily = Math.max(1, ...(data?.daily ?? []).map((d) => d.total));
|
||||
const grandTotal = (data?.rows ?? []).reduce((s, r) => s + r.total, 0);
|
||||
|
||||
return (
|
||||
<div className={q.isPlaceholderData ? "opacity-60" : ""} aria-busy={q.isPlaceholderData}>
|
||||
<p className="text-xs text-muted mb-3 leading-relaxed">
|
||||
{t("stats.introBefore")}
|
||||
<b className="text-fg/80">{t("stats.introSystem")}</b>
|
||||
{t("stats.introAfter")}
|
||||
</p>
|
||||
{q.isError && !data ? (
|
||||
<StateMessage tone="error" onRetry={() => void q.refetch()}>
|
||||
{t("common.loadError")}
|
||||
</StateMessage>
|
||||
) : q.isLoading || !data ? (
|
||||
<LoadingState label={t("stats.loading")} />
|
||||
) : (
|
||||
<>
|
||||
{q.isError && (
|
||||
<StateMessage tone="error" compact onRetry={() => void q.refetch()} className="mb-3">
|
||||
{t("activity.stale")}
|
||||
</StateMessage>
|
||||
)}
|
||||
<div className={`${SURFACE.card} p-3 mb-4`}>
|
||||
<div className="text-xs text-muted mb-2">
|
||||
{t(shared.window === "all" ? "activity.quota.totalAll" : "activity.quota.total", {
|
||||
count: grandTotal,
|
||||
window: t(`activity.windows.${shared.window}`),
|
||||
units: formatNumber(grandTotal, lang),
|
||||
})}
|
||||
</div>
|
||||
{data.daily.length === 0 ? (
|
||||
<div className="text-muted text-sm">{t("stats.noUsageInRange")}</div>
|
||||
) : (
|
||||
// Each day's figure is in its bar's text too (read out, not shown): the tooltip only
|
||||
// reaches a pointer.
|
||||
<div
|
||||
className="flex items-end gap-0.5 h-24"
|
||||
role="list"
|
||||
aria-label={t("activity.quota.daily")}
|
||||
>
|
||||
{data.daily.map((d) => (
|
||||
<div
|
||||
key={d.day}
|
||||
role="listitem"
|
||||
// The day under the pointer is lit by the B3 layer over its whole column, not by
|
||||
// dimming the bar: an opacity dim reads as "disabled".
|
||||
className="flex-1 h-full flex items-end bg-card/50 rounded-t state-layer"
|
||||
title={t("stats.dailyTooltip", {
|
||||
count: d.total,
|
||||
day: d.day,
|
||||
units: formatNumber(d.total, lang),
|
||||
})}
|
||||
>
|
||||
<span className="sr-only">
|
||||
{t("stats.dailyTooltip", {
|
||||
count: d.total,
|
||||
day: d.day,
|
||||
units: formatNumber(d.total, lang),
|
||||
})}
|
||||
</span>
|
||||
<div
|
||||
className="w-full bg-accent rounded-t"
|
||||
style={{ height: `${Math.max(3, (d.total / maxDaily) * 100)}%` }}
|
||||
/>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-1.5">
|
||||
{data.rows.length === 0 ? (
|
||||
<div className="text-muted py-4">{t("stats.noUsageInRange")}</div>
|
||||
) : (
|
||||
data.rows.map((r) => (
|
||||
<UserRow key={r.user_id ?? "system"} row={r} max={data.rows[0]?.total || 1} />
|
||||
))
|
||||
)}
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function UserRow({ row, max }: { row: AdminQuotaRow; max: number }) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const [open, setOpen] = useState(false);
|
||||
const actions = Object.entries(row.by_action).sort((a, b) => b[1] - a[1]);
|
||||
const isSystem = row.user_id === null;
|
||||
return (
|
||||
<div className={`${SURFACE.card} p-3`}>
|
||||
<Pressable
|
||||
selectable
|
||||
onClick={() => setOpen((o) => !o)}
|
||||
aria-expanded={open}
|
||||
className="w-full flex items-center gap-3"
|
||||
>
|
||||
<span className={`text-sm font-medium truncate flex-1 ${isSystem ? "text-muted" : ""}`}>
|
||||
{isSystem ? t("stats.system") : row.email}
|
||||
</span>
|
||||
<span className="text-sm font-semibold tabular-nums shrink-0">
|
||||
{formatNumber(row.total, lang)}
|
||||
</span>
|
||||
</Pressable>
|
||||
{/* Proportion bar */}
|
||||
<div className="mt-2 h-1.5 bg-border/40 rounded-full overflow-hidden">
|
||||
<div
|
||||
className="h-full bg-accent rounded-full"
|
||||
style={{ width: `${Math.max(2, (row.total / max) * 100)}%` }}
|
||||
/>
|
||||
</div>
|
||||
{open && actions.length > 0 && (
|
||||
<div className="mt-2 space-y-1 text-xs text-muted">
|
||||
{actions.map(([action, units]) => (
|
||||
<div key={action} className="flex items-center justify-between gap-2">
|
||||
<span>{quotaActionLabel(t, action)}</span>
|
||||
<span className="tabular-nums">{formatNumber(units, lang)}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,282 @@
|
||||
import { useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { keepPreviousData, useQuery } from "@tanstack/react-query";
|
||||
|
||||
import { api, type LogFacet, type RunQuery, type RunRow } from "../lib/api";
|
||||
import {
|
||||
RUN_STATUSES,
|
||||
sinceOf,
|
||||
type ActivityUrlState,
|
||||
type RunStatusFilter,
|
||||
} from "../lib/activityUrlState";
|
||||
import { formatDateTimeSec, formatNumber, formatRuntime } from "../lib/format";
|
||||
import { jobFailureText, jobResultText, parseSummary } from "../lib/jobResult";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useHistoryList } from "../lib/useHistoryList";
|
||||
import HistoryList from "./HistoryList";
|
||||
import HistoryRow from "./HistoryRow";
|
||||
import { StateMessage } from "./ui/QueryState";
|
||||
import { RelativeTime } from "./ui/RelativeTime";
|
||||
import { StatusDot } from "./RunStatusDot";
|
||||
import { Button } from "./ui/Button";
|
||||
import FilterChipGroup from "./ui/FilterChipGroup";
|
||||
import FilterSelect from "./ui/FilterSelect";
|
||||
|
||||
// Activity → Runs: every background run, newest first — the scheduled ones, an admin's "run now", a
|
||||
// user's work on one channel. A row opens to who started it, when, what it returned or why it
|
||||
// failed, and two ways on: the lines the run logged, and the other runs of its job.
|
||||
|
||||
const GRID =
|
||||
"grid-cols-[0.75rem_minmax(0,1fr)_auto] sm:grid-cols-[0.75rem_minmax(8rem,12rem)_minmax(0,8rem)_5.5rem_4rem_5.5rem_minmax(10rem,1fr)]";
|
||||
|
||||
/** The shared filters, as the page passes them down. */
|
||||
export interface Shared {
|
||||
window: ActivityUrlState["window"];
|
||||
user: string | null;
|
||||
q: string;
|
||||
}
|
||||
|
||||
export default function ActivityRuns({
|
||||
shared,
|
||||
job,
|
||||
status,
|
||||
onFilter,
|
||||
onLogs,
|
||||
}: {
|
||||
shared: Shared;
|
||||
job: string | null;
|
||||
status: RunStatusFilter | null;
|
||||
onFilter: (patch: { job?: string | null; status?: RunStatusFilter | null }) => void;
|
||||
/** Open the server log at this run's lines. */
|
||||
onLogs: (runId: number) => void;
|
||||
}) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const [open, setOpen] = useState<number | null>(null);
|
||||
const query: RunQuery = {
|
||||
user: shared.user ?? undefined,
|
||||
q: shared.q.trim() || undefined,
|
||||
job: job ?? undefined,
|
||||
status: status ?? undefined,
|
||||
};
|
||||
const key = { ...query, window: shared.window };
|
||||
const state = useHistoryList(qk.adminRuns(key), (before) =>
|
||||
api.adminRuns({ ...query, since: sinceOf(shared.window) }, before),
|
||||
);
|
||||
const facets = useQuery({
|
||||
queryKey: qk.adminRunFacets(key),
|
||||
queryFn: () => api.adminRunFacets({ ...query, since: sinceOf(shared.window) }),
|
||||
placeholderData: keepPreviousData,
|
||||
});
|
||||
const count = (list: LogFacet[] | undefined, value: string) =>
|
||||
list?.find((f) => f.value === value)?.count ?? (facets.data ? 0 : undefined);
|
||||
const jobName = (id: string) => t(`scheduler.jobs.${id}`, id);
|
||||
// The job in use stays offered at zero: under the other filters it may count nothing, and a menu
|
||||
// without it would show no way back.
|
||||
const jobs = [
|
||||
...(facets.data?.jobs ?? []),
|
||||
...(job && !facets.data?.jobs.some((f) => f.value === job) ? [{ value: job, count: 0 }] : []),
|
||||
];
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-2">
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
<FilterSelect
|
||||
label={t("activity.runs.job")}
|
||||
options={[
|
||||
{ value: "", label: t("activity.any") },
|
||||
...jobs.map((f) => ({ value: f.value, label: jobName(f.value), count: f.count })),
|
||||
]}
|
||||
selected={[job ?? ""]}
|
||||
onChange={([value]) => onFilter({ job: value || null })}
|
||||
testId="runs-job"
|
||||
/>
|
||||
<FilterChipGroup
|
||||
ariaLabel={t("activity.runs.statusLabel")}
|
||||
value={status ?? "all"}
|
||||
onChange={(value) =>
|
||||
onFilter({ status: value === "all" ? null : (value as RunStatusFilter) })
|
||||
}
|
||||
options={[
|
||||
{ value: "all", label: t("activity.any"), count: facets.data?.total },
|
||||
...RUN_STATUSES.map((s) => ({
|
||||
value: s,
|
||||
label: t(`activity.runs.status.${s}`),
|
||||
count: count(facets.data?.statuses, s),
|
||||
})),
|
||||
]}
|
||||
/>
|
||||
</div>
|
||||
{/* The menus above are built from these counts: without them they offer only "Any". */}
|
||||
{facets.isError && (
|
||||
<StateMessage tone="error" compact onRetry={() => void facets.refetch()}>
|
||||
{t("activity.filtersFailed")}
|
||||
</StateMessage>
|
||||
)}
|
||||
<HistoryList
|
||||
state={state}
|
||||
empty={t("activity.runs.empty")}
|
||||
count={t("activity.runs.count", { count: state.rows.length })}
|
||||
>
|
||||
{state.rows.map((run, index) => (
|
||||
<HistoryRow
|
||||
key={run.id}
|
||||
navIndex={index}
|
||||
navText={`${jobName(run.job)} ${run.target_title ?? ""}`}
|
||||
grid={GRID}
|
||||
open={open === run.id}
|
||||
onToggle={() => setOpen(open === run.id ? null : run.id)}
|
||||
title={formatDateTimeSec(run.started_at, lang)}
|
||||
cells={
|
||||
<RunCells
|
||||
run={run}
|
||||
actor={run.actor_id === null ? null : state.accounts[String(run.actor_id)]}
|
||||
/>
|
||||
}
|
||||
>
|
||||
<RunDetail
|
||||
run={run}
|
||||
actor={run.actor_id === null ? null : state.accounts[String(run.actor_id)]}
|
||||
onLogs={() => onLogs(run.id)}
|
||||
onJob={() => onFilter({ job: run.job })}
|
||||
/>
|
||||
</HistoryRow>
|
||||
))}
|
||||
</HistoryList>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/** How long the run took; null while it runs, 0 when its start and end are one instant (the runs
|
||||
* moved over from the audit log kept only when they were written — the length is unknown). */
|
||||
function seconds(run: RunRow): number | null {
|
||||
if (!run.finished_at) return null;
|
||||
return (new Date(run.finished_at).getTime() - new Date(run.started_at).getTime()) / 1000;
|
||||
}
|
||||
|
||||
/** What the run came to, in the reader's language: its failure, that the process stopped under it,
|
||||
* or its result. */
|
||||
function useOutcome(run: RunRow): { text: string; tone: string } {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
if (run.status === "error")
|
||||
return {
|
||||
text: jobFailureText(t, run.failure, lang) ?? run.error ?? t("scheduler.dot.error"),
|
||||
tone: "text-danger",
|
||||
};
|
||||
if (run.status === "interrupted") return { text: t("scheduler.interrupted"), tone: "text-warn" };
|
||||
if (run.status === "running") return { text: t("scheduler.runningNow"), tone: "text-accent" };
|
||||
// A run moved over from the audit log before its result rode along as data holds only the
|
||||
// server's English one-liner: read back into a result, as the old audit rows were.
|
||||
const result =
|
||||
run.result ??
|
||||
parseSummary(run.summary) ??
|
||||
(run.status === "skipped" && run.summary ? { skipped: run.summary } : null);
|
||||
if (result) return { text: jobResultText(t, result, lang), tone: "text-muted" };
|
||||
return { text: run.summary ?? "", tone: "text-muted" };
|
||||
}
|
||||
|
||||
function RunCells({ run, actor }: { run: RunRow; actor: string | null | undefined }) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const outcome = useOutcome(run);
|
||||
const secs = seconds(run);
|
||||
const name = t(`scheduler.jobs.${run.job}`, run.job);
|
||||
return (
|
||||
<>
|
||||
<StatusDot k={run.status} label={t(`scheduler.dot.${run.status}`)} />
|
||||
<span className="truncate font-medium">
|
||||
{name}
|
||||
{run.target_title && <span className="font-normal text-muted"> · {run.target_title}</span>}
|
||||
</span>
|
||||
{/* A phone has room for the name and the time only; the row opens to the rest. */}
|
||||
<RelativeTime at={run.started_at} seconds className="tabular-nums text-muted sm:hidden" />
|
||||
<span className="hidden truncate text-muted sm:block">
|
||||
{t(`scheduler.trigger.${run.trigger}`)}
|
||||
{actor && ` · ${actor}`}
|
||||
</span>
|
||||
<RelativeTime
|
||||
at={run.started_at}
|
||||
seconds
|
||||
className="hidden tabular-nums text-muted sm:block"
|
||||
/>
|
||||
<span className="hidden tabular-nums text-muted sm:block">
|
||||
{secs ? formatRuntime(Math.max(1, secs), lang) : "—"}
|
||||
</span>
|
||||
{/* The spend carries its unit (a bare figure in a row has no meaning); none spent is blank. */}
|
||||
<span className="hidden truncate tabular-nums text-muted sm:block">
|
||||
{run.quota_units > 0 &&
|
||||
t("activity.runs.units", {
|
||||
count: run.quota_units,
|
||||
units: formatNumber(run.quota_units, lang),
|
||||
})}
|
||||
</span>
|
||||
<span className={`col-span-3 truncate sm:col-span-1 ${outcome.tone}`}>{outcome.text}</span>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
function RunDetail({
|
||||
run,
|
||||
actor,
|
||||
onLogs,
|
||||
onJob,
|
||||
}: {
|
||||
run: RunRow;
|
||||
actor: string | null | undefined;
|
||||
onLogs: () => void;
|
||||
onJob: () => void;
|
||||
}) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const link = "underline decoration-dotted underline-offset-2 hover:decoration-solid";
|
||||
return (
|
||||
<div className="flex flex-col gap-2">
|
||||
<dl className="grid grid-cols-[max-content_minmax(0,1fr)] gap-x-3 gap-y-0.5">
|
||||
<dt>{t("activity.runs.fields.started")}</dt>
|
||||
<dd>
|
||||
{run.actor_id === null
|
||||
? t(`scheduler.trigger.${run.trigger}`)
|
||||
: `${actor ?? `#${run.actor_id}`} · ${t(`scheduler.trigger.${run.trigger}`)}`}
|
||||
</dd>
|
||||
<dt>{t("activity.runs.fields.when")}</dt>
|
||||
<dd className="tabular-nums">
|
||||
{formatDateTimeSec(run.started_at, lang)}
|
||||
{run.finished_at && ` → ${formatDateTimeSec(run.finished_at, lang)}`}
|
||||
</dd>
|
||||
{run.target && (
|
||||
<>
|
||||
<dt>{t("activity.runs.fields.target")}</dt>
|
||||
<dd>
|
||||
{run.target_title ? `${run.target_title} · ` : ""}
|
||||
<span className="font-mono">{run.target}</span>
|
||||
</dd>
|
||||
</>
|
||||
)}
|
||||
<dt>{t("activity.runs.fields.quota")}</dt>
|
||||
<dd className="tabular-nums">{formatNumber(run.quota_units, lang)}</dd>
|
||||
</dl>
|
||||
{run.error && (
|
||||
<pre className="whitespace-pre-wrap break-words text-danger/90">{run.error}</pre>
|
||||
)}
|
||||
{run.result && (
|
||||
<pre className="overflow-x-auto whitespace-pre font-mono">
|
||||
{JSON.stringify(run.result, null, 2)}
|
||||
</pre>
|
||||
)}
|
||||
<div className="flex flex-wrap gap-x-4 gap-y-1">
|
||||
{/* Most runs store no line at all; a link there would only open an empty list. */}
|
||||
{run.log_lines > 0 ? (
|
||||
<Button variant="link" className={link} onClick={onLogs}>
|
||||
{t("activity.runs.logLines", { count: run.log_lines })}
|
||||
</Button>
|
||||
) : (
|
||||
<span>{t("activity.runs.noLogLines")}</span>
|
||||
)}
|
||||
<Button variant="link" className={link} onClick={onJob}>
|
||||
{t("activity.runs.jobRuns")}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,9 +1,11 @@
|
||||
// @vitest-environment jsdom
|
||||
//
|
||||
// The admin log console. A request id narrows the list to that request (and says so, in the
|
||||
// filter row and the URL); a filter value the facets no longer count stays pickable; "Live" watches
|
||||
// the newest page only, and stops on a failed read instead of re-raising it; a failed read is an
|
||||
// error, not an empty log; Copy takes every loaded line, oldest first.
|
||||
// The activity page's server-log tab, rendered through the page (which owns the shared filters and
|
||||
// the pivots). A request or run id narrows the list to that request or run (and says so, in the
|
||||
// filter row and the URL), and Back returns to the view the pivot was made from; the account in use
|
||||
// stays pickable; "Live" watches the newest page only, and stops on a failed read instead of
|
||||
// re-raising it; a failed read is an error, not an empty log; Copy takes every loaded line, oldest
|
||||
// first.
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { act, cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
@@ -32,6 +34,7 @@ vi.mock("../lib/api", async (importOriginal) => {
|
||||
...actual.api,
|
||||
adminLogs: (query: LogQuery, before: string | null) => logsMock(query, before),
|
||||
adminLogFacets: (query: LogQuery) => facetsMock(query),
|
||||
adminUsers: () => Promise.resolve([{ id: 1, email: "admin@example.com" }]),
|
||||
},
|
||||
};
|
||||
});
|
||||
@@ -44,7 +47,11 @@ vi.mock("../lib/clipboard", () => ({
|
||||
},
|
||||
}));
|
||||
|
||||
import AdminLogs from "./AdminLogs";
|
||||
import Activity from "./Activity";
|
||||
import { PageToolbarSlot } from "./ui/pageToolbarSlot";
|
||||
|
||||
// The filter bar lives in the toolbar, which portals into the shell's band: give it one.
|
||||
const slot = document.body.appendChild(document.createElement("div"));
|
||||
|
||||
function row(id: number, extra: Partial<ServerLogRow> = {}): ServerLogRow {
|
||||
return {
|
||||
@@ -60,6 +67,7 @@ function row(id: number, extra: Partial<ServerLogRow> = {}): ServerLogRow {
|
||||
request_id: `req-${String(id).padStart(8, "0")}`,
|
||||
user_id: 1,
|
||||
job_id: null,
|
||||
run_id: null,
|
||||
debug_session_id: null,
|
||||
version: "0.90.0",
|
||||
...extra,
|
||||
@@ -79,7 +87,9 @@ function renderConsole() {
|
||||
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
||||
return render(
|
||||
<QueryClientProvider client={qc}>
|
||||
<AdminLogs />
|
||||
<PageToolbarSlot.Provider value={slot}>
|
||||
<Activity />
|
||||
</PageToolbarSlot.Provider>
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
}
|
||||
@@ -93,7 +103,7 @@ const lines = () =>
|
||||
.filter((b) => b.hasAttribute("aria-expanded"));
|
||||
|
||||
beforeEach(() => {
|
||||
window.history.replaceState(null, "", "/?page=logs");
|
||||
window.history.replaceState(null, "", "/?page=activity&at=logs");
|
||||
facetsMock.mockResolvedValue(FACETS);
|
||||
});
|
||||
afterEach(() => {
|
||||
@@ -104,8 +114,21 @@ afterEach(() => {
|
||||
});
|
||||
|
||||
describe("AdminLogs", () => {
|
||||
it("a dialog closing onto the same address leaves the open line open", async () => {
|
||||
logsMock.mockResolvedValue({ items: [row(1)], accounts: {}, next_cursor: null });
|
||||
renderConsole();
|
||||
await waitFor(() => expect(lines()).toHaveLength(1));
|
||||
fireEvent.click(lines()[0]!);
|
||||
expect(lines()[0]!.getAttribute("aria-expanded")).toBe("true");
|
||||
// A closing dialog steps back over its own history entry: a popstate onto the same address.
|
||||
act(() => {
|
||||
window.dispatchEvent(new PopStateEvent("popstate"));
|
||||
});
|
||||
expect(lines()[0]!.getAttribute("aria-expanded")).toBe("true");
|
||||
});
|
||||
|
||||
it("a request id narrows the list to that request, with every other filter off", async () => {
|
||||
window.history.replaceState(null, "", "/?page=logs&lgrp=error&lq=boom");
|
||||
window.history.replaceState(null, "", "/?page=activity&at=logs&lgrp=error&aq=boom");
|
||||
logsMock.mockResolvedValue({ items: [row(1)], accounts: {}, next_cursor: null });
|
||||
renderConsole();
|
||||
|
||||
@@ -121,11 +144,15 @@ describe("AdminLogs", () => {
|
||||
);
|
||||
expect(screen.getByText("adminLogs.pivotRequest:req-00000001")).toBeTruthy();
|
||||
const url = new URLSearchParams(window.location.search);
|
||||
expect([url.get("lreq"), url.get("lgrp"), url.get("lq")]).toEqual(["req-00000001", null, null]);
|
||||
expect([url.get("lreq"), url.get("lgrp"), url.get("aq")]).toEqual(["req-00000001", null, null]);
|
||||
});
|
||||
|
||||
it("Back from a request's lines returns to the view the pivot was made from", async () => {
|
||||
window.history.replaceState({ sfPage: "logs" }, "", "/?page=logs&lgrp=error&lq=boom");
|
||||
window.history.replaceState(
|
||||
{ sfPage: "activity" },
|
||||
"",
|
||||
"/?page=activity&at=logs&lgrp=error&aq=boom",
|
||||
);
|
||||
const depth = window.history.length;
|
||||
logsMock.mockResolvedValue({ items: [row(1)], accounts: {}, next_cursor: null });
|
||||
renderConsole();
|
||||
@@ -157,23 +184,46 @@ describe("AdminLogs", () => {
|
||||
);
|
||||
expect(screen.queryByText("adminLogs.pivotRequest:req-00000001")).toBeNull();
|
||||
const url = new URLSearchParams(window.location.search);
|
||||
expect([url.get("lreq"), url.get("lgrp"), url.get("lq")]).toEqual([null, "error", "boom"]);
|
||||
expect([url.get("lreq"), url.get("lgrp"), url.get("aq")]).toEqual([null, "error", "boom"]);
|
||||
// Neither jump waited out the typing debounce: no read ever mixed one view's filters with the
|
||||
// other's search text (the request with "boom", or the restored view without it).
|
||||
const mixed = logsMock.mock.calls.filter(([q]) => (q.request_id != null) === (q.q === "boom"));
|
||||
expect(mixed).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps the account in use offered when the facets no longer count it", async () => {
|
||||
window.history.replaceState(null, "", "/?page=logs&lu=7");
|
||||
it("a run id narrows the list to that run's lines", async () => {
|
||||
logsMock.mockResolvedValue({
|
||||
items: [row(1, { run_id: 77 })],
|
||||
accounts: {},
|
||||
next_cursor: null,
|
||||
});
|
||||
renderConsole();
|
||||
await waitFor(() => expect(lines()).toHaveLength(1));
|
||||
fireEvent.click(lines()[0]!);
|
||||
fireEvent.click(screen.getByRole("button", { name: "77" }));
|
||||
|
||||
await waitFor(() =>
|
||||
expect(logsMock).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ run_id: 77, request_id: undefined }),
|
||||
null,
|
||||
),
|
||||
);
|
||||
expect(screen.getByText("adminLogs.pivotRun:77")).toBeTruthy();
|
||||
expect(new URLSearchParams(window.location.search).get("lrun")).toBe("77");
|
||||
});
|
||||
|
||||
it("keeps the account in use offered when the account list does not have it", async () => {
|
||||
window.history.replaceState(null, "", "/?page=activity&at=logs&au=7");
|
||||
logsMock.mockResolvedValue({ items: [], accounts: {}, next_cursor: null });
|
||||
renderConsole();
|
||||
await waitFor(() => expect(facetsMock).toHaveBeenCalled());
|
||||
await waitFor(() =>
|
||||
expect(logsMock).toHaveBeenCalledWith(expect.objectContaining({ user: "7" }), null),
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByTestId("logs-user"));
|
||||
fireEvent.click(screen.getByTestId("activity-user"));
|
||||
|
||||
expect(await screen.findByTestId("logs-user-7")).toBeTruthy();
|
||||
expect(screen.getByTestId("logs-user-1")).toBeTruthy();
|
||||
expect(await screen.findByTestId("activity-user-7")).toBeTruthy();
|
||||
expect(screen.getByTestId("activity-user-1")).toBeTruthy();
|
||||
});
|
||||
|
||||
it("Live watches the newest page only", async () => {
|
||||
@@ -258,6 +308,14 @@ describe("AdminLogs", () => {
|
||||
expect((await screen.findByRole("alert")).textContent).toContain("adminLogs.olderFailed");
|
||||
});
|
||||
|
||||
it("facets that failed say the filter menus may be short, in the other tabs' words", async () => {
|
||||
logsMock.mockResolvedValue({ items: [row(1)], accounts: {}, next_cursor: null });
|
||||
facetsMock.mockRejectedValue(new Error("500"));
|
||||
renderConsole();
|
||||
await waitFor(() => expect(lines()).toHaveLength(1));
|
||||
expect((await screen.findByRole("alert")).textContent).toContain("activity.filtersFailed");
|
||||
});
|
||||
|
||||
it("offers no older page while a filter change loads", async () => {
|
||||
let landErrors!: (page: LogPage) => void;
|
||||
logsMock.mockImplementation((q) =>
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
import clsx from "clsx";
|
||||
import { useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useMemo, useRef, useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import {
|
||||
keepPreviousData,
|
||||
@@ -7,74 +6,66 @@ import {
|
||||
useQuery,
|
||||
useQueryClient,
|
||||
} from "@tanstack/react-query";
|
||||
import { Copy, Radio, Search } from "lucide-react";
|
||||
import { Copy, Radio } from "lucide-react";
|
||||
|
||||
import { api, type LogFacet, type LogQuery, type ServerLogRow } from "../lib/api";
|
||||
import { copyText } from "../lib/clipboard";
|
||||
import { formatDateTimeSec } from "../lib/format";
|
||||
import { nextNavId, notifyNav, rememberNavPosition } from "../lib/history";
|
||||
import { nextPageOnClick } from "../lib/nextPage";
|
||||
import { LEVEL_GROUPS, type LevelGroup } from "../lib/log";
|
||||
import {
|
||||
DEFAULT_LOGS_STATE,
|
||||
logsParamsToState,
|
||||
logsParamUpdates,
|
||||
WINDOW_HOURS,
|
||||
type LogsUrlState,
|
||||
type LogWindow,
|
||||
} from "../lib/logsUrlState";
|
||||
import { sinceOf } from "../lib/activityUrlState";
|
||||
import { logsParamsToState, logsParamUpdates, type LogsUrlState } from "../lib/logsUrlState";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useDebounced } from "../lib/useDebounced";
|
||||
import { urlWithParams } from "../lib/urlState";
|
||||
import { useUrlMirror } from "../lib/useUrlMirror";
|
||||
import { HistoryFrame } from "./HistoryList";
|
||||
import LogLine from "./LogLine";
|
||||
import NavGrid from "./ui/NavGrid";
|
||||
import { Button } from "./ui/Button";
|
||||
import FilterChipGroup from "./ui/FilterChipGroup";
|
||||
import FilterSelect from "./ui/FilterSelect";
|
||||
import { FilterFoldButton } from "./ui/FilterFold";
|
||||
import { useFilterFold } from "../lib/useFilterFold";
|
||||
import { useStickyTop } from "./ui/stickyTop";
|
||||
import { Section } from "./ui/form";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import { DismissibleChip } from "./ui/Chip";
|
||||
import { InnerInput } from "./ui/Input";
|
||||
import { fieldClass } from "./ui/fieldStyles";
|
||||
import type { Shared } from "./ActivityRuns";
|
||||
|
||||
// Admin → Szervernapló: the server's stored log (`log_events`). One list, newest first; a line
|
||||
// opens in place to its full record, and its request id / job id narrow the list to that request or
|
||||
// job. Filters live in the URL, so a filtered view is a link. "Élő" re-reads the newest page every
|
||||
// Activity → Server log: the server's stored log (`log_events`). One list, newest first; a line
|
||||
// opens in place to its full record, and its request / job / run id narrows the list to that
|
||||
// request, job or run (the page does that: a pivot is a new history entry with every other filter
|
||||
// off). The time range, account and text come from the page's shared bar; the rest live in the URL
|
||||
// (./logsUrlState), so a filtered view is a link. "Élő" re-reads the newest page every
|
||||
// few seconds while the tab is visible — the newest page only: live is watching the top, so paging
|
||||
// into older rows turns it off. A failed read stops the polling (the request already reported it
|
||||
// once) and marks the list as possibly stale, instead of re-raising the failure every few seconds.
|
||||
|
||||
const LIVE_MS = 4000;
|
||||
const GROUPS: readonly LevelGroup[] = ["all", "error", "warning", "info", "detail"];
|
||||
const WINDOWS: readonly LogWindow[] = ["1h", "24h", "7d"];
|
||||
// A pivot shows everything about one request or job: the other filters would only hide part of it.
|
||||
const PIVOT_RESET: Omit<LogsUrlState, "requestId" | "jobId"> = {
|
||||
...DEFAULT_LOGS_STATE,
|
||||
window: "7d",
|
||||
};
|
||||
|
||||
function sinceOf(window: LogWindow): string {
|
||||
return new Date(Date.now() - WINDOW_HOURS[window] * 3_600_000).toISOString();
|
||||
}
|
||||
|
||||
function toQuery(state: LogsUrlState, q: string): LogQuery {
|
||||
function toQuery(state: LogsUrlState, shared: Shared): LogQuery {
|
||||
return {
|
||||
levels: state.group === "all" ? undefined : LEVEL_GROUPS[state.group],
|
||||
sources: state.sources,
|
||||
user: state.user ?? undefined,
|
||||
user: shared.user ?? undefined,
|
||||
event: state.event ?? undefined,
|
||||
request_id: state.requestId ?? undefined,
|
||||
job_id: state.jobId ?? undefined,
|
||||
q: q.trim() || undefined,
|
||||
run_id: state.runId ?? undefined,
|
||||
q: shared.q.trim() || undefined,
|
||||
};
|
||||
}
|
||||
|
||||
export default function AdminLogs() {
|
||||
/** The ids a line can narrow the list to. */
|
||||
export type LogPivot = Pick<LogsUrlState, "requestId" | "jobId" | "runId">;
|
||||
|
||||
export default function AdminLogs({
|
||||
shared,
|
||||
landed,
|
||||
onPivot,
|
||||
}: {
|
||||
shared: Shared;
|
||||
/** The filters of the entry Back/Forward landed on (the page reads the address bar, once). */
|
||||
landed: { logs: LogsUrlState } | null;
|
||||
/** Show only one request's, job's or run's lines (the page clears every other filter). */
|
||||
onPivot: (pivot: LogPivot) => void;
|
||||
}) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const qc = useQueryClient();
|
||||
@@ -82,14 +73,9 @@ export default function AdminLogs() {
|
||||
logsParamsToState(new URLSearchParams(window.location.search)),
|
||||
);
|
||||
const [live, setLiveState] = useState(false);
|
||||
// The toolbar sticks at the scroller's top (./ui/stickyTop), and folds its facet row when short.
|
||||
useStickyTop();
|
||||
// The facet row folds away when the screen is short.
|
||||
const fold = useFilterFold();
|
||||
const facetFilters =
|
||||
state.sources.length +
|
||||
(state.user ? 1 : 0) +
|
||||
(state.event ? 1 : 0) +
|
||||
(state.window !== DEFAULT_LOGS_STATE.window ? 1 : 0);
|
||||
const facetFilters = state.sources.length + (state.event ? 1 : 0);
|
||||
// Read by `getNextPageParam` when a page is fetched, not when it renders: while live, a refetch
|
||||
// stops after the newest page, however many pages the cache holds — a "Load older" that lands
|
||||
// after Live was switched on, or a filter set revisited with older pages still cached.
|
||||
@@ -99,41 +85,37 @@ export default function AdminLogs() {
|
||||
setLiveState(on);
|
||||
};
|
||||
const [open, setOpen] = useState<number | null>(null);
|
||||
// Bumped when the view changes by a jump (a pivot, Back/Forward) rather than by typing: the
|
||||
// search box's text then applies at once instead of after the typing debounce.
|
||||
const [jumps, setJumps] = useState(0);
|
||||
const q = useDebounced(state.q, 300, jumps);
|
||||
useUrlMirror(logsParamUpdates(state));
|
||||
// Back/Forward lands on another view of the console (a pivot pushed one, below): read it back.
|
||||
useEffect(() => {
|
||||
const onPop = () => {
|
||||
// Back/Forward lands on another entry's filters; a dialog closing lands on these same ones, and
|
||||
// changes nothing (Live and the open line stay). Adopted during the render that brings it, with
|
||||
// the page's half.
|
||||
const [seen, setSeen] = useState(landed);
|
||||
if (landed !== seen) {
|
||||
setSeen(landed);
|
||||
if (landed && JSON.stringify(landed.logs) !== JSON.stringify(state)) {
|
||||
setOpen(null);
|
||||
setJumps((n) => n + 1);
|
||||
setState(logsParamsToState(new URLSearchParams(window.location.search)));
|
||||
};
|
||||
window.addEventListener("popstate", onPop);
|
||||
return () => window.removeEventListener("popstate", onPop);
|
||||
}, []);
|
||||
|
||||
setState(landed.logs);
|
||||
}
|
||||
}
|
||||
const update = (patch: Partial<LogsUrlState>) => {
|
||||
setOpen(null);
|
||||
setState((s) => ({ ...s, ...patch }));
|
||||
};
|
||||
|
||||
const query = toQuery(state, q);
|
||||
const key = qk.adminLogs({ ...query, window: state.window });
|
||||
const query = toQuery(state, shared);
|
||||
const key = qk.adminLogs({ ...query, window: shared.window });
|
||||
const logs = useInfiniteQuery({
|
||||
queryKey: key,
|
||||
queryFn: ({ pageParam }) =>
|
||||
api.adminLogs({ ...query, since: sinceOf(state.window) }, pageParam),
|
||||
api.adminLogs({ ...query, since: sinceOf(shared.window) }, pageParam),
|
||||
initialPageParam: null as string | null,
|
||||
getNextPageParam: (last) => (liveRef.current ? undefined : (last.next_cursor ?? undefined)),
|
||||
placeholderData: keepPreviousData,
|
||||
refetchInterval: (q) => (live && q.state.status !== "error" ? LIVE_MS : false),
|
||||
});
|
||||
const facets = useQuery({
|
||||
queryKey: qk.adminLogFacets({ ...query, window: state.window }),
|
||||
queryFn: () => api.adminLogFacets({ ...query, since: sinceOf(state.window) }),
|
||||
queryKey: qk.adminLogFacets({ ...query, window: shared.window }),
|
||||
queryFn: () => api.adminLogFacets({ ...query, since: sinceOf(shared.window) }),
|
||||
placeholderData: keepPreviousData,
|
||||
refetchInterval: (q) => (live && q.state.status !== "error" ? LIVE_MS * 4 : false),
|
||||
});
|
||||
@@ -146,15 +128,16 @@ export default function AdminLogs() {
|
||||
? null
|
||||
: (logs.data?.pages[logs.data.pages.length - 1]?.next_cursor ?? null);
|
||||
// A read that failed while lines were on screen: they are kept, and the message says what the
|
||||
// failure means for them. (With no lines, the list's own error state says it.)
|
||||
// failure means for them. (With no lines, the list's own error state says it.) Without the facets
|
||||
// the Source and Event menus offer only "Any", as on the other tabs, so the same words say it.
|
||||
const failure = !logs.data
|
||||
? null
|
||||
: logs.isFetchNextPageError
|
||||
? "olderFailed"
|
||||
? "adminLogs.olderFailed"
|
||||
: logs.isError || (facets.isError && facets.data)
|
||||
? "stale"
|
||||
? "adminLogs.stale"
|
||||
: facets.isError
|
||||
? "countsFailed"
|
||||
? "activity.filtersFailed"
|
||||
: null;
|
||||
const retryFailed = () => {
|
||||
if (logs.isFetchNextPageError)
|
||||
@@ -217,19 +200,9 @@ export default function AdminLogs() {
|
||||
return facets.data ? sum(levels.filter((f) => inGroup.includes(f.value))) : undefined;
|
||||
};
|
||||
|
||||
// A pivot follows a link to another view, so it is its own history entry and Back returns to the
|
||||
// view it was made from. The other filters are adjustments of the view on screen and replace it.
|
||||
const pivot = (patch: Pick<LogsUrlState, "requestId" | "jobId">) => {
|
||||
const pivot = (patch: Partial<LogPivot>) => {
|
||||
setLive(false);
|
||||
const next = { ...state, ...PIVOT_RESET, ...patch };
|
||||
window.history.pushState(
|
||||
{ ...rememberNavPosition(), sfScroll: nextNavId() },
|
||||
"",
|
||||
urlWithParams(logsParamUpdates(next)),
|
||||
);
|
||||
notifyNav();
|
||||
setJumps((n) => n + 1);
|
||||
update(next);
|
||||
onPivot({ requestId: null, jobId: null, runId: null, ...patch });
|
||||
};
|
||||
|
||||
// "Any", then what the facet counted. The value in use stays offered even at zero — under the
|
||||
@@ -253,194 +226,130 @@ export default function AdminLogs() {
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="p-4 max-w-6xl w-full mx-auto">
|
||||
<Section card title={t("adminLogs.title")}>
|
||||
<p className="text-xs text-muted leading-relaxed mb-3">{t("adminLogs.intro")}</p>
|
||||
<HistoryFrame
|
||||
testId="admin-logs"
|
||||
list={logs}
|
||||
rowCount={rows.length}
|
||||
rowsClassName="font-mono text-2xs"
|
||||
failure={failure && { text: t(failure), onRetry: retryFailed }}
|
||||
older={
|
||||
olderCursor !== null
|
||||
? { label: t("adminLogs.older"), onClick: loadOlder, disabled: logs.isFetchingNextPage }
|
||||
: null
|
||||
}
|
||||
count={t("adminLogs.count", { count: rows.length })}
|
||||
empty={t("adminLogs.empty")}
|
||||
header={
|
||||
// The page's own band carries the shared filters and stays in reach; this one scrolls.
|
||||
<div>
|
||||
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
|
||||
<FilterChipGroup
|
||||
ariaLabel={t("adminLogs.levelLabel")}
|
||||
value={state.group}
|
||||
onChange={(group) => update({ group })}
|
||||
options={GROUPS.map((g) => ({
|
||||
value: g,
|
||||
label: t(`settings.myLog.filters.${g}`),
|
||||
count: groupCount(g),
|
||||
}))}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div
|
||||
data-testid="admin-logs"
|
||||
// overflow-clip, not -hidden: a hidden box is a scroll container, and the toolbar would stick
|
||||
// to it instead of to the page.
|
||||
className="overflow-clip rounded-xl border border-border bg-bg"
|
||||
>
|
||||
{/* The console's toolbar is the page's band (B4, B9): it sticks at the top with the fade
|
||||
under it while the lines scroll, so a filter is in reach at line 500. */}
|
||||
<div className="sticky top-0 z-base bg-bg fade-below">
|
||||
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
|
||||
<label
|
||||
className={clsx(
|
||||
fieldClass({ size: "sm", pill: true }),
|
||||
"flex min-w-[12rem] flex-1 items-center gap-1.5",
|
||||
)}
|
||||
>
|
||||
<Search className="h-3.5 w-3.5 shrink-0 text-muted" aria-hidden />
|
||||
<InnerInput
|
||||
type="search"
|
||||
value={state.q}
|
||||
onChange={(e) => update({ q: e.target.value.slice(0, 200) })}
|
||||
placeholder={t("adminLogs.search")}
|
||||
aria-label={t("adminLogs.search")}
|
||||
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
|
||||
<FilterFoldButton fold={fold} label={t("sidebar.filters")} count={facetFilters} />
|
||||
{!fold.folded && (
|
||||
<>
|
||||
<FilterSelect
|
||||
label={t("adminLogs.source")}
|
||||
options={facetOptions(facets.data?.sources, state.sources[0] ?? null, (v) => v)}
|
||||
selected={[state.sources[0] ?? ""]}
|
||||
onChange={([source]) => update({ sources: source ? [source] : [] })}
|
||||
testId="logs-source"
|
||||
/>
|
||||
</label>
|
||||
<FilterChipGroup
|
||||
ariaLabel={t("adminLogs.levelLabel")}
|
||||
value={state.group}
|
||||
onChange={(group) => update({ group })}
|
||||
options={GROUPS.map((g) => ({
|
||||
value: g,
|
||||
label: t(`settings.myLog.filters.${g}`),
|
||||
count: groupCount(g),
|
||||
}))}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
|
||||
<FilterFoldButton fold={fold} label={t("sidebar.filters")} count={facetFilters} />
|
||||
{!fold.folded && (
|
||||
<>
|
||||
<FilterSelect
|
||||
label={t("adminLogs.source")}
|
||||
options={facetOptions(facets.data?.sources, state.sources[0] ?? null, (v) => v)}
|
||||
selected={[state.sources[0] ?? ""]}
|
||||
onChange={([source]) => update({ sources: source ? [source] : [] })}
|
||||
testId="logs-source"
|
||||
/>
|
||||
<FilterSelect
|
||||
label={t("adminLogs.account")}
|
||||
options={facetOptions(facets.data?.users, state.user, accountLabel)}
|
||||
selected={[state.user ?? ""]}
|
||||
onChange={([user]) => update({ user: user || null })}
|
||||
testId="logs-user"
|
||||
/>
|
||||
<FilterSelect
|
||||
label={t("adminLogs.event")}
|
||||
options={facetOptions(facets.data?.events, state.event, (v) =>
|
||||
v === "none" ? t("adminLogs.noEvent") : v,
|
||||
)}
|
||||
selected={[state.event ?? ""]}
|
||||
onChange={([event]) => update({ event: event || null })}
|
||||
testId="logs-event"
|
||||
/>
|
||||
<FilterSelect
|
||||
label={t("adminLogs.window")}
|
||||
options={WINDOWS.map((w) => ({ value: w, label: t(`adminLogs.windows.${w}`) }))}
|
||||
selected={[state.window]}
|
||||
onChange={([window]) => update({ window: window as LogWindow })}
|
||||
testId="logs-window"
|
||||
/>
|
||||
</>
|
||||
)}
|
||||
<span className="ml-auto flex gap-1.5">
|
||||
<Button
|
||||
variant="outline"
|
||||
size="xs"
|
||||
icon={<Radio className="h-3.5 w-3.5" />}
|
||||
pressed={polling}
|
||||
onClick={toggleLive}
|
||||
>
|
||||
{t("adminLogs.live")}
|
||||
</Button>
|
||||
<Button
|
||||
variant="outline"
|
||||
size="xs"
|
||||
icon={<Copy className="h-3.5 w-3.5" />}
|
||||
// While a filter change loads, the rows on screen are the previous filter's.
|
||||
disabled={rows.length === 0 || logs.isPlaceholderData}
|
||||
onClick={() => void copy()}
|
||||
>
|
||||
{t("adminLogs.copy")}
|
||||
</Button>
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{(state.requestId || state.jobId !== null) && (
|
||||
<div className="flex flex-wrap items-center gap-2 border-b border-border px-2 py-1.5 text-xs">
|
||||
{state.requestId && (
|
||||
<PivotChip
|
||||
label={t("adminLogs.pivotRequest", { id: state.requestId })}
|
||||
clearLabel={t("adminLogs.clearPivot")}
|
||||
onClear={() => update({ requestId: null })}
|
||||
/>
|
||||
)}
|
||||
{state.jobId !== null && (
|
||||
<PivotChip
|
||||
label={t("adminLogs.pivotJob", { id: state.jobId })}
|
||||
clearLabel={t("adminLogs.clearPivot")}
|
||||
onClear={() => update({ jobId: null })}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
<FilterSelect
|
||||
label={t("adminLogs.event")}
|
||||
options={facetOptions(facets.data?.events, state.event, (v) =>
|
||||
v === "none" ? t("adminLogs.noEvent") : v,
|
||||
)}
|
||||
selected={[state.event ?? ""]}
|
||||
onChange={([event]) => update({ event: event || null })}
|
||||
testId="logs-event"
|
||||
/>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{failure && (
|
||||
<StateMessage
|
||||
tone="error"
|
||||
compact
|
||||
onRetry={retryFailed}
|
||||
className="border-b border-border"
|
||||
>
|
||||
{t(`adminLogs.${failure}`)}
|
||||
</StateMessage>
|
||||
)}
|
||||
|
||||
<div
|
||||
className={`font-mono text-2xs ${logs.isPlaceholderData ? "opacity-60" : ""}`}
|
||||
data-testid="admin-logs-lines"
|
||||
aria-busy={logs.isPlaceholderData}
|
||||
>
|
||||
{logs.isError && !logs.data ? (
|
||||
<StateMessage tone="error" onRetry={() => void logs.refetch()}>
|
||||
{t("common.loadError")}
|
||||
</StateMessage>
|
||||
) : logs.isLoading ? (
|
||||
<LoadingState shape="table" />
|
||||
) : rows.length === 0 ? (
|
||||
<p className="px-3 py-6 text-center font-sans text-sm text-muted">
|
||||
{t("adminLogs.empty")}
|
||||
</p>
|
||||
) : (
|
||||
<NavGrid>
|
||||
{rows.map((row, index) => (
|
||||
<LogLine
|
||||
navIndex={index}
|
||||
key={row.id}
|
||||
ts={row.ts}
|
||||
level={row.level}
|
||||
event={row.event ?? "—"}
|
||||
msg={row.msg}
|
||||
title={formatDateTimeSec(row.ts, lang)}
|
||||
open={open === row.id}
|
||||
onToggle={() => setOpen(open === row.id ? null : row.id)}
|
||||
>
|
||||
<RowDetail
|
||||
row={row}
|
||||
account={row.user_id === null ? null : accountLabel(String(row.user_id))}
|
||||
onRequest={(requestId) => pivot({ requestId, jobId: null })}
|
||||
onJob={(jobId) => pivot({ requestId: null, jobId })}
|
||||
/>
|
||||
</LogLine>
|
||||
))}
|
||||
</NavGrid>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap items-center justify-between gap-2 border-t border-border px-3 py-1.5 text-xs text-muted">
|
||||
<span>{t("adminLogs.count", { count: rows.length })}</span>
|
||||
{olderCursor !== null && (
|
||||
<span className="ml-auto flex gap-1.5">
|
||||
<Button
|
||||
variant="outline"
|
||||
size="xs"
|
||||
disabled={logs.isFetchingNextPage}
|
||||
onClick={loadOlder}
|
||||
icon={<Radio className="h-3.5 w-3.5" />}
|
||||
pressed={polling}
|
||||
onClick={toggleLive}
|
||||
>
|
||||
{t("adminLogs.older")}
|
||||
{t("adminLogs.live")}
|
||||
</Button>
|
||||
)}
|
||||
<Button
|
||||
variant="outline"
|
||||
size="xs"
|
||||
icon={<Copy className="h-3.5 w-3.5" />}
|
||||
// While a filter change loads, the rows on screen are the previous filter's.
|
||||
disabled={rows.length === 0 || logs.isPlaceholderData}
|
||||
onClick={() => void copy()}
|
||||
>
|
||||
{t("adminLogs.copy")}
|
||||
</Button>
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{(state.requestId || state.jobId !== null || state.runId !== null) && (
|
||||
<div className="flex flex-wrap items-center gap-2 border-b border-border px-2 py-1.5 text-xs">
|
||||
{state.requestId && (
|
||||
<PivotChip
|
||||
label={t("adminLogs.pivotRequest", { id: state.requestId })}
|
||||
clearLabel={t("adminLogs.clearPivot")}
|
||||
onClear={() => update({ requestId: null })}
|
||||
/>
|
||||
)}
|
||||
{state.jobId !== null && (
|
||||
<PivotChip
|
||||
label={t("adminLogs.pivotJob", { id: state.jobId })}
|
||||
clearLabel={t("adminLogs.clearPivot")}
|
||||
onClear={() => update({ jobId: null })}
|
||||
/>
|
||||
)}
|
||||
{state.runId !== null && (
|
||||
<PivotChip
|
||||
label={t("adminLogs.pivotRun", { id: state.runId })}
|
||||
clearLabel={t("adminLogs.clearPivot")}
|
||||
onClear={() => update({ runId: null })}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</Section>
|
||||
</div>
|
||||
}
|
||||
>
|
||||
{rows.map((row, index) => (
|
||||
<LogLine
|
||||
navIndex={index}
|
||||
key={row.id}
|
||||
ts={row.ts}
|
||||
level={row.level}
|
||||
event={row.event ?? "—"}
|
||||
msg={row.msg}
|
||||
title={formatDateTimeSec(row.ts, lang)}
|
||||
open={open === row.id}
|
||||
onToggle={() => setOpen(open === row.id ? null : row.id)}
|
||||
>
|
||||
<RowDetail
|
||||
row={row}
|
||||
account={row.user_id === null ? null : accountLabel(String(row.user_id))}
|
||||
onRequest={(requestId) => pivot({ requestId })}
|
||||
onJob={(jobId) => pivot({ jobId })}
|
||||
onRun={(runId) => pivot({ runId })}
|
||||
/>
|
||||
</LogLine>
|
||||
))}
|
||||
</HistoryFrame>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -465,11 +374,13 @@ function RowDetail({
|
||||
account,
|
||||
onRequest,
|
||||
onJob,
|
||||
onRun,
|
||||
}: {
|
||||
row: ServerLogRow;
|
||||
account: string | null;
|
||||
onRequest: (id: string) => void;
|
||||
onJob: (id: number) => void;
|
||||
onRun: (id: number) => void;
|
||||
}) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
@@ -516,6 +427,21 @@ function RowDetail({
|
||||
</dd>
|
||||
</>
|
||||
)}
|
||||
{row.run_id !== null && (
|
||||
<>
|
||||
<dt>{t("adminLogs.fields.run")}</dt>
|
||||
<dd>
|
||||
<Button
|
||||
variant="link"
|
||||
className={pivotCls}
|
||||
title={t("adminLogs.showRun")}
|
||||
onClick={() => onRun(row.run_id!)}
|
||||
>
|
||||
{row.run_id}
|
||||
</Button>
|
||||
</dd>
|
||||
</>
|
||||
)}
|
||||
{row.version && (
|
||||
<>
|
||||
<dt>{t("adminLogs.fields.version")}</dt>
|
||||
|
||||
@@ -0,0 +1,269 @@
|
||||
// @vitest-environment jsdom
|
||||
//
|
||||
// Settings → API tokens: a row per token says what it is, what it opens with which rights, when it was
|
||||
// last used and when it runs out (marked when that is soon); revoking asks first. A new token is the
|
||||
// Hub exchange in one click, or folders and rights picked one by one — never Write where an upload
|
||||
// cannot land — and its value is shown once, with a curl line to try it. i18n is key-passthrough
|
||||
// (a key's options ride along in brackets, so a test can read what was named).
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { api, type ApiToken, type FileFolder } from "../lib/api";
|
||||
|
||||
const mocks = vi.hoisted(() => ({ confirm: vi.fn(async (_o?: unknown) => true) }));
|
||||
vi.mock("react-i18next", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("react-i18next")>()),
|
||||
useTranslation: () => ({
|
||||
i18n: { language: "en" },
|
||||
t: (key: string, o?: Record<string, unknown>) =>
|
||||
o && Object.keys(o).length ? `${key}(${Object.values(o).join(",")})` : key,
|
||||
}),
|
||||
}));
|
||||
vi.mock("./confirmContext", () => ({ useConfirm: () => mocks.confirm }));
|
||||
vi.mock("../lib/clipboard", () => ({ copyText: vi.fn(async () => true) }));
|
||||
|
||||
import ApiTokens from "./ApiTokens";
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.restoreAllMocks();
|
||||
mocks.confirm.mockClear();
|
||||
});
|
||||
|
||||
const token = (id: number, extra: Partial<ApiToken> = {}): ApiToken => ({
|
||||
id,
|
||||
name: `Token ${id}`,
|
||||
prefix: `sfl_${id}abcdefg`,
|
||||
created_at: "2026-10-01T00:00:00Z",
|
||||
expires_at: "2027-10-01T00:00:00Z",
|
||||
expired: false,
|
||||
expiring: false,
|
||||
last_used_at: null,
|
||||
grants: [],
|
||||
...extra,
|
||||
});
|
||||
const folder = (id: number, extra: Partial<FileFolder>): FileFolder => ({
|
||||
id,
|
||||
kind: "source",
|
||||
source_key: null,
|
||||
name: null,
|
||||
retention_days: 30,
|
||||
count: 1,
|
||||
size_bytes: 0,
|
||||
created_at: null,
|
||||
has_link: false,
|
||||
...extra,
|
||||
});
|
||||
|
||||
function mount(tokens: ApiToken[] | Error, folders: FileFolder[] = []) {
|
||||
if (tokens instanceof Error) vi.spyOn(api, "apiTokens").mockRejectedValue(tokens);
|
||||
else vi.spyOn(api, "apiTokens").mockResolvedValue(tokens);
|
||||
vi.spyOn(api, "fileFolders").mockResolvedValue(folders);
|
||||
render(
|
||||
<QueryClientProvider
|
||||
client={new QueryClient({ defaultOptions: { queries: { retry: false } } })}
|
||||
>
|
||||
<ApiTokens />
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
}
|
||||
|
||||
const rows = () => screen.getAllByTestId("api-token-row");
|
||||
const dialog = () => screen.getByRole("dialog");
|
||||
const pressed = (b: HTMLElement) => b.getAttribute("aria-pressed") === "true";
|
||||
const inRow = (group: string, name: string) =>
|
||||
[...screen.getByRole("group", { name: group }).querySelectorAll("button")].find(
|
||||
(b) => b.textContent === name,
|
||||
);
|
||||
|
||||
describe("API tokens · the list", () => {
|
||||
it("reads what a token opens, with which rights, when it was used and when it runs out", async () => {
|
||||
mount([
|
||||
token(1, {
|
||||
name: "Hub exchange",
|
||||
last_used_at: new Date().toISOString(),
|
||||
grants: [
|
||||
{
|
||||
folder_id: 7,
|
||||
kind: "hub",
|
||||
source_key: null,
|
||||
name: null,
|
||||
rights: ["write", "list", "read", "delete"],
|
||||
},
|
||||
],
|
||||
}),
|
||||
token(2, {
|
||||
expiring: true,
|
||||
expires_at: null,
|
||||
grants: [
|
||||
{ folder_id: 3, kind: "own", source_key: null, name: "Receptek", rights: ["read"] },
|
||||
],
|
||||
}),
|
||||
token(3, { expired: true }),
|
||||
]);
|
||||
await waitFor(() => expect(rows()).toHaveLength(3));
|
||||
const [hub, soon, gone] = rows();
|
||||
expect(hub!.textContent).toContain("sfl_1abcdefg…");
|
||||
// Rights in the server's order, whatever order they came in.
|
||||
expect(hub!.textContent).toContain(
|
||||
"files.folder.hub· settings.tokens.rights.list, settings.tokens.rights.read, settings.tokens.rights.write, settings.tokens.rights.delete",
|
||||
);
|
||||
expect(hub!.textContent).toContain("settings.tokens.lastUsed");
|
||||
expect(hub!.textContent).not.toContain("settings.tokens.expiring");
|
||||
expect(soon!.textContent).toContain("Receptek· settings.tokens.rights.read");
|
||||
expect(soon!.textContent).toContain("settings.tokens.expiring");
|
||||
expect(soon!.textContent).toContain("settings.tokens.neverUsed");
|
||||
expect(soon!.textContent).toContain("settings.tokens.neverExpires");
|
||||
expect(gone!.textContent).toContain("settings.tokens.expired");
|
||||
expect(gone!.textContent).toContain("settings.tokens.expiredOn(");
|
||||
});
|
||||
|
||||
it("says when there is none, and when the list could not be read", async () => {
|
||||
mount([]);
|
||||
expect(await screen.findByText("settings.tokens.empty")).toBeTruthy();
|
||||
cleanup();
|
||||
mount(new Error("down"));
|
||||
expect(await screen.findByText("common.loadError")).toBeTruthy();
|
||||
expect(screen.queryByText("settings.tokens.empty")).toBeNull();
|
||||
});
|
||||
|
||||
it("revokes from the row's menu, only once the confirm says yes", async () => {
|
||||
mount([token(5, { name: "Phone" })]);
|
||||
const revoke = vi.spyOn(api, "revokeApiToken").mockResolvedValue({ ok: true, deleted: 5 });
|
||||
await waitFor(() => expect(rows()).toHaveLength(1));
|
||||
mocks.confirm.mockResolvedValueOnce(false);
|
||||
const open = async () => {
|
||||
act(() => {
|
||||
fireEvent.contextMenu(rows()[0]!);
|
||||
});
|
||||
await waitFor(() =>
|
||||
expect(screen.getAllByRole("menuitem", { hidden: true }).length).toBeGreaterThan(0),
|
||||
);
|
||||
fireEvent.click(
|
||||
screen
|
||||
.getAllByRole("menuitem", { hidden: true })
|
||||
.find((b) => b.textContent === "settings.tokens.revoke")!,
|
||||
);
|
||||
};
|
||||
await open();
|
||||
await waitFor(() => expect(mocks.confirm).toHaveBeenCalledTimes(1));
|
||||
expect(mocks.confirm.mock.calls[0]![0]).toMatchObject({
|
||||
title: "settings.tokens.revokeTitle(Phone)",
|
||||
danger: true,
|
||||
});
|
||||
expect(revoke).not.toHaveBeenCalled();
|
||||
await open();
|
||||
await waitFor(() => expect(revoke).toHaveBeenCalledWith(5));
|
||||
});
|
||||
});
|
||||
|
||||
describe("API tokens · making one", () => {
|
||||
const made = (extra: Partial<ApiToken> = {}) => ({
|
||||
ok: true as const,
|
||||
...token(9, extra),
|
||||
token: "sfl_SECRETVALUE",
|
||||
});
|
||||
|
||||
it("makes the Hub exchange in one click, a year long, and shows the token once with a curl line", async () => {
|
||||
mount([]);
|
||||
const create = vi.spyOn(api, "createApiToken").mockResolvedValue(made());
|
||||
fireEvent.click(await screen.findByText("settings.tokens.new"));
|
||||
expect(pressed(screen.getByRole("button", { name: "settings.tokens.hubTitle" }))).toBe(true);
|
||||
expect((screen.getByLabelText("settings.tokens.name") as HTMLInputElement).value).toBe(
|
||||
"settings.tokens.hubName",
|
||||
);
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.create" }));
|
||||
await waitFor(() =>
|
||||
expect(create).toHaveBeenCalledWith({
|
||||
name: "settings.tokens.hubName",
|
||||
expires_days: 365,
|
||||
preset: "hub",
|
||||
}),
|
||||
);
|
||||
await screen.findByText("settings.tokens.madeTitle");
|
||||
const codes = [...dialog().querySelectorAll("code")].map((c) => c.textContent);
|
||||
expect(codes[0]).toBe("sfl_SECRETVALUE");
|
||||
expect(codes[1]).toBe(
|
||||
`curl -H "Authorization: Bearer sfl_SECRETVALUE" ${window.location.origin}/api/files/folders`,
|
||||
);
|
||||
// The keyboard lands on copying it, not on the closed form's button.
|
||||
expect(document.activeElement?.textContent).toBe("settings.tokens.copy");
|
||||
// Two Copy buttons in one dialog: each is named by the value it copies.
|
||||
expect(
|
||||
screen.getByRole("button", { name: "settings.tokens.copyField(settings.tokens.token)" }),
|
||||
).toBe(document.activeElement);
|
||||
screen.getByRole("button", { name: "settings.tokens.copyField(settings.tokens.tryIt)" });
|
||||
// Once copied, its name says what it shows, so a voice command saying "Copied" finds it.
|
||||
fireEvent.click(document.activeElement!);
|
||||
await screen.findByRole("button", {
|
||||
name: "settings.tokens.copiedField(settings.tokens.token)",
|
||||
});
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.done" }));
|
||||
expect(screen.queryByRole("dialog")).toBeNull();
|
||||
});
|
||||
|
||||
it("a custom token takes the folders and rights picked; no Write where nothing can be uploaded", async () => {
|
||||
mount(
|
||||
[],
|
||||
[
|
||||
folder(1, { source_key: "youtube" }),
|
||||
folder(2, { source_key: "upload" }),
|
||||
folder(3, { source_key: "site:x.com", count: 0 }), // empty: not offered, as the rail hides it
|
||||
folder(4, { kind: "hub", count: 0 }),
|
||||
folder(5, { kind: "own", name: "Receptek" }),
|
||||
],
|
||||
);
|
||||
const create = vi.spyOn(api, "createApiToken").mockResolvedValue(made());
|
||||
fireEvent.click(await screen.findByText("settings.tokens.new"));
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.customTitle" }));
|
||||
const name = screen.getByLabelText("settings.tokens.name") as HTMLInputElement;
|
||||
expect(name.value).toBe("");
|
||||
await screen.findByRole("group", { name: "Receptek" });
|
||||
expect(
|
||||
screen
|
||||
.getAllByRole("group")
|
||||
.map((g) => g.getAttribute("aria-label"))
|
||||
.filter((l) => l !== "settings.tokens.kind" && l !== "settings.tokens.expiry"),
|
||||
).toEqual(["files.source.upload", "files.source.youtube", "files.folder.hub", "Receptek"]);
|
||||
expect(inRow("files.source.youtube", "settings.tokens.rights.write")).toBeUndefined();
|
||||
expect(inRow("files.source.upload", "settings.tokens.rights.write")).toBeTruthy();
|
||||
expect(inRow("files.folder.hub", "settings.tokens.rights.write")).toBeTruthy();
|
||||
|
||||
const createBtn = screen.getByRole("button", { name: "settings.tokens.create" });
|
||||
fireEvent.change(name, { target: { value: " Phone " } });
|
||||
expect(createBtn.hasAttribute("disabled")).toBe(true); // a name, but no right yet
|
||||
fireEvent.click(inRow("Receptek", "settings.tokens.rights.list")!);
|
||||
fireEvent.click(inRow("Receptek", "settings.tokens.rights.write")!);
|
||||
fireEvent.click(inRow("files.source.youtube", "settings.tokens.rights.read")!);
|
||||
// Picked and dropped again: no grant for that folder.
|
||||
fireEvent.click(inRow("files.folder.hub", "settings.tokens.rights.read")!);
|
||||
fireEvent.click(inRow("files.folder.hub", "settings.tokens.rights.read")!);
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.expiryOption.30" }));
|
||||
expect(createBtn.hasAttribute("disabled")).toBe(false);
|
||||
fireEvent.click(createBtn);
|
||||
await waitFor(() => expect(create).toHaveBeenCalledTimes(1));
|
||||
const body = create.mock.calls[0]![0];
|
||||
expect(body).toMatchObject({ name: "Phone", expires_days: 30 });
|
||||
expect(body.preset).toBeUndefined();
|
||||
expect(body.grants).toEqual(
|
||||
expect.arrayContaining([
|
||||
{ folder_id: 5, rights: ["list", "write"] },
|
||||
{ folder_id: 1, rights: ["read"] },
|
||||
]),
|
||||
);
|
||||
expect(body.grants).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("a token that never runs out says what that means, and is sent without an expiry", async () => {
|
||||
mount([]);
|
||||
const create = vi.spyOn(api, "createApiToken").mockResolvedValue(made());
|
||||
fireEvent.click(await screen.findByText("settings.tokens.new"));
|
||||
expect(screen.queryByText("settings.tokens.neverWarning")).toBeNull();
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.expiryOption.never" }));
|
||||
expect(screen.getByText("settings.tokens.neverWarning")).toBeTruthy();
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.tokens.create" }));
|
||||
await waitFor(() =>
|
||||
expect(create).toHaveBeenCalledWith(expect.objectContaining({ expires_days: null })),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,504 @@
|
||||
import { useId, useState, type ReactNode } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import type { TFunction } from "i18next";
|
||||
import { useMutation, useQuery, useQueryClient, type UseQueryResult } from "@tanstack/react-query";
|
||||
import { Check, Copy, KeyRound, Plus, SlidersHorizontal, Trash2 } from "lucide-react";
|
||||
import {
|
||||
api,
|
||||
TOKEN_EXPIRY_DAYS,
|
||||
TOKEN_EXPIRY_DEFAULT,
|
||||
TOKEN_RIGHTS,
|
||||
type ApiToken,
|
||||
type FileFolder,
|
||||
type TokenGrant,
|
||||
type TokenRight,
|
||||
} from "../lib/api";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { compareText } from "../lib/collate";
|
||||
import { formatDate } from "../lib/format";
|
||||
import { navItemAttrs } from "../lib/listNav";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import { folderIcon, folderLabel, takesUploads } from "./fileFolders";
|
||||
import { type ActionMenuItem } from "./ui/ActionMenu";
|
||||
import { Badge } from "./ui/Badge";
|
||||
import { Button } from "./ui/Button";
|
||||
import { Chip } from "./ui/Chip";
|
||||
import { ContextMenu } from "./ui/ContextMenu";
|
||||
import InlineNotice from "./ui/InlineNotice";
|
||||
import { Input } from "./ui/Input";
|
||||
import Modal, { ModalFooter } from "./ui/Modal";
|
||||
import { MoreButton } from "./ui/MoreButton";
|
||||
import NavGrid from "./ui/NavGrid";
|
||||
import { Pressable } from "./ui/Pressable";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import { RelativeTime } from "./ui/RelativeTime";
|
||||
import SegmentedToggle from "./ui/SegmentedToggle";
|
||||
import { SURFACE } from "./ui/surfaces";
|
||||
import { useContextMenu } from "./ui/useContextMenu";
|
||||
import { useCopyFeedback } from "./useCopyFeedback";
|
||||
|
||||
// Settings → API tokens: a machine's key to the user's Files library (docs/DESIGN.md decision log,
|
||||
// "API tokens"). A row per token in three lines — what it is, what it opens, when it was used and
|
||||
// when it runs out — revoked from its "…" behind a confirm. A new token is made in a dialog that
|
||||
// asks the kind first (the one-click Hub exchange, or folders and rights of the user's own picking)
|
||||
// and, once made, shows the token in the same dialog: the one time it can be seen.
|
||||
|
||||
/** The rights a token holds in a folder, in the reader's words: "list, read, write". */
|
||||
function rightsText(rights: readonly TokenRight[], t: TFunction, lang: string): string {
|
||||
return TOKEN_RIGHTS.filter((r) => rights.includes(r))
|
||||
.map((r) => t(`settings.tokens.rights.${r}`).toLocaleLowerCase(lang))
|
||||
.join(", ");
|
||||
}
|
||||
|
||||
/** A folder's glyph (`folderIcon`), drawn at a row's size. */
|
||||
function Glyph({ icon: Icon }: { icon: ReturnType<typeof folderIcon> }) {
|
||||
return <Icon className="size-3.5 shrink-0" aria-hidden />;
|
||||
}
|
||||
|
||||
function GrantLine({ grant }: { grant: TokenGrant }) {
|
||||
const { t, i18n } = useTranslation();
|
||||
return (
|
||||
<span className="flex min-w-0 items-center gap-1.5">
|
||||
<Glyph icon={folderIcon(grant)} />
|
||||
<span className="text-fg">{folderLabel(grant, t)}</span>
|
||||
<span>· {rightsText(grant.rights, t, i18n.language)}</span>
|
||||
</span>
|
||||
);
|
||||
}
|
||||
|
||||
function TokenRow({
|
||||
token,
|
||||
index,
|
||||
rowProps,
|
||||
more,
|
||||
}: {
|
||||
token: ApiToken;
|
||||
index: number;
|
||||
rowProps: Record<string, unknown>;
|
||||
more: ReactNode;
|
||||
}) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const expires = formatDate(token.expires_at, i18n.language);
|
||||
return (
|
||||
<div
|
||||
{...navItemAttrs(index, token.name, { inset: true })}
|
||||
{...rowProps}
|
||||
data-testid="api-token-row"
|
||||
className="flex items-start gap-3 py-2.5"
|
||||
>
|
||||
<KeyRound className="mt-0.5 size-4 shrink-0 text-muted" aria-hidden />
|
||||
<div className="flex min-w-0 flex-1 flex-col gap-1">
|
||||
<span className="flex flex-wrap items-center gap-2">
|
||||
<span className="text-sm font-medium break-all">{token.name}</span>
|
||||
<code className="text-xs text-muted">{token.prefix}…</code>
|
||||
{token.expired ? (
|
||||
<Badge tone="warn">{t("settings.tokens.expired")}</Badge>
|
||||
) : (
|
||||
token.expiring && <Badge tone="warn">{t("settings.tokens.expiring")}</Badge>
|
||||
)}
|
||||
</span>
|
||||
<span className="flex flex-wrap items-center gap-x-3 gap-y-1 text-xs text-muted">
|
||||
{token.grants.map((g) => (
|
||||
<GrantLine key={g.folder_id} grant={g} />
|
||||
))}
|
||||
</span>
|
||||
<span className="text-xs text-muted">
|
||||
{token.last_used_at ? (
|
||||
<>
|
||||
{t("settings.tokens.lastUsed")} <RelativeTime at={token.last_used_at} />
|
||||
</>
|
||||
) : (
|
||||
t("settings.tokens.neverUsed")
|
||||
)}
|
||||
{" · "}
|
||||
{token.expires_at === null
|
||||
? t("settings.tokens.neverExpires")
|
||||
: token.expired
|
||||
? t("settings.tokens.expiredOn", { date: expires })
|
||||
: t("settings.tokens.expiresOn", { date: expires })}
|
||||
</span>
|
||||
</div>
|
||||
{more}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function ApiTokens() {
|
||||
const { t } = useTranslation();
|
||||
const qc = useQueryClient();
|
||||
const confirm = useConfirm();
|
||||
const [making, setMaking] = useState(false);
|
||||
const tokensQ = useQuery({ queryKey: qk.apiTokens(), queryFn: api.apiTokens });
|
||||
const revoke = useMutation({
|
||||
mutationFn: (id: number) => api.revokeApiToken(id),
|
||||
onSuccess: () => qc.invalidateQueries({ queryKey: qk.apiTokens() }),
|
||||
meta: { failTitle: "errors.writeFailed.apiToken.revoke" },
|
||||
});
|
||||
const onRevoke = async (token: ApiToken) => {
|
||||
const ok = await confirm({
|
||||
title: t("settings.tokens.revokeTitle", { name: token.name }),
|
||||
message: t("settings.tokens.revokeBody"),
|
||||
confirmLabel: t("settings.tokens.revokeAction"),
|
||||
danger: true,
|
||||
});
|
||||
if (ok) revoke.mutate(token.id);
|
||||
};
|
||||
const menu = useContextMenu<ApiToken>({ keyOf: (tk) => tk.id });
|
||||
const items = (token: ApiToken): ActionMenuItem[] => [
|
||||
{
|
||||
id: "revoke",
|
||||
label: t("settings.tokens.revoke"),
|
||||
icon: <Trash2 className="w-4 h-4" />,
|
||||
danger: true,
|
||||
onSelect: () => void onRevoke(token),
|
||||
},
|
||||
];
|
||||
const tokens = tokensQ.data;
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-3">
|
||||
<div className="flex items-start gap-3">
|
||||
<p className="flex-1 text-xs leading-relaxed text-muted">{t("settings.tokens.intro")}</p>
|
||||
<Button
|
||||
variant="primary"
|
||||
size="sm"
|
||||
icon={<Plus className="w-4 h-4" />}
|
||||
onClick={() => setMaking(true)}
|
||||
>
|
||||
{t("settings.tokens.new")}
|
||||
</Button>
|
||||
</div>
|
||||
{tokensQ.isError && !tokens ? (
|
||||
<StateMessage tone="error" onRetry={() => void tokensQ.refetch()}>
|
||||
{t("common.loadError")}
|
||||
</StateMessage>
|
||||
) : !tokens ? (
|
||||
<LoadingState shape="list" />
|
||||
) : tokens.length === 0 ? (
|
||||
<p className="py-4 text-sm text-muted">{t("settings.tokens.empty")}</p>
|
||||
) : (
|
||||
<NavGrid className="flex flex-col divide-y divide-border/60">
|
||||
{tokens.map((token, i) => (
|
||||
<TokenRow
|
||||
key={token.id}
|
||||
token={token}
|
||||
index={i}
|
||||
rowProps={menu.triggerProps(token)}
|
||||
more={
|
||||
<MoreButton
|
||||
label={t("settings.tokens.rowMenu")}
|
||||
expanded={menu.open && menu.target?.id === token.id}
|
||||
menuProps={menu.buttonProps(token)}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
))}
|
||||
</NavGrid>
|
||||
)}
|
||||
<ContextMenu
|
||||
menu={menu}
|
||||
ariaLabel={t("settings.tokens.rowMenu")}
|
||||
items={menu.target ? items(menu.target) : []}
|
||||
/>
|
||||
{making && <NewTokenDialog onClose={() => setMaking(false)} />}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
// --- making one -----------------------------------------------------------------------------------
|
||||
|
||||
type Kind = "hub" | "custom";
|
||||
type Grants = Record<number, readonly TokenRight[]>;
|
||||
|
||||
/** The folders a custom token may be given, as the Files rail shows them: the sources that hold
|
||||
* something, then the Hub folder, then the user's own. A right a folder cannot take (writing into a
|
||||
* site's folder) is not offered. */
|
||||
function grantable(folders: readonly FileFolder[], t: TFunction, lang: string): FileFolder[] {
|
||||
const byName = (a: FileFolder, b: FileFolder) =>
|
||||
compareText(folderLabel(a, t), folderLabel(b, t), lang);
|
||||
return [
|
||||
...folders.filter((f) => f.kind === "source" && f.count > 0).sort(byName),
|
||||
...folders.filter((f) => f.kind === "hub"),
|
||||
...folders.filter((f) => f.kind === "own").sort(byName),
|
||||
];
|
||||
}
|
||||
|
||||
function RightsGrid({
|
||||
foldersQ,
|
||||
grants,
|
||||
onChange,
|
||||
}: {
|
||||
foldersQ: UseQueryResult<FileFolder[]>;
|
||||
grants: Grants;
|
||||
onChange: (g: Grants) => void;
|
||||
}) {
|
||||
const { t, i18n } = useTranslation();
|
||||
if (foldersQ.isError && !foldersQ.data)
|
||||
return (
|
||||
<StateMessage tone="error" onRetry={() => void foldersQ.refetch()}>
|
||||
{t("common.loadError")}
|
||||
</StateMessage>
|
||||
);
|
||||
if (!foldersQ.data) return <LoadingState />;
|
||||
const rows = grantable(foldersQ.data, t, i18n.language);
|
||||
if (rows.length === 0)
|
||||
return <p className="text-sm text-muted">{t("settings.tokens.noFolders")}</p>;
|
||||
const toggle = (folder: FileFolder, right: TokenRight) => {
|
||||
const had = grants[folder.id] ?? [];
|
||||
const next = had.includes(right) ? had.filter((r) => r !== right) : [...had, right];
|
||||
onChange({ ...grants, [folder.id]: next });
|
||||
};
|
||||
return (
|
||||
<div className="flex flex-col divide-y divide-border/50 rounded-lg border border-border/60">
|
||||
{rows.map((f) => {
|
||||
const label = folderLabel(f, t);
|
||||
return (
|
||||
<div
|
||||
key={f.id}
|
||||
role="group"
|
||||
aria-label={label}
|
||||
className="flex flex-wrap items-center gap-2 px-2.5 py-2"
|
||||
>
|
||||
<span className="flex min-w-28 flex-1 items-center gap-1.5 text-sm">
|
||||
<Glyph icon={folderIcon(f)} />
|
||||
<span className="break-all">{label}</span>
|
||||
</span>
|
||||
<span className="flex flex-wrap gap-1">
|
||||
{TOKEN_RIGHTS.map((r) =>
|
||||
r === "write" && !takesUploads(f) ? null : (
|
||||
<Chip
|
||||
key={r}
|
||||
pressed={(grants[f.id] ?? []).includes(r)}
|
||||
onClick={() => toggle(f, r)}
|
||||
>
|
||||
{t(`settings.tokens.rights.${r}`)}
|
||||
</Chip>
|
||||
),
|
||||
)}
|
||||
</span>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function KindTile({
|
||||
on,
|
||||
icon,
|
||||
title,
|
||||
text,
|
||||
onPick,
|
||||
}: {
|
||||
on: boolean;
|
||||
icon: ReactNode;
|
||||
title: string;
|
||||
text: string;
|
||||
onPick: () => void;
|
||||
}) {
|
||||
const textId = useId();
|
||||
return (
|
||||
<Pressable
|
||||
aria-pressed={on}
|
||||
aria-describedby={textId}
|
||||
onClick={onPick}
|
||||
className={`${SURFACE.card} flex flex-col items-start gap-1 border p-3 text-left ${
|
||||
on ? "border-accent" : "border-transparent hover:border-border"
|
||||
}`}
|
||||
>
|
||||
<span className="flex items-center gap-1.5 text-sm font-medium">
|
||||
{icon}
|
||||
{title}
|
||||
</span>
|
||||
{/* Read after the name, not into it ("Custom" — then what it means). */}
|
||||
<span id={textId} aria-hidden className="text-xs text-muted">
|
||||
{text}
|
||||
</span>
|
||||
</Pressable>
|
||||
);
|
||||
}
|
||||
|
||||
const HubIcon = folderIcon({ kind: "hub", source_key: null, name: null });
|
||||
|
||||
const expiryValue = (days: number | null) => (days === null ? "never" : String(days));
|
||||
|
||||
function NewTokenDialog({ onClose }: { onClose: () => void }) {
|
||||
const { t } = useTranslation();
|
||||
const qc = useQueryClient();
|
||||
const nameId = useId();
|
||||
const [kind, setKind] = useState<Kind>("hub");
|
||||
// The name as typed; untouched, the Hub exchange is named for what it is and a custom one is blank.
|
||||
const [nameDraft, setNameDraft] = useState<string | null>(null);
|
||||
const name = nameDraft ?? (kind === "hub" ? t("settings.tokens.hubName") : "");
|
||||
const [grants, setGrants] = useState<Grants>({});
|
||||
const [expiry, setExpiry] = useState<number | null>(TOKEN_EXPIRY_DEFAULT);
|
||||
// Read as the dialog opens, not when Custom is picked: the grid is there by the time it is wanted.
|
||||
const foldersQ = useQuery({ queryKey: qk.fileFolders(), queryFn: api.fileFolders });
|
||||
const picked = Object.entries(grants)
|
||||
.filter(([, rights]) => rights.length > 0)
|
||||
.map(([id, rights]) => ({ folder_id: Number(id), rights: [...rights] }));
|
||||
const create = useMutation({
|
||||
mutationFn: () =>
|
||||
api.createApiToken(
|
||||
kind === "hub"
|
||||
? { name: name.trim(), expires_days: expiry, preset: "hub" }
|
||||
: { name: name.trim(), expires_days: expiry, grants: picked },
|
||||
),
|
||||
onSuccess: () => {
|
||||
void qc.invalidateQueries({ queryKey: qk.apiTokens() });
|
||||
// The Hub exchange makes the Hub folder the first time.
|
||||
void qc.invalidateQueries({ queryKey: qk.fileFolders() });
|
||||
},
|
||||
meta: { failTitle: "errors.writeFailed.apiToken.create" },
|
||||
});
|
||||
const made = create.data;
|
||||
const ready = name.trim() !== "" && (kind === "hub" || picked.length > 0);
|
||||
|
||||
if (made)
|
||||
return (
|
||||
<Modal
|
||||
initialFocus="action"
|
||||
title={t("settings.tokens.madeTitle")}
|
||||
onClose={onClose}
|
||||
maxWidth="max-w-md"
|
||||
>
|
||||
<div className="flex flex-col gap-3">
|
||||
<InlineNotice>{t("settings.tokens.once")}</InlineNotice>
|
||||
<CopyField label={t("settings.tokens.token")} value={made.token} autoFocus />
|
||||
<CopyField
|
||||
label={t("settings.tokens.tryIt")}
|
||||
value={`curl -H "Authorization: Bearer ${made.token}" ${window.location.origin}/api/files/folders`}
|
||||
/>
|
||||
</div>
|
||||
<ModalFooter>
|
||||
<Button variant="primary" onClick={onClose}>
|
||||
{t("settings.tokens.done")}
|
||||
</Button>
|
||||
</ModalFooter>
|
||||
</Modal>
|
||||
);
|
||||
|
||||
return (
|
||||
<Modal
|
||||
initialFocus="field"
|
||||
title={t("settings.tokens.newTitle")}
|
||||
onClose={onClose}
|
||||
maxWidth="max-w-md"
|
||||
>
|
||||
<form
|
||||
id={`${nameId}-form`}
|
||||
className="flex flex-col gap-3"
|
||||
onSubmit={(e) => {
|
||||
e.preventDefault();
|
||||
if (ready && !create.isPending) create.mutate();
|
||||
}}
|
||||
>
|
||||
<div
|
||||
role="group"
|
||||
aria-label={t("settings.tokens.kind")}
|
||||
className="grid grid-cols-1 gap-2 sm:grid-cols-2"
|
||||
>
|
||||
<KindTile
|
||||
on={kind === "hub"}
|
||||
icon={<HubIcon className="size-4" aria-hidden />}
|
||||
title={t("settings.tokens.hubTitle")}
|
||||
text={t("settings.tokens.hubText")}
|
||||
onPick={() => setKind("hub")}
|
||||
/>
|
||||
<KindTile
|
||||
on={kind === "custom"}
|
||||
icon={<SlidersHorizontal className="size-4" aria-hidden />}
|
||||
title={t("settings.tokens.customTitle")}
|
||||
text={t("settings.tokens.customText")}
|
||||
onPick={() => setKind("custom")}
|
||||
/>
|
||||
</div>
|
||||
<label htmlFor={nameId} className="flex flex-col gap-1">
|
||||
<span className="text-xs text-muted">{t("settings.tokens.name")}</span>
|
||||
<Input
|
||||
id={nameId}
|
||||
data-initial-focus=""
|
||||
className="w-full"
|
||||
maxLength={80}
|
||||
value={name}
|
||||
onChange={(e) => setNameDraft(e.target.value)}
|
||||
/>
|
||||
</label>
|
||||
{kind === "custom" && (
|
||||
<div className="flex flex-col gap-1">
|
||||
<span className="text-xs text-muted">{t("settings.tokens.folders")}</span>
|
||||
<RightsGrid foldersQ={foldersQ} grants={grants} onChange={setGrants} />
|
||||
</div>
|
||||
)}
|
||||
<div className="flex flex-col gap-1">
|
||||
<span className="text-xs text-muted" aria-hidden>
|
||||
{t("settings.tokens.expiry")}
|
||||
</span>
|
||||
<SegmentedToggle
|
||||
size="md"
|
||||
ariaLabel={t("settings.tokens.expiry")}
|
||||
className="self-start"
|
||||
options={TOKEN_EXPIRY_DAYS.map((d) => ({
|
||||
value: expiryValue(d),
|
||||
label: t(`settings.tokens.expiryOption.${expiryValue(d)}`),
|
||||
}))}
|
||||
value={expiryValue(expiry)}
|
||||
onChange={(v) => setExpiry(v === "never" ? null : Number(v))}
|
||||
/>
|
||||
{expiry === null && <InlineNotice>{t("settings.tokens.neverWarning")}</InlineNotice>}
|
||||
</div>
|
||||
</form>
|
||||
<ModalFooter>
|
||||
<Button variant="outline" onClick={onClose}>
|
||||
{t("common.cancel")}
|
||||
</Button>
|
||||
<Button
|
||||
variant="primary"
|
||||
type="submit"
|
||||
form={`${nameId}-form`}
|
||||
disabled={!ready || create.isPending}
|
||||
>
|
||||
{t("settings.tokens.create")}
|
||||
</Button>
|
||||
</ModalFooter>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
|
||||
/** A value to copy, shown whole (it wraps), with its Copy button. */
|
||||
function CopyField({
|
||||
label,
|
||||
value,
|
||||
autoFocus = false,
|
||||
}: {
|
||||
label: string;
|
||||
value: string;
|
||||
autoFocus?: boolean;
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
const { copied, copy } = useCopyFeedback();
|
||||
return (
|
||||
<div className="flex flex-col gap-1">
|
||||
<span className="text-xs text-muted">{label}</span>
|
||||
<div className="flex items-start gap-1.5">
|
||||
<code className="min-w-0 flex-1 break-all rounded-lg border border-border bg-card px-2.5 py-1.5 font-mono text-xs">
|
||||
{value}
|
||||
</code>
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
autoFocus={autoFocus}
|
||||
icon={copied ? <Check className="w-4 h-4" /> : <Copy className="w-4 h-4" />}
|
||||
// Two of these sit in one dialog: each is named by the value it copies, and by the word
|
||||
// it shows, so a voice command saying what is on screen finds it.
|
||||
aria-label={t(copied ? "settings.tokens.copiedField" : "settings.tokens.copyField", {
|
||||
field: label,
|
||||
})}
|
||||
onClick={() => void copy(value)}
|
||||
>
|
||||
{copied ? t("settings.tokens.copied") : t("settings.tokens.copy")}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,48 +0,0 @@
|
||||
import { useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import clsx from "clsx";
|
||||
import { ChevronRight } from "lucide-react";
|
||||
import type { AuditRow } from "../lib/api";
|
||||
import { Button } from "./ui/Button";
|
||||
import { RelativeTime } from "./ui/RelativeTime";
|
||||
|
||||
/** A folded run of one job's successful runs (lib/auditGroups): the row shows the newest, this
|
||||
* opens the earlier ones, each with its moment and its result (`detailOf`, the row's own wording). */
|
||||
export default function AuditEarlierRuns({
|
||||
runs,
|
||||
detailOf,
|
||||
}: {
|
||||
runs: AuditRow[];
|
||||
detailOf: (run: AuditRow) => string | null;
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
const [open, setOpen] = useState(false);
|
||||
const earlier = runs.slice(1);
|
||||
return (
|
||||
<div className="mt-1">
|
||||
<Button
|
||||
variant="link-muted"
|
||||
aria-expanded={open}
|
||||
onClick={() => setOpen(!open)}
|
||||
icon={<ChevronRight className={clsx("transition-transform", open && "rotate-90")} />}
|
||||
>
|
||||
{t("audit.earlierRuns", { count: earlier.length })}
|
||||
</Button>
|
||||
{open && (
|
||||
<ul className="mt-1 space-y-0.5 text-2xs text-muted">
|
||||
{earlier.map((run) => (
|
||||
<li key={run.id} className="flex gap-2">
|
||||
<RelativeTime
|
||||
at={run.created_at}
|
||||
seconds
|
||||
empty="—"
|
||||
className="shrink-0 tabular-nums"
|
||||
/>
|
||||
<span className="min-w-0 wrap-break-word">{detailOf(run)}</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,107 +0,0 @@
|
||||
import { useMemo } from "react";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Trash2 } from "lucide-react";
|
||||
import { api } from "../lib/api";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { LS } from "../lib/storage";
|
||||
import { Button } from "./ui/Button";
|
||||
import { Section } from "./ui/form";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import DataTable from "./ui/DataTable";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import { auditColumns } from "./auditColumns";
|
||||
import { notifyActionError } from "../lib/actionErrors";
|
||||
import { compareText } from "../lib/collate";
|
||||
import { foldJobRuns } from "../lib/auditGroups";
|
||||
|
||||
// Admin-only audit trail: an append-only who/what/when log of admin actions + scheduler run
|
||||
// summaries (never per-row). Read-only apart from Clear. Client-side sort/filter/paginate via
|
||||
// DataTable; the API returns the most recent window (capped) — the log stays small by design.
|
||||
export default function AuditLog() {
|
||||
const { t, i18n } = useTranslation();
|
||||
const lang = i18n.language;
|
||||
const qc = useQueryClient();
|
||||
const confirm = useConfirm();
|
||||
const q = useQuery({ queryKey: qk.adminAudit(), queryFn: () => api.adminAudit() });
|
||||
const data = q.data;
|
||||
// Back-to-back successful runs of one job fold into one row (lib/auditGroups).
|
||||
const rows = useMemo(() => foldJobRuns(data?.items ?? []), [data]);
|
||||
|
||||
const actionOptions = useMemo(() => {
|
||||
// By what the reader sees — the named action, in their alphabet — not by the code.
|
||||
return Array.from(new Set(rows.map((r) => r.action)))
|
||||
.map((a) => ({ value: a, label: t(`auditActions.${a}`, a) }))
|
||||
.sort((a, b) => compareText(a.label, b.label, lang));
|
||||
}, [rows, t, lang]);
|
||||
const columns = useMemo(() => auditColumns(t, lang, actionOptions), [t, lang, actionOptions]);
|
||||
|
||||
const clear = useMutation({
|
||||
mutationFn: () => api.clearAudit(),
|
||||
onSuccess: (r) => {
|
||||
qc.invalidateQueries({ queryKey: qk.adminAudit() });
|
||||
notify({ level: "success", message: t("audit.cleared", { count: r.cleared }) });
|
||||
},
|
||||
onError: (e) => notifyActionError(e, t("audit.clearFailed")),
|
||||
});
|
||||
const onClear = async () => {
|
||||
if (
|
||||
await confirm({
|
||||
title: t("audit.clearTitle"),
|
||||
message: t("audit.clearConfirm"),
|
||||
confirmLabel: t("audit.clear"),
|
||||
danger: true,
|
||||
})
|
||||
)
|
||||
clear.mutate();
|
||||
};
|
||||
|
||||
const truncated = !!data && data.total > data.returned;
|
||||
|
||||
return (
|
||||
<div className="p-4 max-w-5xl w-full mx-auto">
|
||||
<Section card title={t("audit.title")}>
|
||||
<p className="text-xs text-muted leading-relaxed mb-3">{t("audit.intro")}</p>
|
||||
{q.isError && !q.data ? (
|
||||
<StateMessage tone="error" onRetry={() => q.refetch()}>
|
||||
{t("common.loadError")}
|
||||
</StateMessage>
|
||||
) : q.isLoading ? (
|
||||
<LoadingState shape="table" />
|
||||
) : rows.length === 0 ? (
|
||||
<p className="text-sm text-muted">{t("audit.empty")}</p>
|
||||
) : (
|
||||
<>
|
||||
<DataTable
|
||||
rows={rows}
|
||||
columns={columns}
|
||||
rowKey={(r) => String(r.id)}
|
||||
persistKey={LS.auditTable}
|
||||
controlsPosition="top"
|
||||
controlsInBand
|
||||
controlsBandClassName="px-4 max-w-5xl mx-auto"
|
||||
controlsLeading={
|
||||
<Button
|
||||
variant="outline-danger"
|
||||
size="xs"
|
||||
onClick={onClear}
|
||||
disabled={clear.isPending}
|
||||
icon={<Trash2 className="w-3.5 h-3.5" />}
|
||||
>
|
||||
{t("audit.clear")}
|
||||
</Button>
|
||||
}
|
||||
emptyText={t("audit.empty")}
|
||||
/>
|
||||
{truncated && (
|
||||
<p className="text-2xs text-muted mt-2">
|
||||
{t("audit.truncated", { returned: data!.returned, total: data!.total })}
|
||||
</p>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</Section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -99,10 +99,10 @@ describe("ChannelPage — a busy toggle keeps the keyboard", () => {
|
||||
});
|
||||
|
||||
// The banner is retried on error (googleusercontent drops a burst); while an attempt fails and the
|
||||
// next waits, a visible <img> shows the browser's broken-image glyph (seen live). It stays
|
||||
// invisible until a source has actually loaded.
|
||||
// next waits, a visible <img> showed the browser's broken-image glyph (seen live). While a retry
|
||||
// waits there is no <img> at all — the bar's own ground — and the retry asks past the cache.
|
||||
describe("ChannelPage — the banner", () => {
|
||||
it("is invisible until it loads, and again while a failed attempt is retried", async () => {
|
||||
it("leaves the bar empty while a failed attempt waits, then asks again past the cache", async () => {
|
||||
vi.useFakeTimers({ shouldAdvanceTime: true });
|
||||
onTestFinished(() => {
|
||||
vi.useRealTimers();
|
||||
@@ -134,16 +134,14 @@ describe("ChannelPage — the banner", () => {
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
const banner = () =>
|
||||
container.querySelector<HTMLImageElement>('img[src^="https://yt3.example/banner"]')!;
|
||||
container.querySelector<HTMLImageElement>('img[src^="https://yt3.example/banner"]');
|
||||
await waitFor(() => expect(banner()).toBeTruthy());
|
||||
expect(banner().classList.contains("opacity-0")).toBe(true);
|
||||
fireEvent.error(banner());
|
||||
expect(banner()!.getAttribute("referrerpolicy")).toBe("no-referrer");
|
||||
fireEvent.error(banner()!);
|
||||
expect(banner()).toBeNull();
|
||||
await act(async () => {
|
||||
await vi.advanceTimersByTimeAsync(600);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
});
|
||||
expect(banner().getAttribute("src")).toContain("?r=1");
|
||||
expect(banner().classList.contains("opacity-0")).toBe(true);
|
||||
fireEvent.load(banner());
|
||||
expect(banner().classList.contains("opacity-0")).toBe(false);
|
||||
expect(banner()!.getAttribute("src")).toContain("?r=1");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -47,6 +47,7 @@ import { IconButton } from "./ui/IconButton";
|
||||
import { IconLink } from "./ui/LinkButton";
|
||||
import { PageHeading } from "./ui/Headings";
|
||||
import { useTags } from "../lib/useTags";
|
||||
import RemoteImage from "./ui/RemoteImage";
|
||||
|
||||
// A dedicated channel page: an "About"-style header (banner, avatar, stats, subscribe) over the
|
||||
// channel's videos (the catalog filtered to this channel). For an un-subscribed channel it
|
||||
@@ -94,17 +95,6 @@ export default function ChannelPage({
|
||||
// is fixed, so unfolded it kept ~40% of a 133% screen for good. Folded, the name rides in the tab row.
|
||||
const foldable = useRef<HTMLDivElement>(null);
|
||||
const folded = useScrollFold(foldable);
|
||||
// The banner comes from Google's image CDN, which refuses any request carrying a Referer from our
|
||||
// origin — every time, not only in a burst (measured 2026-09-26) — so the <img> sends none
|
||||
// (`referrerPolicy`, as ui/Avatar). A load can still drop, and the browser NEGATIVE-CACHES the miss,
|
||||
// so on error we retry a few times with a cache-buster (forcing a fresh request), and only fall
|
||||
// back to the blank bar if it keeps failing (a genuinely dead URL — rare).
|
||||
const MAX_BANNER_RETRIES = 3;
|
||||
const [bannerAttempt, setBannerAttempt] = useState(0);
|
||||
// The banner source that has actually loaded. Until then the <img> is invisible: an attempt that
|
||||
// fails shows the browser's broken-image glyph for as long as the retry waits (seen live).
|
||||
const [bannerLoaded, setBannerLoaded] = useState<string | null>(null);
|
||||
|
||||
// A LIVE query: while THIS user's full-history backfill is running, poll so the stored count (and
|
||||
// the N/S/L breakdown) climb on their own — the immediate backfill fills it in seconds, and the
|
||||
// scheduler tops up any tail on its ~10-minute ticks — then stop polling once the back-catalogue is
|
||||
@@ -113,13 +103,6 @@ export default function ChannelPage({
|
||||
intervalMs: channelDetailPollMs,
|
||||
});
|
||||
const ch = detail.data;
|
||||
// A fresh channel/banner gets the retries back, in the render that changes it: as an effect the
|
||||
// new banner was committed with the previous one's exhausted attempt count still in place.
|
||||
const [bannerFor, setBannerFor] = useState(ch?.banner_url);
|
||||
if (bannerFor !== ch?.banner_url) {
|
||||
setBannerFor(ch?.banner_url);
|
||||
setBannerAttempt(0);
|
||||
}
|
||||
// The poll above is also the ONLY signal that the running backfill has written videos, so it
|
||||
// drives the video grid's refresh too (the "Get full history" mutation below can't: it fires on
|
||||
// the PATCH response, before the background job has stored anything).
|
||||
@@ -348,9 +331,7 @@ export default function ChannelPage({
|
||||
// The channel page owns no scroller of its own (PageShell provides the one app scroller). Its
|
||||
// whole chrome is fixed — wrapped in <PageToolbar> it portals into the shell's fixed band, so the
|
||||
// tabs/toolbar stay put and only the video grid scrolls; the banner/identity fold away meanwhile.
|
||||
const bannerSrc = ch?.banner_url
|
||||
? `${ch.banner_url}=w1707${bannerAttempt ? `?r=${bannerAttempt}` : ""}`
|
||||
: null;
|
||||
const bannerSrc = ch?.banner_url ? `${ch.banner_url}=w1707` : null;
|
||||
return (
|
||||
<>
|
||||
<PageToolbar>
|
||||
@@ -359,44 +340,21 @@ export default function ChannelPage({
|
||||
<div ref={foldable}>
|
||||
{/* Banner — OPTION C: inset, rounded card (not full-bleed) */}
|
||||
<div className="px-4 sm:px-6 pt-3">
|
||||
{bannerSrc && bannerAttempt <= MAX_BANNER_RETRIES ? (
|
||||
// Match YouTube's banner: the stored bannerExternalUrl is the full 16:9 template at a
|
||||
// low default size, so request a crisp wide version (=w1707) and object-cover the
|
||||
// desktop "safe area" — the centre 2560×423 (~6:1) band.
|
||||
<div
|
||||
className="w-full overflow-hidden rounded-2xl bg-surface"
|
||||
style={{ aspectRatio: "2560 / 423", maxHeight: "150px" }}
|
||||
>
|
||||
<img
|
||||
// key forces a fresh <img> per attempt; the ?r= cache-buster on a retry bypasses the
|
||||
// browser's negative cache (googleusercontent accepts an extra query param).
|
||||
key={bannerAttempt}
|
||||
src={bannerSrc}
|
||||
referrerPolicy="no-referrer"
|
||||
alt=""
|
||||
className={`w-full h-full object-cover object-center ${
|
||||
bannerLoaded === bannerSrc ? "" : "opacity-0"
|
||||
}`}
|
||||
onLoad={() => setBannerLoaded(bannerSrc)}
|
||||
onError={() => {
|
||||
if (bannerAttempt <= MAX_BANNER_RETRIES) {
|
||||
window.setTimeout(
|
||||
() => setBannerAttempt((a) => a + 1),
|
||||
500 * (bannerAttempt + 1),
|
||||
);
|
||||
}
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
) : (
|
||||
// No banner (or a dead one): a neutral bar the SAME height as a real banner, so the
|
||||
// overlapping avatar + the Back button have the same room and don't collide (a short bar
|
||||
// let the -mt-8 avatar ride up into the Back button).
|
||||
<div
|
||||
className="w-full rounded-2xl bg-surface"
|
||||
style={{ aspectRatio: "2560 / 423", maxHeight: "150px" }}
|
||||
{/* Match YouTube's banner: the stored bannerExternalUrl is the full 16:9 template at a
|
||||
low default size, so request a crisp wide version (=w1707) and object-cover the
|
||||
desktop "safe area" — the centre 2560×423 (~6:1) band. With no banner (or a dead one)
|
||||
the bar keeps the SAME height, so the overlapping avatar and the Back button keep
|
||||
their room (a short bar let the -mt-8 avatar ride up into the Back button). */}
|
||||
<div
|
||||
className="w-full overflow-hidden rounded-2xl bg-surface"
|
||||
style={{ aspectRatio: "2560 / 423", maxHeight: "150px" }}
|
||||
>
|
||||
<RemoteImage
|
||||
src={bannerSrc}
|
||||
alt=""
|
||||
className="w-full h-full object-cover object-center"
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="px-4 sm:px-6">
|
||||
|
||||
@@ -515,7 +515,9 @@ export default function Channels() {
|
||||
id: "backfill",
|
||||
label: t("channels.backfillEverything"),
|
||||
title: t("channels.backfillEverythingHint"),
|
||||
icon: <History className={`h-4 w-4 ${deepAll.isPending ? "animate-pulse" : ""}`} />,
|
||||
icon: (
|
||||
<History className={`h-4 w-4 ${deepAll.isPending ? "motion-safe:animate-pulse" : ""}`} />
|
||||
),
|
||||
onSelect: () => deepAll.mutate(),
|
||||
disabled: deepAll.isPending,
|
||||
},
|
||||
|
||||
@@ -54,7 +54,10 @@ export default function DownloadButton({
|
||||
>
|
||||
{/* The state is in the glyph: downloaded or queued is the accent, a running one pulses. */}
|
||||
<Download
|
||||
className={clsx((downloaded || inQueue) && "text-accent", inQueue && "animate-pulse")}
|
||||
className={clsx(
|
||||
(downloaded || inQueue) && "text-accent",
|
||||
inQueue && "motion-safe:animate-pulse",
|
||||
)}
|
||||
/>
|
||||
</IconButton>
|
||||
{open && (
|
||||
|
||||
@@ -70,6 +70,7 @@ vi.mock("../lib/notifications", async (importOriginal) => ({
|
||||
import DownloadTray from "./DownloadTray";
|
||||
import { noteQueue, resetQueueWatch, showInTray } from "../lib/downloadQueue";
|
||||
import { pushLayer, removeLayer } from "../lib/layerStack";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
|
||||
beforeEach(() => {
|
||||
resetQueueWatch();
|
||||
@@ -237,6 +238,28 @@ describe("DownloadTray", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("a finish refreshes Files at once, so the notice's Open finds the file listed", async () => {
|
||||
const invalidate = vi.spyOn(QueryClient.prototype, "invalidateQueries");
|
||||
poll([job(1, "running")]);
|
||||
const view = mount();
|
||||
expect(invalidate).not.toHaveBeenCalledWith({ queryKey: qk.files() });
|
||||
poll([job(1, "done")], view);
|
||||
await waitFor(() => expect(h.notify).toHaveBeenCalledTimes(1));
|
||||
expect(invalidate).toHaveBeenCalledWith({ queryKey: qk.files() });
|
||||
expect(invalidate).toHaveBeenCalledWith({ queryKey: qk.fileFolders() });
|
||||
});
|
||||
|
||||
it("an upload the worker converted first is announced as uploaded, not downloaded", async () => {
|
||||
poll([job(1, "running", { source_key: "upload" })]);
|
||||
const view = mount();
|
||||
poll([job(1, "done", { source_key: "upload" })], view);
|
||||
await waitFor(() =>
|
||||
expect(h.notify).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ title: "tray.done.uploadedTitle:Clip 1" }),
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
it("closed, it forgets what finished and stays away until the work changes", () => {
|
||||
poll([job(1, "running"), job(2, "running")]);
|
||||
const view = mount();
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user