release: 0.88.0 (feature/e22-activity-trail)
This commit is contained in:
@@ -28,6 +28,14 @@ ALLOWED_EMAILS=
|
||||
# Admin emails (subset of the above) get the admin role.
|
||||
ADMIN_EMAILS=
|
||||
|
||||
# Logging (E22). LOG_LEVEL is the BOOTSTRAP level — the level a process starts at and the only one
|
||||
# the worker/download child read; the API's running level is the admin Configuration page's
|
||||
# `log_level` knob, which takes effect without a restart. TRACE | DEBUG | INFO | WARNING | ERROR, in
|
||||
# any letter case; any other value stops the boot with an error. LOG_FORMAT is "json" (one object per
|
||||
# line — what a log collector ingests) or "text" (a readable line for a local terminal).
|
||||
LOG_LEVEL=INFO
|
||||
LOG_FORMAT=json
|
||||
|
||||
# Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
|
||||
FRONTEND_ORIGIN=
|
||||
|
||||
|
||||
@@ -7,6 +7,34 @@ edit the release notes and re-run the generator.
|
||||
|
||||
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
|
||||
|
||||
## 0.88.0 — 2026-09-18
|
||||
|
||||
When something goes wrong, you can now turn on detailed logging for yourself for 24 hours — and read your own log on the spot, before any of it is sent anywhere. Siftlode also has a privacy policy and terms at last, in both languages, saying exactly what is recorded and for how long.
|
||||
|
||||
### Features
|
||||
|
||||
- Settings → Diagnostics: choosing Detailed or Full changes nothing until you confirm a box that tells you what will be recorded, that the server keeps it, for how long, and that it switches itself off after 24 hours. Going back to Basic needs no confirmation and takes effect at once.
|
||||
- “My log”, on the same tab, shows what your browser has recorded — newest first, any line openable. Nothing leaves the browser before you have consented.
|
||||
- What you search for and filter by is recorded only at Full, and only under your own consent — never an administrator's. Passwords, tokens, cookies and email addresses are recorded at no level, by anyone.
|
||||
- Your last sign-in is shown in your account panel, and in the administrator's user list.
|
||||
- A privacy policy and terms of service, in Hungarian and English, covering what the server logs and what each diagnostics level adds.
|
||||
- Administrators get a Diagnostics section in Configuration: the log level takes effect without a restart, and stored logs are cleaned up on a schedule (7 days by default).
|
||||
- Every setting with a fixed set of values is now picked from those values instead of typed in, so a wrong one can no longer be saved.
|
||||
|
||||
### Fixes
|
||||
|
||||
- Every message arrived twice if you left a conversation and came back quickly: two chat connections stayed open where there should have been one.
|
||||
- Settings was unusable on a phone — the tab rail took the width and left almost nothing for the panel beside it.
|
||||
- Dates and numbers follow the language you picked in Siftlode, not the one your computer happens to be set to. An English Siftlode on a Hungarian machine no longer shows Hungarian dates.
|
||||
- An administrator setting shown as a dropdown opened on its first option instead of the one actually in force.
|
||||
- In Hungarian, the audit log and the new log settings had names you could not tell apart.
|
||||
- The scheduled feed poll now covers only channels somebody actually follows — on this instance 582 of 979 polled channels had no follower at all. The trade-off is deliberate: new uploads from a channel nobody follows no longer appear in the shared Library.
|
||||
|
||||
### Internal
|
||||
|
||||
- One logging mechanism replaced four. The API, the worker, the downloader and the migration tool now write the same structured line; secrets are scrubbed where the line is written rather than at each of ~125 places that write one; and every request carries an id that joins a line from your browser to the server line of the same request. The browser's log uses the same shape and the same redaction rules, checked from both sides against one shared set of cases.
|
||||
- Two adversarial review rounds over the epic: 56 findings raised, 32 survived three independent attempts to refute each, all fixed. Every regression test was proven to fail with its own fix removed — which is how a test that passed only on leftover state from an earlier test was caught, and how a focus bug that jsdom could not reproduce at all was settled in a real browser.
|
||||
|
||||
## 0.87.1 — 2026-09-16
|
||||
|
||||
A YouTube video that YouTube will only show to a signed-in viewer can now be downloaded: paste your YouTube cookies in Settings the way the failure message already told you to — until now that list had no YouTube in it.
|
||||
|
||||
@@ -58,7 +58,7 @@ how it gets built. Concretely:
|
||||
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
|
||||
`downloads/`, `youtube/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
|
||||
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`,
|
||||
`security`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
|
||||
`security`, `logs`, `logsink`, `diagnostics`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
|
||||
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → module
|
||||
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
|
||||
its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
|
||||
@@ -95,6 +95,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
|
||||
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
|
||||
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
|
||||
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
|
||||
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
|
||||
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
|
||||
|
||||
## House rules that bite (learned from real bugs)
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
[alembic]
|
||||
script_location = alembic
|
||||
prepend_sys_path = .
|
||||
# Split the path list the way the OS does (alembic's legacy space/comma/colon split is deprecated).
|
||||
path_separator = os
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
from logging.config import fileConfig
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
|
||||
from app.config import settings
|
||||
from app.db import Base
|
||||
from app.logs import configure_logging
|
||||
import app.models # noqa: F401 (import so models register on Base.metadata)
|
||||
|
||||
config = context.config
|
||||
config.set_main_option("sqlalchemy.url", settings.database_url)
|
||||
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
# Migrations run at container start (entrypoint.sh), so their output is part of the API's log
|
||||
# stream — one format with everything else, not alembic.ini's own fileConfig shape.
|
||||
configure_logging("alembic")
|
||||
|
||||
target_metadata = Base.metadata
|
||||
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
"""add users.last_login_at — when the account last signed in (E22 S1)
|
||||
|
||||
Revision ID: 0085_users_last_login_at
|
||||
Revises: 0084_dl_job_error_detail
|
||||
Create Date: 2026-09-16
|
||||
|
||||
Stamped by the one sign-in path (`routes/auth._establish_session`), which the Google callback and
|
||||
password login share. No backfill is possible: the signal exists only at sign-in, so existing rows
|
||||
read NULL until each account signs in again, and the admin UI says so rather than inventing a date.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0085_users_last_login_at"
|
||||
down_revision: str | None = "0084_dl_job_error_detail"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("users", sa.Column("last_login_at", sa.DateTime(timezone=True), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("users", "last_login_at")
|
||||
@@ -0,0 +1,48 @@
|
||||
"""add log_events — stored log events for diagnostics (E22 S3)
|
||||
|
||||
Revision ID: 0086_log_events
|
||||
Revises: 0085_users_last_login_at
|
||||
Create Date: 2026-09-17
|
||||
|
||||
Written by app/logsink.py's background writer (WARNING and above, request lines), pruned by the log_gc
|
||||
job after `log_retention_days`. Sized against prod on 2026-09-16: ~2k rows on a normal day, ~14k on a
|
||||
YouTube-RSS-outage day before flood control — a few MB over the 7-day window. `user_id` carries no
|
||||
foreign key on purpose (see models.LogEvent).
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision: str = "0086_log_events"
|
||||
down_revision: str | None = "0085_users_last_login_at"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"log_events",
|
||||
sa.Column("id", sa.BigInteger(), primary_key=True),
|
||||
sa.Column("ts", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("level", sa.String(10), nullable=False),
|
||||
sa.Column("sev", sa.SmallInteger(), nullable=False),
|
||||
sa.Column("source", sa.String(16), nullable=False),
|
||||
sa.Column("logger", sa.String(128), nullable=False),
|
||||
sa.Column("event", sa.String(64), nullable=True),
|
||||
sa.Column("msg", sa.Text(), nullable=False),
|
||||
sa.Column("attrs", postgresql.JSONB(), nullable=True),
|
||||
sa.Column("exc", sa.Text(), nullable=True),
|
||||
sa.Column("request_id", sa.String(64), nullable=True),
|
||||
sa.Column("user_id", sa.Integer(), nullable=True),
|
||||
sa.Column("debug_session_id", sa.String(64), nullable=True),
|
||||
sa.Column("job_id", sa.Integer(), nullable=True),
|
||||
sa.Column("version", sa.String(64), nullable=True),
|
||||
)
|
||||
op.create_index("ix_log_events_ts", "log_events", ["ts"])
|
||||
op.create_index("ix_log_events_request_id", "log_events", ["request_id"])
|
||||
op.create_index("ix_log_events_user_ts", "log_events", ["user_id", "ts"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("log_events")
|
||||
@@ -0,0 +1,31 @@
|
||||
"""add users.diagnostics_level + diagnostics_until — a user's consented diagnostics level (E22 S5)
|
||||
|
||||
Revision ID: 0087_users_diagnostics_level
|
||||
Revises: 0086_log_events
|
||||
Create Date: 2026-09-17
|
||||
|
||||
Both NULL = the base level (Alap). A raised level is ALWAYS paired with its expiry, computed by the
|
||||
server at consent time (`app/diagnostics.py`), so no client can ask for a level that never ends.
|
||||
Columns rather than `users.preferences`: that JSON is written wholesale by the client, and a consent
|
||||
with a lifetime must not be something a client can write.
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0087_users_diagnostics_level"
|
||||
down_revision: str | None = "0086_log_events"
|
||||
branch_labels: str | None = None
|
||||
depends_on: str | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("users", sa.Column("diagnostics_level", sa.String(8), nullable=True))
|
||||
op.add_column(
|
||||
"users", sa.Column("diagnostics_until", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("users", "diagnostics_until")
|
||||
op.drop_column("users", "diagnostics_level")
|
||||
@@ -20,9 +20,10 @@ Three layers, by *who* runs the code — not by feature:
|
||||
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline.
|
||||
|
||||
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
|
||||
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, … are single-file primitives many
|
||||
layers import. They depend on nothing above them (a core module must never import a `routes/…`
|
||||
module).
|
||||
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, `logs` (the one logging
|
||||
configuration), `logsink` (its `log_events` table) and `diagnostics` (a user's consented log
|
||||
level), … are single-file primitives many layers import. They depend on nothing above them (a core
|
||||
module must never import a `routes/…` module).
|
||||
|
||||
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
|
||||
*second* caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down
|
||||
|
||||
+16
-1
@@ -6,12 +6,14 @@ from sqlalchemy import delete, func, select
|
||||
from sqlalchemy.orm import Session
|
||||
from starlette.requests import HTTPConnection
|
||||
|
||||
from app import diagnostics
|
||||
from app import email as email_mod
|
||||
from app import logs
|
||||
from app import mailtext
|
||||
from app import sysconfig
|
||||
from app.config import settings
|
||||
from app.db import get_db
|
||||
from app.models import Invite, User
|
||||
from app.models import Invite, LogEvent, User
|
||||
from app.security import decrypt_optional
|
||||
from app.utils import canonical_email
|
||||
|
||||
@@ -245,6 +247,12 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
|
||||
)
|
||||
# Raw delete so Postgres applies ON DELETE CASCADE / SET NULL on every dependent table.
|
||||
db.execute(delete(User).where(User.id == user_id))
|
||||
# Stored log events name the account by id with no foreign key to cascade from (models.LogEvent),
|
||||
# so they go explicitly — they are request history, i.e. personal data. Forgotten FIRST: this very
|
||||
# request's access line, a queued row, a later background task would otherwise put rows back
|
||||
# under the id after this DELETE (logs.forget_user).
|
||||
logs.forget_user(user_id)
|
||||
db.execute(delete(LogEvent).where(LogEvent.user_id == user_id))
|
||||
# Every canonically-equal invite goes, not just the literal spelling: is_allowed matches on the
|
||||
# canonical form, so leaving a sibling row behind would keep granting access to a mailbox whose
|
||||
# account was just erased — and prod really does carry approved pairs for one mailbox.
|
||||
@@ -348,6 +356,13 @@ def _session_user(request: Request, db: Session, user_id: int | None, is_default
|
||||
if default_id and default_id not in accounts:
|
||||
accounts.append(default_id)
|
||||
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
|
||||
# Attribute this request's log lines to the account (E22). Here, not in the request middleware:
|
||||
# this is the one place a session becomes an account, and the middleware runs outside both the
|
||||
# session middleware and this task's context. `request.state` is shared across middleware layers
|
||||
# (it lives on the ASGI scope), so the access-log line can read it back. The user's consented
|
||||
# diagnostics level rides along: their requests are logged — and stored — in that detail (S5).
|
||||
logs.set_user(user.id, diagnostics.effective_level(user), user.diagnostics_until)
|
||||
request.state.user_id = user.id
|
||||
return user
|
||||
|
||||
|
||||
|
||||
+61
-5
@@ -1,6 +1,7 @@
|
||||
from pathlib import Path
|
||||
from typing import Literal, get_args, get_origin
|
||||
|
||||
from pydantic import model_validator
|
||||
from pydantic import ValidationInfo, field_validator, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
# Shipped placeholder; production must override it. Used by the startup guard below.
|
||||
@@ -20,6 +21,33 @@ def _version_from_file() -> str:
|
||||
return "dev"
|
||||
|
||||
|
||||
# Closed value sets. A setting typed with one of these is a CHOICE: pydantic rejects anything else at
|
||||
# boot, and sysconfig derives the admin picker's options from the same Literal — so the list lives in
|
||||
# exactly one place and a consumer never has to guess what a typo should fall back to.
|
||||
SearchSource = Literal["scrape", "api"]
|
||||
DownloadLayout = Literal["plex", "flat"]
|
||||
LogLevel = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"]
|
||||
LogFormat = Literal["json", "text"]
|
||||
TranscodeHw = Literal["auto", "on", "off"]
|
||||
# libx264's presets, fastest first (`placebo` left out on purpose: no measurable gain for its cost).
|
||||
TranscodePreset = Literal[
|
||||
"ultrafast", "superfast", "veryfast", "faster", "fast", "medium", "slow", "slower", "veryslow"
|
||||
]
|
||||
|
||||
|
||||
def choices_of(annotation: object) -> tuple[str, ...]:
|
||||
"""The allowed values of a choice-typed setting, or () for any other type."""
|
||||
return tuple(get_args(annotation)) if get_origin(annotation) is Literal else ()
|
||||
|
||||
|
||||
def canonical_choice(options: tuple[str, ...], raw: object) -> str | None:
|
||||
"""The option `raw` names, in its canonical spelling (any letter case, surrounding space
|
||||
ignored), or None. The ONE matching rule for a choice value from the env and from the admin
|
||||
API, so a spelling one accepts the other cannot refuse."""
|
||||
folded = str(raw).strip().casefold()
|
||||
return next((option for option in options if option.casefold() == folded), None)
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||
|
||||
@@ -85,7 +113,7 @@ class Settings(BaseSettings):
|
||||
search_daily_limit_per_user: int = 70
|
||||
# Source for live YouTube search: "scrape" (InnerTube, no API quota) or "api" (search.list,
|
||||
# 100 units/page). Scrape is the default; flip to "api" if scraping is blocked/breaks.
|
||||
search_source: str = "scrape"
|
||||
search_source: SearchSource = "scrape"
|
||||
# Recent-first backfill: how far back to fetch on the first pass per channel.
|
||||
backfill_recent_max_videos: int = 100
|
||||
backfill_recent_max_days: int = 365
|
||||
@@ -224,10 +252,27 @@ class Settings(BaseSettings):
|
||||
audit_retention_days: int = 180
|
||||
audit_gc_minutes: int = 1440
|
||||
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
|
||||
download_layout: str = "plex"
|
||||
download_layout: DownloadLayout = "plex"
|
||||
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
|
||||
sponsorblock_default: bool = False
|
||||
|
||||
# --- Diagnostics / logging (E22) ---
|
||||
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
|
||||
# startup and on change without a restart. This is what a process logs at before it can reach
|
||||
# the database — and the only knob the worker/runner children read, since they are short-lived.
|
||||
# TRACE (below DEBUG) is the "Teljes" diagnostics level; see app/logs.py.
|
||||
log_level: LogLevel = "INFO"
|
||||
# "json" (one object per line — what Loki and the log_events sink ingest) or "text" (a readable
|
||||
# line for a local terminal). Redaction applies to both.
|
||||
log_format: LogFormat = "json"
|
||||
# Also write events to the `log_events` table (app/logsink.py): WARNING and above, the request
|
||||
# lines, and — once they exist — client batches. Off only where no table is wanted (the tests).
|
||||
log_db_sink: bool = True
|
||||
# Days a `log_events` row is kept (the privacy policy promises this), and how often the log_gc job
|
||||
# enforces it — hourly, so a row never outlives the promise by more than an hour.
|
||||
log_retention_days: int = 7
|
||||
log_gc_minutes: int = 60
|
||||
|
||||
# --- Plex integration (optional) ---
|
||||
# The Plex module plays the LOCAL physical media file directly (our own ffmpeg/direct-serve);
|
||||
# Plex is used only for metadata + optional watch-state sync. Requires the Plex server to be
|
||||
@@ -269,7 +314,7 @@ class Settings(BaseSettings):
|
||||
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
|
||||
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
|
||||
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
|
||||
plex_transcode_hw: str = "auto"
|
||||
plex_transcode_hw: TranscodeHw = "auto"
|
||||
# Software-fallback libx264 tuning (IGNORED on the VAAPI path). `preset` trades CPU for size; `crf`
|
||||
# trades quality for size (lower = better + bigger). `max_height` caps the SOFTWARE transcode's output
|
||||
# height (0 = keep source) so a CPU re-encode of a tall source stays affordable — the GPU path always
|
||||
@@ -279,7 +324,7 @@ class Settings(BaseSettings):
|
||||
# call, not a fix. Only affects files that go through remux/transcode — a direct-played file is
|
||||
# served untouched, but those are browser-native stereo anyway, not the 5.1 mixes this is for.
|
||||
plex_audio_loudnorm: bool = False
|
||||
plex_transcode_preset: str = "veryfast"
|
||||
plex_transcode_preset: TranscodePreset = "veryfast"
|
||||
plex_transcode_crf: int = 20
|
||||
plex_transcode_max_height: int = 1080
|
||||
# Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
|
||||
@@ -328,6 +373,17 @@ class Settings(BaseSettings):
|
||||
https OAUTH_REDIRECT_URL as the signal for 'real deployment' vs local dev."""
|
||||
return self.oauth_redirect_url.lower().startswith("https://")
|
||||
|
||||
@field_validator("*", mode="before")
|
||||
@classmethod
|
||||
def _canonical_choice(cls, value: object, info: ValidationInfo) -> object:
|
||||
"""A choice setting accepts its value in any letter case (`LOG_LEVEL=debug`) and stores the
|
||||
canonical spelling; an unknown value still fails the boot with pydantic's own message."""
|
||||
field = cls.model_fields.get(info.field_name or "")
|
||||
options = choices_of(field.annotation) if field else ()
|
||||
if options and isinstance(value, str):
|
||||
return canonical_choice(options, value) or value
|
||||
return value
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _enforce_production_secrets(self) -> "Settings":
|
||||
"""Fail fast rather than boot a public instance with the shipped placeholder
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
"""A user's diagnostics level (E22 S5): what they consented to, and for how long.
|
||||
|
||||
Three levels, named in the UI Alap / Részletes / Teljes:
|
||||
|
||||
* INFO — the base, for everyone, no consent. The browser keeps its log in memory; nothing extra
|
||||
is sent or stored.
|
||||
* DEBUG — consented. The server logs this user's requests in detail and stores every line of
|
||||
them, and the browser streams its own log to `POST /api/diagnostics/events`.
|
||||
* TRACE — consented. As DEBUG, plus what the user typed into a search.
|
||||
|
||||
A raised level always lapses after `ELEVATION_HOURS`: the expiry is computed HERE at consent time,
|
||||
never taken from the client, and an expired level reads as INFO without anything having to clear it.
|
||||
The shared demo account cannot raise its level — its log would mix every visitor's session.
|
||||
|
||||
Core layer: imports `models` and `sysconfig`.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Literal, get_args
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import sysconfig
|
||||
from app.models import User
|
||||
|
||||
DiagnosticsLevel = Literal["INFO", "DEBUG", "TRACE"]
|
||||
LEVELS: tuple[str, ...] = get_args(DiagnosticsLevel)
|
||||
BASE: DiagnosticsLevel = "INFO"
|
||||
ELEVATION_HOURS = 24
|
||||
|
||||
|
||||
def effective_level(user: User, now: datetime | None = None) -> DiagnosticsLevel:
|
||||
"""The level in force for `user` right now: the consented one until it expires, else BASE."""
|
||||
level, until = user.diagnostics_level, user.diagnostics_until
|
||||
if level not in LEVELS or level == BASE or until is None:
|
||||
return BASE
|
||||
if until <= (now or datetime.now(timezone.utc)):
|
||||
return BASE
|
||||
return level # type: ignore[return-value] — narrowed by the LEVELS check above
|
||||
|
||||
|
||||
def state(db: Session, user: User) -> dict:
|
||||
"""The `/api/me` view of it: the level in force, when a raised one lapses, and how long stored
|
||||
log lines are kept — the consent text states that number, so it must be the instance's own."""
|
||||
level = effective_level(user)
|
||||
return {
|
||||
"level": level,
|
||||
"until": user.diagnostics_until.isoformat() if level != BASE else None,
|
||||
"retention_days": sysconfig.get_int(db, "log_retention_days"),
|
||||
}
|
||||
|
||||
|
||||
def set_level(user: User, level: DiagnosticsLevel) -> None:
|
||||
"""Record a consent (or its withdrawal). The caller commits. Raising the level again restarts
|
||||
the clock — each consent is for the next ELEVATION_HOURS, not an extension of an older one."""
|
||||
if level == BASE:
|
||||
user.diagnostics_level = None
|
||||
user.diagnostics_until = None
|
||||
return
|
||||
user.diagnostics_level = level
|
||||
user.diagnostics_until = datetime.now(timezone.utc) + timedelta(hours=ELEVATION_HOURS)
|
||||
@@ -17,6 +17,8 @@ import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from app import logs
|
||||
from app import logsink
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.downloads import edit as editmod
|
||||
@@ -52,12 +54,16 @@ def _deprioritize() -> None:
|
||||
|
||||
|
||||
def main() -> None:
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logs.configure_logging("runner")
|
||||
_deprioritize()
|
||||
if len(sys.argv) != 2:
|
||||
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
job_id = int(sys.argv[1])
|
||||
# This whole process exists to run ONE job — tag every line it prints with it, so the child's
|
||||
# output joins up with the worker's and the API's view of the same download.
|
||||
logs.set_job(job_id)
|
||||
logsink.start() # flushed at exit (atexit) — this process has no shutdown hook of its own
|
||||
|
||||
with SessionLocal() as db:
|
||||
job = db.get(DownloadJob, job_id)
|
||||
|
||||
@@ -0,0 +1,685 @@
|
||||
"""Structured logging for every Siftlode process — the ONE place logging is configured.
|
||||
|
||||
Before this module a single deployment printed FOUR different log formats: uvicorn's
|
||||
`log_config.json` for the API, a hand-rolled StreamHandler in `main.py` for everything that did not
|
||||
start under uvicorn (tests, scripts), `logging.basicConfig()` in the worker and in the download
|
||||
child (no timestamp at all), and alembic's `fileConfig`. None of them carried a correlation id, and
|
||||
uvicorn's access log printed the query string of every request (`?account=1`, `?source=feed`) — so
|
||||
the logs were both unjoinable and leaky. `configure_logging()` replaces all four.
|
||||
|
||||
Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
|
||||
ts · level · sev · source · logger · event · msg · attrs
|
||||
plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
|
||||
`job_id`) and the app version. That shape is what E22.6's admin console filters on and what a Gitea
|
||||
issue attachment will carry, so it is deliberately flat and stable.
|
||||
|
||||
Call sites stay ordinary stdlib logging:
|
||||
|
||||
log = logging.getLogger("siftlode.downloads")
|
||||
log.info("job finished", extra=logs.event("download.job.done", job_id=7, bytes=n))
|
||||
|
||||
`event()` is optional — a bare `log.info("…")` still comes out well-formed — but the dotted code is
|
||||
what makes a line queryable later, so add one wherever the line matters.
|
||||
|
||||
REDACTION IS APPLIED AT THE HANDLER, not at the call site: every message, every attr value and
|
||||
every traceback passes `redact()` on the way out. That is the whole point — a secret cannot be
|
||||
leaked by forgetting to redact it at one of ~125 call sites, and a new call site is safe by
|
||||
default. Never logged, at any level: passwords, tokens (session, reset/verify, setup, share,
|
||||
OAuth), cookie blobs, secret config values, and email addresses.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import contextvars
|
||||
import json
|
||||
import logging
|
||||
import logging.config
|
||||
import re
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qsl
|
||||
|
||||
from app.config import settings
|
||||
from app.ratelimit import RateLimiter, client_key
|
||||
|
||||
# One level below DEBUG for the "Teljes / TRACE" diagnostics level (search strings, redacted request
|
||||
# bodies). Registered on import so `logging.getLogger(...).log(logs.TRACE, …)` and a `LOG_LEVEL=TRACE`
|
||||
# both work in every process, including one that never calls configure_logging (a test).
|
||||
TRACE = 5
|
||||
logging.addLevelName(TRACE, "TRACE")
|
||||
|
||||
# OpenTelemetry severity numbers for our level names. Kept beside the level because the number is
|
||||
# what a log backend sorts and filters on; the name is for humans reading a terminal.
|
||||
SEVERITY: dict[str, int] = {
|
||||
"TRACE": 1,
|
||||
"DEBUG": 5,
|
||||
"INFO": 9,
|
||||
"WARNING": 13,
|
||||
"ERROR": 17,
|
||||
"CRITICAL": 21,
|
||||
}
|
||||
|
||||
# Which process this is. Set by configure_logging; every line carries it so one Loki/DB query can
|
||||
# separate the API's view of a job from the worker's.
|
||||
_source = "api"
|
||||
|
||||
|
||||
# --- correlation context -------------------------------------------------------------------
|
||||
# contextvars, like quota's actor attribution (app/quota.py): set at an entry point, read by the
|
||||
# formatter, never threaded through call signatures. A child task copies the context at creation,
|
||||
# so anything set before `await call_next(...)` is visible to the whole request.
|
||||
_request_id: contextvars.ContextVar[str | None] = contextvars.ContextVar(
|
||||
"log_request_id", default=None
|
||||
)
|
||||
_user_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_user_id", default=None)
|
||||
_debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
|
||||
"log_debug_session", default=None
|
||||
)
|
||||
_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
|
||||
|
||||
_CONTEXT_VARS = {
|
||||
"request_id": _request_id,
|
||||
"user_id": _user_id,
|
||||
"debug_session_id": _debug_session,
|
||||
"job_id": _job_id,
|
||||
}
|
||||
|
||||
|
||||
class _RequestScope:
|
||||
"""What the auth dependency learns about a request, for every line logged after it.
|
||||
|
||||
A MUTABLE object behind a contextvar, not plain contextvars, because of how a request actually
|
||||
runs: FastAPI resolves a sync dependency (`auth.current_user`) in a threadpool on a COPY of the
|
||||
context, so a `ContextVar.set()` there is gone by the time the handler runs — measured: that is
|
||||
how the account never reached a handler's own lines. `log_requests` installs one scope per
|
||||
request; every copied context shares the same object, so what the dependency writes is seen by
|
||||
the handler and by the request line itself."""
|
||||
|
||||
__slots__ = ("user_id", "level")
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.user_id: int | None = None
|
||||
# The user's consented diagnostics level as a logging number; None at the base level.
|
||||
self.level: int | None = None
|
||||
|
||||
|
||||
_scope: contextvars.ContextVar[_RequestScope | None] = contextvars.ContextVar(
|
||||
"log_request_scope", default=None
|
||||
)
|
||||
|
||||
|
||||
def set_user(user_id: int | None, level: str = "INFO", until: datetime | None = None) -> None:
|
||||
"""Attribute the rest of this request's log lines to `user_id`, and log them at the user's
|
||||
consented diagnostics `level` until `until` (E22 S5; "INFO" = no elevation). Called from the
|
||||
auth dependency — the one place the session is actually resolved into an account — rather than
|
||||
from the request middleware, which runs OUTSIDE the session middleware."""
|
||||
_user_id.set(user_id)
|
||||
scope = _scope.get()
|
||||
if scope is None: # outside log_requests (a script, a unit test): this context only
|
||||
scope = _RequestScope()
|
||||
_scope.set(scope)
|
||||
elevated = level != "INFO" and until is not None and until > datetime.now(timezone.utc)
|
||||
scope.user_id = user_id
|
||||
scope.level = resolve_level(level) if elevated else None
|
||||
if user_id is not None:
|
||||
_track_elevation(user_id, scope.level, until)
|
||||
|
||||
|
||||
def set_job(job_id: int | None) -> None:
|
||||
"""Attribute this process's log lines to a download job — for the download CHILD, whose whole
|
||||
life is one job (the worker uses `bind()` per job instead, since it juggles several)."""
|
||||
_job_id.set(job_id)
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def bind(**fields: Any):
|
||||
"""Bind correlation fields for the duration of a block (a worker job, a scheduler run):
|
||||
|
||||
with logs.bind(job_id=job.id):
|
||||
...
|
||||
|
||||
Unknown field names are ignored rather than raising — a caller binding a field this module does
|
||||
not know about is a typo, not a reason to take down a download."""
|
||||
tokens = [(_CONTEXT_VARS[k].set(v)) for k, v in fields.items() if k in _CONTEXT_VARS]
|
||||
keys = [k for k in fields if k in _CONTEXT_VARS]
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
for key, token in zip(keys, tokens):
|
||||
_CONTEXT_VARS[key].reset(token)
|
||||
|
||||
|
||||
def event(code: str, **attrs: Any) -> dict[str, Any]:
|
||||
"""`extra=` payload for a log call: a stable dotted event code plus structured attributes.
|
||||
|
||||
Returns a plain dict so it composes with stdlib logging and costs nothing when the level is off
|
||||
(the dict is built either way; keep the attrs cheap — no DB reads inside the call)."""
|
||||
return {"event": code, "attrs": attrs}
|
||||
|
||||
|
||||
# --- redaction -----------------------------------------------------------------------------
|
||||
# Substring matching, not exact keys: the same secret travels as `password`, `current_password`,
|
||||
# `new_password`, `smtp_password`, `plex_server_token`, `access_token`, … Matching on the PART is
|
||||
# what makes a key we have never seen safe by default.
|
||||
_SECRET_KEY_PARTS = (
|
||||
"password",
|
||||
"passphrase",
|
||||
"token",
|
||||
"secret",
|
||||
"cookie",
|
||||
"authorization",
|
||||
"auth_header",
|
||||
"ciphertext",
|
||||
"private_key",
|
||||
"api_key",
|
||||
"apikey",
|
||||
"credential",
|
||||
"session_key",
|
||||
)
|
||||
|
||||
# Query/fragment parameters that carry a credential in a URL we log (OAuth `code`/`state`, our own
|
||||
# `?token=` setup and share links, `#verify=` / `#reset=` fragments).
|
||||
_SECRET_PARAM_NAMES = (
|
||||
"access_token",
|
||||
"refresh_token",
|
||||
"id_token",
|
||||
"token",
|
||||
"code",
|
||||
"state",
|
||||
"key",
|
||||
"secret",
|
||||
"password",
|
||||
"pwd",
|
||||
"sig",
|
||||
"signature",
|
||||
"verify",
|
||||
"reset",
|
||||
"setup",
|
||||
)
|
||||
_SECRET_PARAM_RE = re.compile(
|
||||
r"\b((?:" + "|".join(_SECRET_PARAM_NAMES) + r")=)([^&\s\"']+)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Deliberately loose: over-matching redacts a non-address that looks like one (harmless), while
|
||||
# under-matching puts a real address in a log we attach to an issue (not harmless).
|
||||
_EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
|
||||
|
||||
REDACTED = "[redacted]"
|
||||
# `extra={PERSIST: False}` keeps a line out of the `log_events` table while stdout still gets it —
|
||||
# how `log_requests` caps anonymous traffic per client (app/logsink.py reads it).
|
||||
PERSIST = "persist"
|
||||
REDACTED_EMAIL = "[email]"
|
||||
|
||||
|
||||
def redact_text(text: str) -> str:
|
||||
"""Scrub a free-text string: credential-bearing URL params and email addresses."""
|
||||
scrubbed = _SECRET_PARAM_RE.sub(lambda m: f"{m.group(1)}{REDACTED}", text)
|
||||
return _EMAIL_RE.sub(REDACTED_EMAIL, scrubbed)
|
||||
|
||||
|
||||
def redact_query(query: str) -> str:
|
||||
"""Scrub a raw query string. DECODED first: the text rules match what was typed, and a browser
|
||||
sends it percent-encoded — `someone%40example.com` is an address `_EMAIL_RE` cannot see. Each
|
||||
parameter also goes through the key rules, so `?new_password=` goes as `redact` would drop it."""
|
||||
pairs = parse_qsl(query, keep_blank_values=True)
|
||||
return "&".join(f"{key}={_redact_param(key, value)}" for key, value in pairs)
|
||||
|
||||
|
||||
def _redact_param(key: str, value: str) -> str:
|
||||
"""One decoded `key=value` pair. The parameter-NAME rule is applied to the key here rather than
|
||||
left to `_SECRET_PARAM_RE` over the rebuilt pair: that pattern was written for the raw,
|
||||
percent-encoded query, where a value holds no literal space, `&` or quote, and its value class
|
||||
stops at the first one. On decoded text `?state=a%20b` would keep its tail and `?code=%20abc`
|
||||
would not match at all — the whole credential straight into the line."""
|
||||
if key.lower() in _SECRET_PARAM_NAMES:
|
||||
return REDACTED
|
||||
return str(redact(value, key=key))
|
||||
|
||||
|
||||
def redact(value: Any, *, key: str = "") -> Any:
|
||||
"""Recursively scrub a structure destined for the log.
|
||||
|
||||
A value whose KEY names a secret is dropped whole (we never log a prefix of a token — a prefix
|
||||
is still a leak and still useless). Everything else is scrubbed as text. Depth is bounded by
|
||||
the structures we actually log (attrs dicts), and non-JSON types fall through to `str`."""
|
||||
lowered = key.lower()
|
||||
if key and any(part in lowered for part in _SECRET_KEY_PARTS):
|
||||
return REDACTED
|
||||
if isinstance(value, dict):
|
||||
return {k: redact(v, key=str(k)) for k, v in value.items()}
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [redact(v, key=key) for v in value]
|
||||
if isinstance(value, str):
|
||||
return redact_text(value)
|
||||
return value
|
||||
|
||||
|
||||
# --- formatters ----------------------------------------------------------------------------
|
||||
# Attributes every LogRecord carries; anything else a caller passed via `extra=` is a structured
|
||||
# attribute and is merged into `attrs` (so `extra={"job_id": 7}` works as well as `event(...)`).
|
||||
_RECORD_STANDARD = frozenset(
|
||||
vars(logging.LogRecord("", 0, "", 0, "", None, None)).keys()
|
||||
| {"message", "asctime", "taskName", "event", "attrs", PERSIST, "elevated"}
|
||||
)
|
||||
|
||||
|
||||
class ContextFilter(logging.Filter):
|
||||
"""Copies the correlation contextvars onto the record. A Filter (not the formatter) so the
|
||||
values are captured at emit time, before a QueueHandler can move the record to another
|
||||
thread — the DB sink in S3 depends on that."""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
for name, var in _CONTEXT_VARS.items():
|
||||
value = var.get()
|
||||
if value is not None and not hasattr(record, name):
|
||||
setattr(record, name, value)
|
||||
scope = _scope.get()
|
||||
if scope is not None and scope.user_id is not None and not hasattr(record, "user_id"):
|
||||
record.user_id = scope.user_id
|
||||
return True
|
||||
|
||||
|
||||
class LevelGate(logging.Filter):
|
||||
"""The configured level, enforced at the HANDLER, with one exception: the lines of a request
|
||||
whose user raised their diagnostics level (E22 S5).
|
||||
|
||||
The root logger's level cannot do this alone: it is one number for the whole process, and the
|
||||
point is to log ONE user's requests in detail while everyone else's stay at the base level. So
|
||||
the root is lowered to the lowest level any active elevation needs (`_track_elevation`), and
|
||||
this gate drops what only an elevation let through unless the line belongs to that elevated
|
||||
request. It also marks such a line `elevated`, which is what makes the `log_events` sink keep
|
||||
every line of that request, not just its warnings."""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
scope = _scope.get()
|
||||
if scope is not None and scope.level is not None:
|
||||
record.elevated = True
|
||||
return record.levelno >= min(scope.level, _base_level)
|
||||
return record.levelno >= _base_level
|
||||
|
||||
|
||||
def _record_attrs(record: logging.LogRecord) -> dict[str, Any]:
|
||||
extra = {k: v for k, v in vars(record).items() if k not in _RECORD_STANDARD}
|
||||
attrs = dict(getattr(record, "attrs", None) or {})
|
||||
attrs.update({k: v for k, v in extra.items() if k not in _CONTEXT_VARS})
|
||||
return attrs
|
||||
|
||||
|
||||
def _safe_message(record: logging.LogRecord) -> str:
|
||||
"""`record.getMessage()` raises if the args don't match the format string. A broken log call
|
||||
must not take down the caller, so the failure is reported IN the line instead."""
|
||||
try:
|
||||
return record.getMessage()
|
||||
except Exception as exc: # noqa: BLE001 — any formatting error, reported not raised
|
||||
return f"{record.msg!r} (log formatting failed: {exc})"
|
||||
|
||||
|
||||
def record_fields(record: logging.LogRecord) -> dict[str, Any]:
|
||||
"""The event as structured fields, REDACTED — the one definition both the stdout line and the
|
||||
`log_events` row are built from, so the two sinks can never disagree about what is safe to keep.
|
||||
Absent optional fields are left out; `ts` is an aware datetime."""
|
||||
fields: dict[str, Any] = {
|
||||
"ts": datetime.fromtimestamp(record.created, timezone.utc),
|
||||
"level": record.levelname,
|
||||
"sev": SEVERITY.get(record.levelname, 9),
|
||||
"source": _source,
|
||||
"logger": record.name,
|
||||
"msg": redact_text(_safe_message(record)),
|
||||
}
|
||||
code = getattr(record, "event", None)
|
||||
if code:
|
||||
fields["event"] = code
|
||||
for name in _CONTEXT_VARS:
|
||||
value = getattr(record, name, None)
|
||||
if value is not None:
|
||||
fields[name] = value
|
||||
attrs = _record_attrs(record)
|
||||
if attrs:
|
||||
fields["attrs"] = redact(attrs)
|
||||
if record.exc_info:
|
||||
fields["exc"] = redact_text(logging.Formatter().formatException(record.exc_info))
|
||||
elif record.exc_text:
|
||||
# A QueueHandler-style hand-off formats the traceback into exc_text and drops exc_info.
|
||||
fields["exc"] = redact_text(record.exc_text)
|
||||
fields["version"] = settings.app_version
|
||||
return fields
|
||||
|
||||
|
||||
class JsonFormatter(logging.Formatter):
|
||||
"""One JSON object per line — the shape Loki and the `log_events` table both ingest."""
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
payload = record_fields(record)
|
||||
payload["ts"] = payload["ts"].isoformat(timespec="milliseconds")
|
||||
return json.dumps(payload, ensure_ascii=False, default=str)
|
||||
|
||||
|
||||
class TextFormatter(logging.Formatter):
|
||||
"""Human-readable single line for a terminal (`LOG_FORMAT=text`): the same content, ordered for
|
||||
reading. Redaction applies here too — a dev log is still a log someone pastes into an issue."""
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
stamp = datetime.fromtimestamp(record.created, timezone.utc).strftime("%H:%M:%S.%f")[:-3]
|
||||
head = f"{stamp} {record.levelname:<5} [{record.name}]"
|
||||
rid = getattr(record, "request_id", None)
|
||||
if rid:
|
||||
head += f" {rid}"
|
||||
code = getattr(record, "event", None)
|
||||
line = f"{head} {code + ' — ' if code else ''}{redact_text(_safe_message(record))}"
|
||||
attrs = _record_attrs(record)
|
||||
if attrs:
|
||||
line += f" {json.dumps(redact(attrs), ensure_ascii=False, default=str)}"
|
||||
if record.exc_info:
|
||||
line += "\n" + redact_text(self.formatException(record.exc_info))
|
||||
return line
|
||||
|
||||
|
||||
# --- configuration -------------------------------------------------------------------------
|
||||
# Third-party loggers that are noisy at our own level. Pinned explicitly so the diagnostics knob can
|
||||
# lower the ROOT level to DEBUG/TRACE without drowning the output in SQL statements and HTTP chatter
|
||||
# — raising THESE is a deliberate, separate act (E22.6's console).
|
||||
_THIRD_PARTY_LEVELS = {
|
||||
"sqlalchemy": "WARNING",
|
||||
"sqlalchemy.engine": "WARNING",
|
||||
"apscheduler": "WARNING",
|
||||
"urllib3": "WARNING",
|
||||
"httpx": "WARNING",
|
||||
"httpcore": "WARNING",
|
||||
"asyncio": "WARNING",
|
||||
"multipart": "WARNING",
|
||||
"PIL": "WARNING",
|
||||
"alembic": "INFO",
|
||||
# uvicorn's own startup/shutdown/WebSocket lines are worth keeping; its ACCESS log is not —
|
||||
# it is one line per request with the query string in it, replaced by `log_requests` below.
|
||||
"uvicorn": "INFO",
|
||||
"uvicorn.error": "INFO",
|
||||
"uvicorn.access": "CRITICAL",
|
||||
}
|
||||
|
||||
|
||||
def resolve_level(name: str) -> int:
|
||||
"""Level name → number. Both ways in (the `LOG_LEVEL` env and the `log_level` admin knob) are
|
||||
already held to the `LogLevel` choices, so an unknown name here is a programming error."""
|
||||
value = logging.getLevelName(name)
|
||||
if not isinstance(value, int):
|
||||
raise ValueError(f"unknown log level {name!r}")
|
||||
return value
|
||||
|
||||
|
||||
def configure_logging(source: str) -> None:
|
||||
"""Configure logging for THIS process. Call once, first thing, in every entry point:
|
||||
`main.py` (api), `worker.main`, `downloads/runner.main`, `alembic/env.py`.
|
||||
|
||||
`source` names the process in every line. The level starts at `LOG_LEVEL` (the bootstrap default);
|
||||
the runtime admin knob goes through `set_level()` instead, so changing it needs no restart.
|
||||
|
||||
Our loggers deliberately keep `propagate=True` and own no handler: everything reaches the single
|
||||
root StreamHandler. That is what keeps `caplog` working in tests, and it means a new logger name
|
||||
is configured the moment it is created."""
|
||||
global _source
|
||||
_source = source
|
||||
formatter = settings.log_format
|
||||
logging.config.dictConfig(
|
||||
{
|
||||
"version": 1,
|
||||
# Never disable loggers created at import time (module-level `getLogger` calls run
|
||||
# before this function does).
|
||||
"disable_existing_loggers": False,
|
||||
"formatters": {
|
||||
"json": {"()": f"{__name__}.JsonFormatter"},
|
||||
"text": {"()": f"{__name__}.TextFormatter"},
|
||||
},
|
||||
"filters": {
|
||||
"context": {"()": f"{__name__}.ContextFilter"},
|
||||
"gate": {"()": f"{__name__}.LevelGate"},
|
||||
},
|
||||
"handlers": {
|
||||
"stdout": {
|
||||
"class": "logging.StreamHandler",
|
||||
"stream": sys.stdout,
|
||||
"formatter": formatter,
|
||||
"filters": ["context", "gate"],
|
||||
}
|
||||
},
|
||||
"root": {"level": resolve_level(settings.log_level), "handlers": ["stdout"]},
|
||||
"loggers": {name: {"level": lvl} for name, lvl in _THIRD_PARTY_LEVELS.items()},
|
||||
}
|
||||
)
|
||||
set_level(settings.log_level)
|
||||
|
||||
|
||||
def set_level(name: str) -> None:
|
||||
"""Change the running process's level (the `log_level` admin knob). Third-party loggers keep
|
||||
their pinned levels, so turning on DEBUG shows OUR detail, not SQLAlchemy's."""
|
||||
global _base_level
|
||||
_base_level = resolve_level(name)
|
||||
_apply_root_level(force=True)
|
||||
|
||||
|
||||
def current_level() -> str:
|
||||
"""The configured level — not the root logger's, which an active elevation may have lowered."""
|
||||
return logging.getLevelName(_base_level or logging.getLogger().getEffectiveLevel())
|
||||
|
||||
|
||||
# --- per-user elevation (E22 S5) -----------------------------------------------------------
|
||||
# The base level (`LOG_LEVEL`, then the admin knob). NOTSET until configure_logging runs — a unit
|
||||
# test, a script — so the gate lets everything through there, as it did before the gate existed.
|
||||
_base_level = logging.NOTSET
|
||||
# user id → (level number, expiry) for every elevation this process has seen and not yet expired.
|
||||
_elevations: dict[int, tuple[int, datetime]] = {}
|
||||
_elevations_lock = threading.Lock()
|
||||
_root_applied: int | None = None
|
||||
|
||||
|
||||
def _track_elevation(user_id: int, level: int | None, until: datetime | None) -> None:
|
||||
with _elevations_lock:
|
||||
if level is None or until is None:
|
||||
_elevations.pop(user_id, None)
|
||||
else:
|
||||
_elevations[user_id] = (level, until)
|
||||
active = bool(_elevations)
|
||||
if active or _root_applied != _base_level:
|
||||
_apply_root_level()
|
||||
|
||||
|
||||
def _apply_root_level(force: bool = False) -> None:
|
||||
"""Root level = the lowest of the base and every unexpired elevation (expired ones are pruned
|
||||
here). `setLevel` clears every logger's level cache, so it runs only when the number changes."""
|
||||
global _root_applied
|
||||
if _base_level == logging.NOTSET:
|
||||
return # not configured: the root keeps whatever the process set up
|
||||
now = datetime.now(timezone.utc)
|
||||
# The level is SET under the lock too, not just computed: two threads that each computed a
|
||||
# different answer would otherwise apply them in either order, and the older one could win —
|
||||
# the root back at the base while an elevation is still active.
|
||||
with _elevations_lock:
|
||||
for user_id in [u for u, (_, until) in _elevations.items() if until <= now]:
|
||||
del _elevations[user_id]
|
||||
lowest = min([_base_level, *(lvl for lvl, _ in _elevations.values())])
|
||||
if force or lowest != _root_applied:
|
||||
logging.getLogger().setLevel(lowest)
|
||||
_root_applied = lowest
|
||||
|
||||
|
||||
# --- erased accounts ------------------------------------------------------------------------
|
||||
# `auth.purge_user` deletes an account's `log_events` rows, but a line about that account can still
|
||||
# be on its way: the erasing request's own access line is logged AFTER the handler committed, a queued
|
||||
# row waits for the sink's writer, a background task logs later. Those rows are stored without the
|
||||
# account (`logsink` asks here). Remembered for an hour in this process — a line later than that has
|
||||
# no request left to come from, and the retention job is the backstop for any other process.
|
||||
ERASED_MEMORY_SECONDS = 3600.0
|
||||
_erased: dict[int, float] = {}
|
||||
_erased_lock = threading.Lock()
|
||||
# Held by whoever checks rows against `is_erased` AND writes them (`erasure_guard`). `forget_user`
|
||||
# takes it too, so an erasure waits for a write that already passed the check to commit — and the
|
||||
# purge's DELETE, which runs after `forget_user`, then removes those rows as well. Without it a row
|
||||
# checked a moment before the erasure could be inserted a moment after the DELETE.
|
||||
_erasure_guard = threading.Lock()
|
||||
|
||||
|
||||
def erasure_guard() -> threading.Lock:
|
||||
"""Hold this around "drop erased accounts from these rows, insert them, commit"."""
|
||||
return _erasure_guard
|
||||
|
||||
|
||||
def forget_user(user_id: int) -> None:
|
||||
"""Mark `user_id` erased: no line logged about it from now on is stored under its id. Call BEFORE
|
||||
deleting its rows (see `_erasure_guard`)."""
|
||||
now = time.monotonic()
|
||||
with _erasure_guard, _erased_lock:
|
||||
for stale in [u for u, at in _erased.items() if now - at > ERASED_MEMORY_SECONDS]:
|
||||
del _erased[stale]
|
||||
_erased[user_id] = now
|
||||
with _elevations_lock:
|
||||
_elevations.pop(user_id, None)
|
||||
# That pop can have removed the lowest elevation, and the root level is the lowest of the base
|
||||
# and every unexpired one. Recomputed here rather than left to whichever later request happens
|
||||
# to call `_track_elevation` — the invariant `_apply_root_level` documents holds at every pop.
|
||||
_apply_root_level()
|
||||
|
||||
|
||||
def is_erased(user_id: int | None) -> bool:
|
||||
if user_id is None:
|
||||
return False
|
||||
with _erased_lock:
|
||||
at = _erased.get(user_id)
|
||||
return at is not None and time.monotonic() - at <= ERASED_MEMORY_SECONDS
|
||||
|
||||
|
||||
def current_source() -> str:
|
||||
"""The process name `configure_logging` was given (api, worker, runner, alembic)."""
|
||||
return _source
|
||||
|
||||
|
||||
# --- request logging -----------------------------------------------------------------------
|
||||
# A client may supply its own request id so a browser log line and a server line join up (E22 S4).
|
||||
# Accepted only in this shape: it ends up in a log field, and an unbounded header would be a way to
|
||||
# write arbitrary text into the log.
|
||||
_REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9._-]{8,64}$")
|
||||
|
||||
# Paths whose success is logged at DEBUG rather than INFO: the container healthcheck runs every 15s
|
||||
# (5.7k lines/day saying nothing) and hashed static assets are served by the hundred per page load.
|
||||
# A FAILURE on either still logs at WARNING/ERROR — this lowers the floor, it does not hide.
|
||||
_QUIET_PREFIXES = ("/healthz", "/assets/", "/static/")
|
||||
|
||||
_request_log = logging.getLogger("siftlode.http")
|
||||
|
||||
|
||||
async def log_requests(request, call_next):
|
||||
"""The single access-log line, replacing uvicorn's.
|
||||
|
||||
What changes versus `uvicorn.access`: the ROUTE TEMPLATE instead of the concrete path (so
|
||||
`/api/videos/{video_id}` groups), never the query string (it carried `?account=`, and at TRACE
|
||||
the redacted query belongs in attrs, not in the URL), a request id echoed back in
|
||||
`X-Request-Id` (with the template in `X-Route`), the duration, and the account — read from `request.state`, which the auth
|
||||
dependency fills, because this middleware sits outside both the session middleware and the
|
||||
downstream task's context."""
|
||||
incoming = request.headers.get("x-request-id", "")
|
||||
rid = incoming if _REQUEST_ID_RE.match(incoming) else uuid.uuid4().hex[:16]
|
||||
token = _request_id.set(rid)
|
||||
scope = _RequestScope()
|
||||
scope_token = _scope.set(scope)
|
||||
request.state.request_id = rid
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except Exception:
|
||||
# The unhandled-exception path: FastAPI's ServerErrorMiddleware re-raises after its own
|
||||
# 500, so without this the ONE request that most needs a log line is the one with no
|
||||
# access-log entry at all.
|
||||
_request_log.exception(
|
||||
"request failed",
|
||||
extra={
|
||||
**event(
|
||||
"http.request",
|
||||
method=request.method,
|
||||
path=_route_template(request),
|
||||
duration_ms=round((time.perf_counter() - started) * 1000, 1),
|
||||
),
|
||||
**_request_owner(request, logging.ERROR),
|
||||
},
|
||||
)
|
||||
_scope.reset(scope_token)
|
||||
_request_id.reset(token)
|
||||
raise
|
||||
duration_ms = round((time.perf_counter() - started) * 1000, 1)
|
||||
status = response.status_code
|
||||
quiet = request.url.path.startswith(_QUIET_PREFIXES)
|
||||
if status >= 500:
|
||||
level = logging.ERROR
|
||||
elif status >= 400:
|
||||
level = logging.WARNING
|
||||
else:
|
||||
level = logging.DEBUG if quiet else logging.INFO
|
||||
attrs: dict[str, Any] = {
|
||||
"method": request.method,
|
||||
"path": _route_template(request),
|
||||
"status": status,
|
||||
"duration_ms": duration_ms,
|
||||
}
|
||||
# Teljes (TRACE) the user consented to: the query string too — redacted, and in attrs only, never
|
||||
# in the message.
|
||||
if request.url.query and _traced(scope):
|
||||
attrs["query"] = redact_query(request.url.query)
|
||||
_request_log.log(
|
||||
level,
|
||||
"%s %s %s",
|
||||
request.method,
|
||||
_route_template(request),
|
||||
status,
|
||||
extra={**event("http.request", **attrs), **_request_owner(request, level)},
|
||||
)
|
||||
response.headers["X-Request-Id"] = rid
|
||||
# The template, so the browser's copy of this line groups the same way (frontend lib/http.ts).
|
||||
# Only a MATCHED route's: a template never carries a path parameter's value (a share token stays
|
||||
# `{token}`), while the raw-path fallback of an unmatched request could — the browser then
|
||||
# templates the path itself.
|
||||
matched = _matched_route(request)
|
||||
if matched:
|
||||
response.headers["X-Route"] = matched
|
||||
_scope.reset(scope_token)
|
||||
_request_id.reset(token)
|
||||
return response
|
||||
|
||||
|
||||
def _traced(scope: _RequestScope) -> bool:
|
||||
"""Whether this request's query string is logged: only under its OWN user's Teljes consent. An
|
||||
admin setting the whole process to TRACE gets more detail, not every user's search terms — the
|
||||
privacy policy promises those are kept only for a user who agreed to it."""
|
||||
return scope.level is not None and scope.level <= TRACE
|
||||
|
||||
|
||||
# Anonymous request lines kept per client per minute. Signed-in traffic is not capped: it is a known
|
||||
# account and exactly what an admin needs to see. Anonymous traffic is where the volume can come from
|
||||
# nobody in particular — a scanner walking `/wp-login.php`, a bot on a share link — and each of its
|
||||
# 404s is a WARNING that would otherwise land in `log_events` without limit. The client key is an
|
||||
# in-memory counter key only; no address is ever written to a log line or a row.
|
||||
ANONYMOUS_LINES_PER_MINUTE = 60
|
||||
_anonymous_lines = RateLimiter(ANONYMOUS_LINES_PER_MINUTE, 60.0)
|
||||
|
||||
|
||||
def _request_owner(request, level: int) -> dict[str, Any]:
|
||||
"""`extra=` fields naming who the request belongs to, and whether its line may be stored. The
|
||||
account comes from `request.state` (filled by the auth dependency), so it lands in the line's
|
||||
top-level `user_id` — the correlation field — rather than in `attrs`."""
|
||||
user_id = getattr(request.state, "user_id", None)
|
||||
if user_id is not None:
|
||||
return {"user_id": user_id}
|
||||
if level < logging.INFO:
|
||||
return {} # a quiet line is never stored anyway — don't spend the client's allowance on it
|
||||
return {} if _anonymous_lines.allow(client_key(request)) else {PERSIST: False}
|
||||
|
||||
|
||||
def _matched_route(request) -> str | None:
|
||||
"""The matched route's template (`/api/videos/{video_id}`), or None when nothing matched."""
|
||||
return getattr(request.scope.get("route"), "path", None)
|
||||
|
||||
|
||||
def _route_template(request) -> str:
|
||||
"""The matched route's template, else the raw path for a 404 or a static file. Never the query
|
||||
string."""
|
||||
return _matched_route(request) or request.url.path
|
||||
@@ -0,0 +1,315 @@
|
||||
"""The `log_events` sink (E22 S3): the stored copy of the log the diagnostics console reads.
|
||||
|
||||
stdout stays the complete log (docker keeps 3×50 MB of it); this table keeps the part worth querying
|
||||
for `log_retention_days`:
|
||||
|
||||
* every WARNING and above,
|
||||
* the `http.request` line of every request at INFO or above (not the DEBUG-level health checks and
|
||||
static files),
|
||||
* every line of a request whose user raised their diagnostics level (E22 S5 — `logs.LevelGate`
|
||||
marks them), which is the detail they consented to have kept,
|
||||
* nothing a caller marked `extra={logs.PERSIST: False}` — how the request log caps anonymous
|
||||
traffic per client.
|
||||
|
||||
What the table must never do is slow down or break the code that logs. So a line is redacted and
|
||||
turned into a row on the CALLER's thread (the same `logs.record_fields` the stdout line is built
|
||||
from), put on a bounded queue without blocking, and written in batches by one background thread
|
||||
on its own connection — never inside a request's transaction. A full queue drops the row and
|
||||
counts it; a failed write drops the batch and says so on stderr. Both are reported rather than
|
||||
retried: a log that retries into a sick database is how a small outage becomes a big one.
|
||||
|
||||
Flood control is the second half. On 2026-09-16 a YouTube RSS outage made prod log 11k near-identical
|
||||
warnings in four hours. stdout keeps them all; the table keeps `REPEATS_PER_MINUTE` of each message
|
||||
template per minute and one `log.repeats_suppressed` row saying how many more there were.
|
||||
|
||||
Core layer: imports `logs`, `db`, `models`, `config` — nothing above them.
|
||||
"""
|
||||
|
||||
import atexit
|
||||
import json
|
||||
import logging
|
||||
import queue
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import delete, insert
|
||||
from sqlalchemy.engine import Engine
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import logs
|
||||
from app.config import settings
|
||||
from app.db import engine
|
||||
from app.models import LogEvent
|
||||
|
||||
QUEUE_MAX = 10_000
|
||||
BATCH_MAX = 500
|
||||
FLUSH_SECONDS = 1.0
|
||||
REPEATS_PER_MINUTE = 30
|
||||
REPEAT_WINDOW_SECONDS = 60.0
|
||||
# Distinct templates tracked per window. Past this many, a new template is stored without counting
|
||||
# — the limiter must not become the unbounded thing it exists to prevent.
|
||||
_REPEAT_KEYS_MAX = 2_000
|
||||
|
||||
# Column widths from models.LogEvent. A value past its column would fail the WHOLE batch it rides in.
|
||||
_LIMITS = {
|
||||
"level": 10,
|
||||
"source": 16,
|
||||
"logger": 128,
|
||||
"event": 64,
|
||||
"request_id": 64,
|
||||
"debug_session_id": 64,
|
||||
"version": 64,
|
||||
"msg": 8_000,
|
||||
"exc": 32_000,
|
||||
}
|
||||
|
||||
_log = logging.getLogger("siftlode.logsink")
|
||||
|
||||
|
||||
def should_store(record: logging.LogRecord) -> bool:
|
||||
"""Whether this record belongs in `log_events` (see the module docstring)."""
|
||||
if getattr(record, logs.PERSIST, True) is False:
|
||||
return False
|
||||
if record.levelno >= logging.WARNING or getattr(record, "elevated", False):
|
||||
return True
|
||||
return getattr(record, "event", None) == "http.request" and record.levelno >= logging.INFO
|
||||
|
||||
|
||||
def _without_nul(value: Any) -> Any:
|
||||
"""`value` with every NUL character replaced, in keys and nested values alike. Postgres refuses
|
||||
NUL in text and `\\u0000` in JSONB, and a refused row fails the WHOLE batch it rides in — every
|
||||
user's rows with it. Anything logged can carry one (`GET /assets/%00` logs the decoded path)."""
|
||||
if isinstance(value, str):
|
||||
return value.replace("\x00", "�")
|
||||
if isinstance(value, dict):
|
||||
return {_without_nul(k): _without_nul(v) for k, v in value.items()}
|
||||
if isinstance(value, list):
|
||||
return [_without_nul(v) for v in value]
|
||||
return value
|
||||
|
||||
|
||||
def to_row(fields: dict[str, Any]) -> dict[str, Any]:
|
||||
"""`logs.record_fields` output → a `log_events` row: known columns only, text clipped to its
|
||||
column and free of NUL, attrs made JSON-safe (anything exotic becomes its `str`, as on stdout)."""
|
||||
row = {
|
||||
column: fields.get(column) for column in LogEvent.__table__.columns.keys() if column != "id"
|
||||
}
|
||||
for column, limit in _LIMITS.items():
|
||||
value = row.get(column)
|
||||
if isinstance(value, str):
|
||||
value = _without_nul(value)
|
||||
row[column] = value[: limit - 1] + "…" if len(value) > limit else value
|
||||
if row["attrs"] is not None:
|
||||
row["attrs"] = _without_nul(
|
||||
json.loads(json.dumps(row["attrs"], ensure_ascii=False, default=str))
|
||||
)
|
||||
return row
|
||||
|
||||
|
||||
def without_erased(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""The rows, with the account taken off any whose account was erased meanwhile
|
||||
(`logs.forget_user`). Checked right before the INSERT, under `logs.erasure_guard()` held until
|
||||
the commit: a row built before the erasure can still be waiting for its batch."""
|
||||
for row in rows:
|
||||
if logs.is_erased(row.get("user_id")):
|
||||
row["user_id"] = None
|
||||
return rows
|
||||
|
||||
|
||||
def _notice(code: str, msg: str, **attrs: Any) -> dict[str, Any]:
|
||||
"""A row the sink writes about ITSELF. Built directly, not logged: logging it would route it back
|
||||
through this sink (and the writer thread's own records are deliberately ignored)."""
|
||||
return to_row(
|
||||
{
|
||||
"ts": datetime.now(timezone.utc),
|
||||
"level": "WARNING",
|
||||
"sev": logs.SEVERITY["WARNING"],
|
||||
"source": logs.current_source(),
|
||||
"logger": _log.name,
|
||||
"event": code,
|
||||
"msg": msg,
|
||||
"attrs": attrs,
|
||||
"version": settings.app_version,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class _Repeats:
|
||||
"""Per-template admission for one minute at a time. Used by the writer thread only (unlocked)."""
|
||||
|
||||
def __init__(self, per_window: int, window_seconds: float):
|
||||
self.per_window = per_window
|
||||
self.window = window_seconds
|
||||
self.started = time.monotonic()
|
||||
self.seen: dict[tuple[str, str, int], int] = {}
|
||||
|
||||
def admit(self, key: tuple[str, str, int] | None) -> bool:
|
||||
if key is None:
|
||||
return True
|
||||
count = self.seen.get(key)
|
||||
if count is None and len(self.seen) >= _REPEAT_KEYS_MAX:
|
||||
return True
|
||||
self.seen[key] = (count or 0) + 1
|
||||
return self.seen[key] <= self.per_window
|
||||
|
||||
def roll(self, now: float) -> list[dict[str, Any]]:
|
||||
"""Close the window if it is over: one notice per template that went past the limit."""
|
||||
if now - self.started < self.window:
|
||||
return []
|
||||
notices = [
|
||||
_notice(
|
||||
"log.repeats_suppressed",
|
||||
f"{count - self.per_window} more like this were not stored",
|
||||
logger=logger,
|
||||
template=logs.redact_text(template),
|
||||
suppressed=count - self.per_window,
|
||||
window_seconds=int(self.window),
|
||||
)
|
||||
for (logger, template, _level), count in self.seen.items()
|
||||
if count > self.per_window
|
||||
]
|
||||
self.seen.clear()
|
||||
self.started = now
|
||||
return notices
|
||||
|
||||
|
||||
class LogSink(logging.Handler):
|
||||
"""The handler plus its writer. `start()` runs the writer thread; a test drives `write_pending()`
|
||||
directly instead, on its own thread, with no timing involved."""
|
||||
|
||||
def __init__(self, bind: Engine = engine):
|
||||
super().__init__(level=logging.NOTSET)
|
||||
# Correlation fields are contextvars: capture them now, on the thread that logged.
|
||||
self.addFilter(logs.ContextFilter())
|
||||
self._bind = bind
|
||||
self._queue: queue.Queue[tuple[tuple[str, str, int] | None, dict[str, Any]]] = queue.Queue(
|
||||
QUEUE_MAX
|
||||
)
|
||||
self._dropped = 0
|
||||
self._dropped_lock = threading.Lock()
|
||||
self._repeats = _Repeats(REPEATS_PER_MINUTE, REPEAT_WINDOW_SECONDS)
|
||||
self._stopping = threading.Event()
|
||||
self._writer: threading.Thread | None = None
|
||||
|
||||
# --- caller side ---------------------------------------------------------------------------
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
if threading.current_thread() is self._writer:
|
||||
return # the writer's own trouble (a driver warning mid-insert) must not feed itself
|
||||
if not should_store(record):
|
||||
return
|
||||
try:
|
||||
row = to_row(logs.record_fields(record))
|
||||
except Exception: # noqa: BLE001 — a record we cannot turn into a row is reported, not raised
|
||||
self.handleError(record)
|
||||
return
|
||||
# Request lines are one per request by nature; only the repeated SAME message is a flood.
|
||||
key = (
|
||||
None
|
||||
if row["event"] == "http.request"
|
||||
else (record.name, str(record.msg), record.levelno)
|
||||
)
|
||||
try:
|
||||
self._queue.put_nowait((key, row))
|
||||
except queue.Full:
|
||||
with self._dropped_lock:
|
||||
self._dropped += 1
|
||||
|
||||
# --- writer side ---------------------------------------------------------------------------
|
||||
def write_pending(self, wait: float = 0.0) -> int:
|
||||
"""Write one batch: whatever is queued (up to BATCH_MAX), waiting up to `wait` seconds for the
|
||||
first item. Returns the number of rows written, notices included."""
|
||||
rows: list[dict[str, Any]] = []
|
||||
try:
|
||||
item = self._queue.get(timeout=wait) if wait > 0 else self._queue.get_nowait()
|
||||
except queue.Empty:
|
||||
item = None
|
||||
while item is not None:
|
||||
key, row = item
|
||||
if self._repeats.admit(key):
|
||||
rows.append(row)
|
||||
if len(rows) >= BATCH_MAX:
|
||||
break
|
||||
try:
|
||||
item = self._queue.get_nowait()
|
||||
except queue.Empty:
|
||||
item = None
|
||||
rows.extend(self._repeats.roll(time.monotonic()))
|
||||
with self._dropped_lock:
|
||||
dropped, self._dropped = self._dropped, 0
|
||||
if dropped:
|
||||
rows.append(
|
||||
_notice("log.sink.dropped", f"{dropped} log events were dropped", dropped=dropped)
|
||||
)
|
||||
if not rows:
|
||||
return 0
|
||||
try:
|
||||
# The connection is checked out BEFORE the guard, and every request path takes them in
|
||||
# that order too (its session already holds a connection when it reaches the guard). The
|
||||
# other way round, this thread would hold a process-global lock while waiting up to
|
||||
# `pool_timeout` for a connection, and the requests that could return one are the ones
|
||||
# blocked on the lock. The guard still spans the check, the insert and the commit.
|
||||
with self._bind.connect() as conn, logs.erasure_guard(), conn.begin():
|
||||
conn.execute(insert(LogEvent), without_erased(rows))
|
||||
except Exception as exc: # noqa: BLE001 — the database is down or refused the batch
|
||||
print(
|
||||
f"log sink: could not store {len(rows)} log events: {logs.redact_text(str(exc))[:300]}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 0
|
||||
return len(rows)
|
||||
|
||||
def _run(self) -> None:
|
||||
while not self._stopping.is_set():
|
||||
self.write_pending(wait=FLUSH_SECONDS)
|
||||
while self.write_pending(): # drain what arrived before the stop
|
||||
pass
|
||||
|
||||
def start(self) -> None:
|
||||
self._writer = threading.Thread(target=self._run, name="log-sink", daemon=True)
|
||||
self._writer.start()
|
||||
|
||||
def stop(self, timeout: float = 5.0) -> None:
|
||||
self._stopping.set()
|
||||
if self._writer is not None:
|
||||
self._writer.join(timeout)
|
||||
|
||||
|
||||
_sink: LogSink | None = None
|
||||
|
||||
|
||||
def start() -> None:
|
||||
"""Attach the sink to the root logger and start its writer. Call after `logs.configure_logging`
|
||||
(which replaces the root handlers) and once the schema exists. A no-op when `LOG_DB_SINK` is off
|
||||
or it is already running."""
|
||||
global _sink
|
||||
if _sink is not None or not settings.log_db_sink:
|
||||
return
|
||||
_sink = LogSink()
|
||||
# The same level gate as stdout: the root may be lowered for one user's elevated requests, and
|
||||
# everyone else's detail must not reach the table through that gap either.
|
||||
_sink.addFilter(logs.LevelGate())
|
||||
logging.getLogger().addHandler(_sink)
|
||||
_sink.start()
|
||||
# A short-lived process (the download child) exits without a lifespan: flush on the way out.
|
||||
atexit.register(stop)
|
||||
|
||||
|
||||
def stop() -> None:
|
||||
"""Detach the sink and write what is still queued."""
|
||||
global _sink
|
||||
if _sink is None:
|
||||
return
|
||||
logging.getLogger().removeHandler(_sink)
|
||||
_sink.stop()
|
||||
_sink = None
|
||||
|
||||
|
||||
def gc(db: Session, retention_days: int) -> int:
|
||||
"""Stage the deletion of rows older than `retention_days` (at least one day — stored logs always
|
||||
expire). The caller commits. Returns the number of rows removed."""
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=max(1, retention_days))
|
||||
result = db.execute(delete(LogEvent).where(LogEvent.ts < cutoff))
|
||||
return result.rowcount or 0
|
||||
+32
-11
@@ -1,7 +1,6 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import math
|
||||
import sys
|
||||
from contextlib import asynccontextmanager, suppress
|
||||
from pathlib import Path
|
||||
|
||||
@@ -9,16 +8,13 @@ from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.encoders import jsonable_encoder
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
|
||||
# When started via uvicorn --log-config the "siftlode" logger is already configured
|
||||
# (see log_config.json). This block is a timestamped fallback for other entrypoints
|
||||
# (tests, scripts) so our logs are never silently dropped.
|
||||
_siftlode_logger = logging.getLogger("siftlode")
|
||||
if not _siftlode_logger.handlers:
|
||||
_handler = logging.StreamHandler(sys.stdout)
|
||||
_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)-5s [%(name)s] %(message)s"))
|
||||
_siftlode_logger.addHandler(_handler)
|
||||
_siftlode_logger.setLevel(logging.INFO)
|
||||
_siftlode_logger.propagate = False
|
||||
# Logging is configured HERE, before the route modules are imported, so every logger they create at
|
||||
# import time is already covered. ONE format for the api, the worker, the download child and alembic
|
||||
# alike (app/logs.py) — this replaces the old uvicorn --log-config + the hand-rolled fallback that
|
||||
# used to live here, and uvicorn's access log is replaced by the `http.request` middleware below.
|
||||
from app.logs import configure_logging, current_level, log_requests, set_level
|
||||
|
||||
configure_logging("api")
|
||||
|
||||
log = logging.getLogger("siftlode.app")
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
@@ -27,7 +23,9 @@ from fastapi.staticfiles import StaticFiles
|
||||
from sqlalchemy.exc import DataError
|
||||
from starlette.middleware.sessions import SessionMiddleware
|
||||
|
||||
from app import logsink
|
||||
from app import state
|
||||
from app import sysconfig
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.plex import stream as plex_stream
|
||||
@@ -38,6 +36,7 @@ from app.routes import (
|
||||
auth as auth_routes,
|
||||
channels,
|
||||
config as config_routes,
|
||||
diagnostics as diagnostics_routes,
|
||||
discover as discover_routes,
|
||||
downloads,
|
||||
feed,
|
||||
@@ -63,9 +62,21 @@ from app.scheduler import shutdown_scheduler, start_scheduler
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
log.info("Siftlode starting up")
|
||||
# The stored copy of the log (log_events). The entrypoint has already migrated the schema.
|
||||
logsink.start()
|
||||
# First-run: if the instance isn't configured yet, mint a fresh one-time setup token and print
|
||||
# the wizard URL so the operator can open it (the app runs in setup mode until they finish).
|
||||
with SessionLocal() as db:
|
||||
# The admin's runtime level (sysconfig `log_level`) wins over the LOG_LEVEL bootstrap
|
||||
# default, so a level chosen in the UI survives a restart without touching the environment.
|
||||
# Best-effort on purpose: a LOGGING knob must never be the thing that stops the app from
|
||||
# booting — an unreachable or not-yet-migrated database has to start into the env level.
|
||||
try:
|
||||
set_level(sysconfig.get_str(db, "log_level"))
|
||||
except Exception as exc: # noqa: BLE001 — any DB/config failure keeps the bootstrap level
|
||||
log.warning(
|
||||
"could not read the stored log level (%s); staying at %s", exc, current_level()
|
||||
)
|
||||
if not state.is_configured(db):
|
||||
token = state.rotate_setup_token(db)
|
||||
url = f"{settings.app_base}/setup?token={token}"
|
||||
@@ -100,6 +111,7 @@ async def lifespan(app: FastAPI):
|
||||
# Kills a running subtitle extraction: its worker would otherwise hold interpreter exit for
|
||||
# the rest of the read (see plex_subtitles.shutdown).
|
||||
plex_subtitles.shutdown()
|
||||
logsink.stop() # last: write what the shutdown itself just logged
|
||||
|
||||
|
||||
# How often idle HLS sessions are reaped (see plex/stream.py `reap_idle`).
|
||||
@@ -243,6 +255,14 @@ async def security_headers(request, call_next):
|
||||
return response
|
||||
|
||||
|
||||
# The access log (app/logs.py): request id, route template, status, duration — one line, never a
|
||||
# query string. Registered LAST so it is the OUTERMOST http middleware and therefore times the whole
|
||||
# request, including what the three above do. Being outermost also puts it OUTSIDE SessionMiddleware,
|
||||
# which is why the account comes from `request.state.user_id` (set by the auth dependency, the one
|
||||
# place a session is actually resolved into an account) rather than from `request.session`.
|
||||
app.middleware("http")(log_requests)
|
||||
|
||||
|
||||
app.include_router(health.router)
|
||||
app.include_router(auth_routes.router)
|
||||
app.include_router(sync.router)
|
||||
@@ -251,6 +271,7 @@ app.include_router(feed.router)
|
||||
app.include_router(discover_routes.router)
|
||||
app.include_router(search_routes.router)
|
||||
app.include_router(me.router)
|
||||
app.include_router(diagnostics_routes.router)
|
||||
app.include_router(notifications.router)
|
||||
app.include_router(messages.router)
|
||||
app.include_router(channels.router)
|
||||
|
||||
@@ -12,6 +12,7 @@ from sqlalchemy import (
|
||||
Index,
|
||||
Integer,
|
||||
JSON,
|
||||
SmallInteger,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
@@ -102,6 +103,16 @@ class User(Base, TimestampMixin, CanonicalEmailMixin):
|
||||
# current_user rejects any cookie whose epoch is stale, so a stolen/copied cookie dies on those
|
||||
# events (client-side signed cookies otherwise stay valid until they expire).
|
||||
session_epoch: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
|
||||
# When this account last signed in (Google or password), stamped by the one sign-in path,
|
||||
# routes/auth._establish_session. NULL = not since the column existed (0085; no backfill is
|
||||
# possible). The shared demo account is never stamped — its "last login" would mean nobody.
|
||||
last_login_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# The diagnostics level this user consented to (E22 S5) and when it lapses back to the base
|
||||
# level. Both NULL = base. Written only by `PUT /api/me/diagnostics`, which computes the expiry
|
||||
# itself; read through `app.diagnostics.effective_level`, never directly (an expired level is
|
||||
# not cleared eagerly — it simply stops counting).
|
||||
diagnostics_level: Mapped[str | None] = mapped_column(String(8))
|
||||
diagnostics_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
display_name: Mapped[str | None] = mapped_column(String(255))
|
||||
avatar_url: Mapped[str | None] = mapped_column(String(1024))
|
||||
role: Mapped[str] = mapped_column(String(16), default="user", server_default="user")
|
||||
@@ -539,6 +550,37 @@ class AuditLog(Base):
|
||||
)
|
||||
|
||||
|
||||
class LogEvent(Base):
|
||||
"""One stored log event (E22 S3) — the `log_events` table the diagnostics console reads. Rows are
|
||||
written by app/logsink.py's background writer, never inside a request's transaction, already
|
||||
redacted (the same `logs.record_fields` the stdout line is built from).
|
||||
|
||||
`user_id` has NO foreign key on purpose: the writer runs behind the event, so a line about an
|
||||
account can arrive after that account was purged, and a constraint would make it fail the whole
|
||||
batch it travels in. `auth.purge_user` deletes an account's rows explicitly instead, after
|
||||
`logs.forget_user` makes every later write in this process store such a line without the account;
|
||||
the retention job (`log_retention_days`) bounds whatever another process writes late."""
|
||||
|
||||
__tablename__ = "log_events"
|
||||
__table_args__ = (Index("ix_log_events_user_ts", "user_id", "ts"),)
|
||||
|
||||
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
|
||||
ts: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
|
||||
level: Mapped[str] = mapped_column(String(10))
|
||||
sev: Mapped[int] = mapped_column(SmallInteger)
|
||||
source: Mapped[str] = mapped_column(String(16))
|
||||
logger: Mapped[str] = mapped_column(String(128))
|
||||
event: Mapped[str | None] = mapped_column(String(64))
|
||||
msg: Mapped[str] = mapped_column(Text)
|
||||
attrs: Mapped[dict | None] = mapped_column(JSONB)
|
||||
exc: Mapped[str | None] = mapped_column(Text)
|
||||
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
|
||||
user_id: Mapped[int | None] = mapped_column(Integer)
|
||||
debug_session_id: Mapped[str | None] = mapped_column(String(64))
|
||||
job_id: Mapped[int | None] = mapped_column(Integer)
|
||||
version: Mapped[str | None] = mapped_column(String(64))
|
||||
|
||||
|
||||
class AppState(Base):
|
||||
"""Single-row global app state (admin-controlled)."""
|
||||
|
||||
|
||||
@@ -404,7 +404,7 @@ def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) ->
|
||||
source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
|
||||
if not transcode:
|
||||
return (False, False)
|
||||
mode = (sysconfig.get_str(db, "plex_transcode_hw") or "auto").lower()
|
||||
mode = sysconfig.get_str(db, "plex_transcode_hw")
|
||||
if mode == "off" or not _render_node_present():
|
||||
return (False, False)
|
||||
if mode == "on":
|
||||
@@ -664,7 +664,7 @@ def start_session(
|
||||
# remux ignore preset/crf/max_height, and remux must not touch the DB for them).
|
||||
preset, crf, max_height = "veryfast", 20, 0
|
||||
if transcode and not hw:
|
||||
preset = sysconfig.get_str(db, "plex_transcode_preset") or "veryfast"
|
||||
preset = sysconfig.get_str(db, "plex_transcode_preset")
|
||||
crf = sysconfig.get_int(db, "plex_transcode_crf")
|
||||
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
|
||||
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
|
||||
|
||||
+15
-11
@@ -17,10 +17,14 @@ from app.config import settings
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
def __init__(self, max_events: int, window_seconds: float):
|
||||
"""At most `max_events` attempts per `window_seconds` per key — or, when each attempt passes a
|
||||
`cost`, at most that much total cost (a byte budget counts bytes, not calls)."""
|
||||
|
||||
def __init__(self, max_events: float, window_seconds: float):
|
||||
self.max_events = max_events
|
||||
self.window = window_seconds
|
||||
self._hits: dict[str, list[float]] = {}
|
||||
# key → (monotonic time, cost) of every counted attempt still inside the window
|
||||
self._hits: dict[str, list[tuple[float, float]]] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def reset(self) -> None:
|
||||
@@ -36,27 +40,27 @@ class RateLimiter:
|
||||
count."""
|
||||
cutoff = time.monotonic() - self.window
|
||||
with self._lock:
|
||||
return sum(1 for t in self._hits.get(key, []) if t > cutoff) >= self.max_events
|
||||
return sum(c for t, c in self._hits.get(key, []) if t > cutoff) >= self.max_events
|
||||
|
||||
def allow(self, key: str) -> bool:
|
||||
"""Record an attempt for ``key`` and return whether it is within the limit.
|
||||
def allow(self, key: str, cost: float = 1) -> bool:
|
||||
"""Record an attempt costing ``cost`` for ``key`` and return whether it is within the limit.
|
||||
|
||||
True -> under the cap (the attempt is counted).
|
||||
False -> the cap for the current window is already reached (attempt NOT counted, so a
|
||||
True -> within the cap (the attempt is counted).
|
||||
False -> the attempt would pass the cap for the current window (attempt NOT counted, so a
|
||||
blocked caller can't keep pushing the window forward)."""
|
||||
now = time.monotonic()
|
||||
cutoff = now - self.window
|
||||
with self._lock:
|
||||
hits = [t for t in self._hits.get(key, []) if t > cutoff]
|
||||
if len(hits) >= self.max_events:
|
||||
hits = [(t, c) for t, c in self._hits.get(key, []) if t > cutoff]
|
||||
if sum(c for _, c in hits) + cost > self.max_events:
|
||||
self._hits[key] = hits
|
||||
return False
|
||||
hits.append(now)
|
||||
hits.append((now, cost))
|
||||
self._hits[key] = hits
|
||||
# Opportunistic cleanup so abandoned keys don't accumulate unboundedly.
|
||||
if len(self._hits) > 4096:
|
||||
for k in list(self._hits):
|
||||
fresh = [t for t in self._hits[k] if t > cutoff]
|
||||
fresh = [(t, c) for t, c in self._hits[k] if t > cutoff]
|
||||
if fresh:
|
||||
self._hits[k] = fresh
|
||||
else:
|
||||
|
||||
@@ -408,6 +408,8 @@ def _serialize_user(u: User) -> dict:
|
||||
# confusing sign-in.
|
||||
"google_email": u.google_email if u.google_email != u.email else None,
|
||||
"created_at": iso(u.created_at),
|
||||
# NULL = no sign-in since the column existed (0085), or the shared demo account.
|
||||
"last_login_at": iso(u.last_login_at),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -479,16 +479,8 @@ async def callback(
|
||||
db.flush()
|
||||
|
||||
_store_token(db, user, token)
|
||||
_establish_session(request, user)
|
||||
db.commit()
|
||||
|
||||
# Multi-session: remember every account that has authenticated in this browser so the
|
||||
# user can switch between them without going through Google again. Only accounts that
|
||||
# actually completed OAuth here land in this list (it's the proof they may be switched to).
|
||||
accounts = [a for a in (request.session.get("account_ids") or []) if a != user.id]
|
||||
accounts.append(user.id)
|
||||
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
|
||||
request.session["user_id"] = user.id
|
||||
_remember_epoch(request, user)
|
||||
log.info("Login: %s (id=%s, role=%s)", email, user.id, user.role)
|
||||
return RedirectResponse(url="/")
|
||||
|
||||
@@ -675,13 +667,18 @@ def demo_login(body: EmailIn, request: Request, db: Session = Depends(get_db)) -
|
||||
|
||||
|
||||
def _establish_session(request: Request, user: User) -> None:
|
||||
"""Sign `user` in for this browser session, mirroring the Google callback's multi-account
|
||||
handling so password sign-in joins the same account switcher."""
|
||||
"""Sign `user` in for this browser session — the ONE sign-in path, shared by the Google callback
|
||||
and password login, so both join the same account switcher and both stamp `last_login_at` (the
|
||||
caller commits). Multi-session: every account that authenticated in this browser is remembered so
|
||||
the user can switch between them; only accounts that actually proved themselves HERE land in that
|
||||
wallet (it is the proof they may be switched to). The demo entry deliberately does not come
|
||||
through here: it clears the wallet instead, and its shared account has no meaningful last login."""
|
||||
accounts = [a for a in (request.session.get("account_ids") or []) if a != user.id]
|
||||
accounts.append(user.id)
|
||||
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
|
||||
request.session["user_id"] = user.id
|
||||
_remember_epoch(request, user)
|
||||
user.last_login_at = datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _app_base() -> str:
|
||||
@@ -1051,6 +1048,7 @@ def password_login(
|
||||
if not user.is_active:
|
||||
raise _login_refused(403, "pending_approval", user)
|
||||
_establish_session(request, user)
|
||||
db.commit()
|
||||
log.info("Password login: %s (id=%s, role=%s)", email, user.id, user.role)
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ from sqlalchemy.orm import Session
|
||||
|
||||
from app import audit
|
||||
from app import email as email_mod
|
||||
from app import logs
|
||||
from app import mailtext
|
||||
from app import sysconfig
|
||||
from app.audit import AuditAction
|
||||
@@ -29,6 +30,14 @@ def _redact(v):
|
||||
return _URL_USERINFO.sub(r"\1***@", v) if isinstance(v, str) else v
|
||||
|
||||
|
||||
def _apply_runtime(db: Session, key: str) -> None:
|
||||
"""Some knobs change how THIS process behaves, not just what a later read returns — they have to
|
||||
take effect now, without a restart. One dispatch for both call sites (set and reset), so the next
|
||||
such key can't be wired into one of them and forgotten in the other."""
|
||||
if key == "log_level":
|
||||
logs.set_level(sysconfig.get_str(db, "log_level"))
|
||||
|
||||
|
||||
@router.get("")
|
||||
def get_config(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
|
||||
return sysconfig.describe(db)
|
||||
@@ -80,6 +89,7 @@ def set_config(
|
||||
after=None if s.secret else {"value": _redact(new)},
|
||||
)
|
||||
db.commit()
|
||||
_apply_runtime(db, key)
|
||||
return sysconfig.describe(db)
|
||||
|
||||
|
||||
@@ -104,6 +114,7 @@ def reset_config(
|
||||
before=None if s.secret else {"value": _redact(old)},
|
||||
)
|
||||
db.commit()
|
||||
_apply_runtime(db, key)
|
||||
return sysconfig.describe(db)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
"""The browser's log, sent while its user has a raised diagnostics level (E22 S5).
|
||||
|
||||
The client keeps its log in memory (frontend `lib/log.ts`). Only after the user consented to Részletes
|
||||
or Teljes in Settings → Diagnosztika does it stream that log here, in batches, and each batch lands in
|
||||
`log_events` beside the server's own rows: `source="web"`, the account from the SESSION (never from
|
||||
the body), and the request id each event carries, which is what joins a browser line to the server
|
||||
line of the same request.
|
||||
|
||||
Nothing from the body is trusted as-is:
|
||||
* only a JSON Content-Type is read, and the whole body is capped before it is parsed, and so are
|
||||
the number of events and each event's attributes;
|
||||
* the pace is capped per account, and so is what an account may store per hour;
|
||||
* everything is redacted AGAIN with the server's own rules — the client redacts at capture, but a
|
||||
modified client would not;
|
||||
* an event below the consented level is dropped (a Részletes consent does not store what only
|
||||
Teljes may carry), and so is the whole batch once the level has lapsed;
|
||||
* the timestamp is clamped to the consent window, so a row cannot dodge the retention job by
|
||||
claiming to come from the future.
|
||||
"""
|
||||
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field, ValidationError
|
||||
from sqlalchemy import insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import diagnostics, logs, logsink
|
||||
from app.auth import current_user
|
||||
from app.db import get_db
|
||||
from app.models import LogEvent, User
|
||||
from app.ratelimit import RateLimiter
|
||||
|
||||
router = APIRouter(prefix="/api/diagnostics", tags=["diagnostics"])
|
||||
|
||||
BODY_MAX_BYTES = 512 * 1024
|
||||
EVENTS_MAX = 200
|
||||
ATTRS_MAX_BYTES = 16 * 1024
|
||||
# The client flushes every few seconds and when its buffer fills; this is room for a burst, not a
|
||||
# pace anything legitimate keeps up for a minute.
|
||||
_batches = RateLimiter(max_events=30, window_seconds=60)
|
||||
# What one account may STORE per hour. The pace cap alone still lets a self-consented account write
|
||||
# ~15 MiB a minute into the one shared database, kept for the whole retention window; a real session
|
||||
# at Teljes stores a few hundred kB an hour. Past the budget a batch is accepted and not stored (a
|
||||
# refusal would only make the client resend it), and one notice per hour says so.
|
||||
STORED_BYTES_PER_HOUR = 5 * 1024 * 1024
|
||||
_stored_bytes = RateLimiter(max_events=STORED_BYTES_PER_HOUR, window_seconds=3600)
|
||||
_budget_notices = RateLimiter(max_events=1, window_seconds=3600)
|
||||
|
||||
ClientLevel = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
|
||||
|
||||
|
||||
class ClientEvent(BaseModel):
|
||||
ts: datetime
|
||||
level: ClientLevel
|
||||
event: str = Field(max_length=64, pattern=r"^[a-z][a-z0-9_]*(\.[a-z0-9_]+)*$")
|
||||
msg: str = Field(max_length=8_000)
|
||||
attrs: dict[str, Any] | None = None
|
||||
request_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$")
|
||||
tab_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9_-]{1,32}$")
|
||||
version: str | None = Field(default=None, max_length=64)
|
||||
|
||||
|
||||
class ClientEventsIn(BaseModel):
|
||||
events: list[ClientEvent] = Field(max_length=EVENTS_MAX)
|
||||
|
||||
|
||||
async def _capped_body(request: Request) -> bytes:
|
||||
"""The request body, refused the moment it passes BODY_MAX_BYTES while still arriving. Counted,
|
||||
not taken from Content-Length: a declared length can be absent (chunked) or wrong, and counting
|
||||
stops reading at the same point a declared-length check would have."""
|
||||
body = bytearray()
|
||||
async for chunk in request.stream():
|
||||
body += chunk
|
||||
if len(body) > BODY_MAX_BYTES:
|
||||
raise HTTPException(status_code=413, detail={"code": "too_large"})
|
||||
return bytes(body)
|
||||
|
||||
|
||||
async def _parsed_events(request: Request) -> ClientEventsIn:
|
||||
"""The body as the model — parsed only after the size cap, which is why this is a dependency and
|
||||
not a plain body parameter (FastAPI reads a declared body in full before anything else runs).
|
||||
|
||||
JSON only, by its Content-Type, as a declared body would be. Parsing any body as JSON would let a
|
||||
CORS-simple `text/plain` form post from a same-site page write events into a signed-in user's
|
||||
log; the app's own client always sends `application/json`, the unload beacon included."""
|
||||
media_type = request.headers.get("content-type", "").split(";")[0].strip().lower()
|
||||
if media_type != "application/json" and not media_type.endswith("+json"):
|
||||
raise HTTPException(status_code=415, detail={"code": "json_required"})
|
||||
raw = await _capped_body(request)
|
||||
try:
|
||||
return ClientEventsIn.model_validate_json(raw)
|
||||
except ValidationError as exc:
|
||||
raise HTTPException(status_code=422, detail=json.loads(exc.json())) from exc
|
||||
|
||||
|
||||
def _row(item: ClientEvent, user: User, now: datetime, window_start: datetime) -> dict[str, Any]:
|
||||
attrs = logs.redact(item.attrs) if item.attrs else None
|
||||
if attrs is not None:
|
||||
size = len(json.dumps(attrs, ensure_ascii=False, default=str).encode())
|
||||
if size > ATTRS_MAX_BYTES:
|
||||
attrs = {"attrs_dropped_bytes": size}
|
||||
if item.tab_id:
|
||||
attrs = {**(attrs or {}), "tab_id": item.tab_id}
|
||||
ts = item.ts if item.ts.tzinfo else item.ts.replace(tzinfo=timezone.utc)
|
||||
return logsink.to_row(
|
||||
{
|
||||
"ts": min(max(ts, window_start), now),
|
||||
"level": item.level,
|
||||
"sev": logs.SEVERITY[item.level],
|
||||
"source": "web",
|
||||
"logger": "web",
|
||||
"event": item.event,
|
||||
"msg": logs.redact_text(item.msg),
|
||||
"attrs": attrs,
|
||||
"request_id": item.request_id,
|
||||
"user_id": user.id,
|
||||
"version": item.version,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.post("/events")
|
||||
def ingest_events(
|
||||
# The account first: dependencies resolve in order, so an anonymous caller is refused before a
|
||||
# byte of its body is read.
|
||||
user: User = Depends(current_user),
|
||||
body: ClientEventsIn = Depends(_parsed_events),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
level = diagnostics.effective_level(user)
|
||||
if level == diagnostics.BASE:
|
||||
# Lapsed or withdrawn: the client stops streaming when it sees this.
|
||||
raise HTTPException(status_code=409, detail={"code": "diagnostics_off"})
|
||||
if not _batches.allow(str(user.id)):
|
||||
raise HTTPException(status_code=429, detail={"code": "rate_limited"})
|
||||
now = datetime.now(timezone.utc)
|
||||
window_start = now - timedelta(hours=diagnostics.ELEVATION_HOURS)
|
||||
floor = logs.SEVERITY[level]
|
||||
rows = [
|
||||
_row(item, user, now, window_start)
|
||||
for item in body.events
|
||||
if logs.SEVERITY[item.level] >= floor
|
||||
]
|
||||
if not rows:
|
||||
return {"ok": True, "stored": 0}
|
||||
size = sum(len(json.dumps(row, ensure_ascii=False, default=str).encode()) for row in rows)
|
||||
stored = len(rows)
|
||||
if not _stored_bytes.allow(str(user.id), size):
|
||||
stored = 0
|
||||
if not _budget_notices.allow(str(user.id)):
|
||||
return {"ok": True, "stored": 0}
|
||||
rows = [_budget_notice(user, now)]
|
||||
# Under the erasure guard until the commit: an account erased meanwhile gets no rows back.
|
||||
with logs.erasure_guard():
|
||||
db.execute(insert(LogEvent), logsink.without_erased(rows))
|
||||
db.commit()
|
||||
return {"ok": True, "stored": stored}
|
||||
|
||||
|
||||
def _budget_notice(user: User, now: datetime) -> dict[str, Any]:
|
||||
return logsink.to_row(
|
||||
{
|
||||
"ts": now,
|
||||
"level": "WARNING",
|
||||
"sev": logs.SEVERITY["WARNING"],
|
||||
"source": "web",
|
||||
"logger": "web",
|
||||
"event": "diagnostics.budget_spent",
|
||||
# What the budget does, exactly: it is a SLIDING hour, and a refused batch is not
|
||||
# charged to it, so smaller batches keep landing while there is room for them. Saying
|
||||
# "the rest of the hour is not kept" would be a row contradicting the rows beneath it.
|
||||
"msg": "The browser log passed its hourly storage budget; a batch bigger than what is left of it is not kept",
|
||||
"attrs": {"budget_bytes": STORED_BYTES_PER_HOUR},
|
||||
"user_id": user.id,
|
||||
}
|
||||
)
|
||||
@@ -1,5 +1,7 @@
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, RootModel
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -15,7 +17,9 @@ from app.auth import (
|
||||
optional_current_user,
|
||||
purge_user,
|
||||
)
|
||||
from app import diagnostics
|
||||
from app import email as email_mod
|
||||
from app import logs
|
||||
from app import sysconfig
|
||||
from app.db import get_db
|
||||
from app.models import Invite, User
|
||||
@@ -147,6 +151,9 @@ def get_me(
|
||||
"pending_activations": pending_activations,
|
||||
"email_configured": email_configured,
|
||||
"preferences": user.preferences or {},
|
||||
# The consented diagnostics level in force and when it lapses (E22 S5). The client applies
|
||||
# it to its own log and streams that log while it is raised.
|
||||
"diagnostics": diagnostics.state(db, user),
|
||||
}
|
||||
|
||||
|
||||
@@ -188,16 +195,45 @@ def delete_my_account(
|
||||
return {"deleted": True}
|
||||
|
||||
|
||||
class PreferencesIn(RootModel[dict[str, Any]]):
|
||||
"""Free-form UI preferences — a JSON object of any keys (theme, filters, …), merged into the
|
||||
stored ones. A model rather than a bare `dict` parameter so a non-object body is a 422 at the
|
||||
boundary like every other request body."""
|
||||
|
||||
|
||||
@router.put("/preferences")
|
||||
def update_preferences(
|
||||
preferences: dict,
|
||||
preferences: PreferencesIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
# Merge so partial updates don't wipe other keys.
|
||||
merged = dict(user.preferences or {})
|
||||
merged.update(preferences)
|
||||
merged.update(preferences.root)
|
||||
user.preferences = merged
|
||||
db.add(user)
|
||||
db.commit()
|
||||
return {"preferences": merged}
|
||||
|
||||
|
||||
class DiagnosticsIn(BaseModel):
|
||||
level: diagnostics.DiagnosticsLevel
|
||||
|
||||
|
||||
@router.put("/diagnostics")
|
||||
def update_diagnostics(
|
||||
body: DiagnosticsIn,
|
||||
user: User = Depends(current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> dict:
|
||||
"""Raise the diagnostics level (the user consented in Settings → Diagnosztika) or return to the
|
||||
base. The expiry is the server's: a raised level always lapses after
|
||||
`diagnostics.ELEVATION_HOURS`, whatever the client asks. The shared demo account cannot raise
|
||||
it — its log would interleave every visitor's session under one account."""
|
||||
if user.is_demo and body.level != diagnostics.BASE:
|
||||
raise HTTPException(status_code=403, detail={"code": "demo", "reason": "diagnostics"})
|
||||
diagnostics.set_level(user, body.level)
|
||||
db.commit()
|
||||
# The rest of THIS request, and the process's elevation registry, follow the new level now.
|
||||
logs.set_user(user.id, diagnostics.effective_level(user), user.diagnostics_until)
|
||||
return {"ok": True, "diagnostics": diagnostics.state(db, user)}
|
||||
|
||||
@@ -22,6 +22,7 @@ from app.scheduler import (
|
||||
trigger_all,
|
||||
trigger_job,
|
||||
)
|
||||
from app.sync.explore import followed_channel
|
||||
from app.sync.runner import estimate_deep_backfill
|
||||
from app.routes._common import not_found
|
||||
|
||||
@@ -137,9 +138,17 @@ def get_scheduler(_: User = Depends(admin_user), db: Session = Depends(get_db))
|
||||
|
||||
queue = {
|
||||
"channels_recent_pending": db.scalar(
|
||||
select(func.count()).select_from(Channel).where(Channel.recent_synced_at.is_(None))
|
||||
# The same selection run_recent_backfill works through, or the card counts channels
|
||||
# the job will never touch and never reaches zero.
|
||||
select(func.count())
|
||||
.select_from(Channel)
|
||||
.where(Channel.recent_synced_at.is_(None), followed_channel())
|
||||
)
|
||||
or 0,
|
||||
# The whole opted-in BACKLOG, deliberately not `run_deep_backfill`'s next-pass selection
|
||||
# (which also wants `recent_synced_at`): the same figure /api/sync/my-status shows the user,
|
||||
# and a channel missing its recent pull is one the same job tick fetches first, not one the
|
||||
# job will never touch. Narrowing it here would under-report work that is genuinely queued.
|
||||
"channels_deep_pending": deep_eta["channels_pending"],
|
||||
"deep_eta_seconds": deep_eta["eta_seconds"],
|
||||
"videos_pending_enrich": db.scalar(
|
||||
|
||||
@@ -231,9 +231,7 @@ def search_youtube(
|
||||
limit = max(1, min(limit, 100))
|
||||
|
||||
# Source: "scrape" (InnerTube, no API quota) or "api" (search.list, 100 units/page).
|
||||
source = (sysconfig.get_str(db, "search_source") or "scrape").lower()
|
||||
if source != "scrape":
|
||||
source = "api"
|
||||
source = sysconfig.get_str(db, "search_source")
|
||||
|
||||
# Per-user daily cap applies to both sources (anti-abuse); with scrape it's the only guard
|
||||
# since the search itself spends no quota.
|
||||
|
||||
@@ -10,7 +10,7 @@ from typing import Callable
|
||||
from apscheduler.schedulers.background import BackgroundScheduler
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import audit, progress, quota
|
||||
from app import audit, logsink, progress, quota
|
||||
from app.audit import AuditAction
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
@@ -72,6 +72,7 @@ JOB_INTERVALS: dict[str, int] = {
|
||||
"plex_watch_reconcile": settings.plex_watch_reconcile_interval_min,
|
||||
"tmdb_enrich": settings.tmdb_enrich_interval_min,
|
||||
"audit_gc": settings.audit_gc_minutes,
|
||||
"log_gc": settings.log_gc_minutes,
|
||||
}
|
||||
|
||||
# Sane bounds for an admin-set interval (minutes).
|
||||
@@ -381,6 +382,19 @@ def _audit_gc_job() -> None:
|
||||
_job("audit_gc", work)
|
||||
|
||||
|
||||
def _log_gc_job() -> None:
|
||||
# Prune log_events rows older than log_retention_days — the limit the privacy policy states, so
|
||||
# this runs hourly rather than daily. Pure DB, no quota.
|
||||
def work(db):
|
||||
from app import sysconfig
|
||||
|
||||
reaped = logsink.gc(db, sysconfig.get_int(db, "log_retention_days"))
|
||||
db.commit()
|
||||
return {"reaped": reaped}
|
||||
|
||||
_job("log_gc", work)
|
||||
|
||||
|
||||
# job_id -> wrapper. The single source of truth for which jobs exist and how to run one,
|
||||
# shared by start_scheduler (recurring registration) and trigger_job (manual "run now").
|
||||
JOB_FUNCS: dict[str, Callable[[], None]] = {
|
||||
@@ -400,6 +414,7 @@ JOB_FUNCS: dict[str, Callable[[], None]] = {
|
||||
"plex_watch_reconcile": _plex_watch_reconcile_job,
|
||||
"tmdb_enrich": _tmdb_enrich_job,
|
||||
"audit_gc": _audit_gc_job,
|
||||
"log_gc": _log_gc_job,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ grace period (mirrors the maintenance job's grace-then-delete safety)."""
|
||||
import logging
|
||||
from datetime import timedelta
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy import delete, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import sysconfig
|
||||
@@ -31,6 +31,23 @@ from app.youtube.client import YouTubeClient
|
||||
log = logging.getLogger("siftlode.explore")
|
||||
|
||||
|
||||
def followed_channel():
|
||||
"""WHERE clause: a channel somebody follows — a subscription, or an exploration still inside its
|
||||
grace window (`purge_unkept_explored` deletes the expired ExploredChannel rows first). ONE
|
||||
definition for both halves of the question: what the scheduled RSS poll and recent backfill cover
|
||||
(sync/runner.py) and what the explore purge may delete.
|
||||
|
||||
The poll and backfill cover only these (owner decision, 2026-09-17). They used to cover EVERY
|
||||
channel row, but rows also come from search results, playlist sync and dropped subscriptions, and
|
||||
nothing deletes those: on prod 582 of 979 polled channels had no follower, three of them dead
|
||||
feeds answering 404 on every poll. The cost of narrowing is deliberate: the shared Library stops
|
||||
receiving new uploads from channels nobody follows."""
|
||||
return or_(
|
||||
select(Subscription.id).where(Subscription.channel_id == Channel.id).exists(),
|
||||
select(ExploredChannel.id).where(ExploredChannel.channel_id == Channel.id).exists(),
|
||||
)
|
||||
|
||||
|
||||
def _enrich_new(db: Session, yt: YouTubeClient, video_ids: list[str]) -> int:
|
||||
"""Enrich just the not-yet-enriched videos among `video_ids` (videos.list, 1 unit/50)."""
|
||||
if not video_ids:
|
||||
@@ -98,12 +115,8 @@ def purge_unkept_explored(db: Session) -> dict:
|
||||
).rowcount
|
||||
db.commit()
|
||||
|
||||
sub_exists = select(Subscription.id).where(Subscription.channel_id == Channel.id).exists()
|
||||
exp_exists = select(ExploredChannel.id).where(ExploredChannel.channel_id == Channel.id).exists()
|
||||
orphan_ids = (
|
||||
db.execute(
|
||||
select(Channel.id).where(Channel.from_explore.is_(True), ~sub_exists, ~exp_exists)
|
||||
)
|
||||
db.execute(select(Channel.id).where(Channel.from_explore.is_(True), ~followed_channel()))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
|
||||
+59
-17
@@ -7,16 +7,18 @@ public reads.
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from collections import Counter
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import progress, quota, sysconfig
|
||||
from app import logs, progress, quota, sysconfig
|
||||
|
||||
log = logging.getLogger("siftlode.sync")
|
||||
from app.db import SessionLocal
|
||||
from app.models import Channel, OAuthToken, Subscription, User
|
||||
from app.sync.explore import followed_channel
|
||||
from app.sync.subscriptions import import_subscriptions
|
||||
from app.sync.videos import (
|
||||
apply_rss_feed,
|
||||
@@ -32,6 +34,8 @@ from app.youtube.rss import RssError, fetch_channel_feed
|
||||
|
||||
# RSS feeds are free and network-bound, so fetch many at once; DB writes stay single-threaded.
|
||||
RSS_POLL_WORKERS = 16
|
||||
# Failed channels named in a poll's summary warning; the rest are counted by reason.
|
||||
RSS_FAILURE_EXAMPLES = 5
|
||||
|
||||
|
||||
def token_users(db: Session) -> list[User]:
|
||||
@@ -52,7 +56,7 @@ def get_service_user(db: Session) -> User | None:
|
||||
|
||||
|
||||
def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str, int]:
|
||||
"""Poll every channel's free RSS feed. Returns `{"new": inserted, "failed": channels_that_did
|
||||
"""Poll the free RSS feed of every followed channel (`followed_channel`). Returns `{"new": inserted, "failed": channels_that_did
|
||||
_not_complete}` — the failure count is part of the RESULT, not just a log line, so the scheduler
|
||||
card and the audit summary can tell "no new uploads" apart from "nothing worked" (an egress
|
||||
outage, or a database that can't accept writes, otherwise renders identically to a quiet hour).
|
||||
@@ -67,7 +71,7 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
|
||||
carries the FIRST underlying error, because on a small instance "all channels failed" is also
|
||||
what a single dead feed id looks like."""
|
||||
if channels is None:
|
||||
channels = db.execute(select(Channel)).scalars().all()
|
||||
channels = db.execute(select(Channel).where(followed_channel())).scalars().all()
|
||||
total = len(channels)
|
||||
new = 0
|
||||
done = 0
|
||||
@@ -76,6 +80,27 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
|
||||
# outage from one permanently dead feed id — which is the whole difference on an instance with
|
||||
# a single subscription, where BOTH render as 100% failed.
|
||||
first_error: str | None = None
|
||||
# What failed, as counts per reason (`http_404`, `transport`, `apply`, …) plus a few examples —
|
||||
# the ONE warning a poll writes. One line per failed channel was what turned YouTube's RSS outage
|
||||
# of 2026-09-16 into 11k warnings in four hours; each channel's own line is DEBUG now.
|
||||
reasons: Counter[str] = Counter()
|
||||
examples: list[str] = []
|
||||
|
||||
def fail(reason: str, detail: str, *, unexpected: str | None = None) -> None:
|
||||
"""Count a channel that did not get polled. `unexpected` is the message for a failure that
|
||||
is a bug on our side: its FIRST occurrence per reason gets a traceback at ERROR, the rest
|
||||
DEBUG — a read-only database fails every channel the same way, and 979 identical tracebacks
|
||||
tell nobody anything the first one didn't. Call it from inside the `except` block."""
|
||||
nonlocal failed, first_error
|
||||
if unexpected is not None:
|
||||
level = logging.ERROR if reasons[reason] == 0 else logging.DEBUG
|
||||
log.log(level, unexpected, exc_info=True)
|
||||
failed += 1
|
||||
reasons[reason] += 1
|
||||
first_error = first_error or detail
|
||||
if len(examples) < RSS_FAILURE_EXAMPLES:
|
||||
examples.append(detail)
|
||||
|
||||
# Fetch feeds concurrently (network-bound, no DB), then apply inserts on this thread as
|
||||
# each fetch returns — a SQLAlchemy session isn't thread-safe. Pass the channel id (a
|
||||
# plain str) to the workers; the ORM instance is touched only here on the main thread.
|
||||
@@ -90,16 +115,15 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
|
||||
# A failed poll is NOT an empty feed: skip the apply so `last_rss_at` stays where it
|
||||
# was (stamping it made a permanently-broken feed look freshly polled). The shared
|
||||
# counter/progress tail lives after the statement, so the two outcomes can stay two
|
||||
# clauses: an RssError is EXPECTED (404, timeout) and gets one warning line; anything
|
||||
# else is a bug on our side and earns a traceback.
|
||||
# clauses: an RssError is EXPECTED (404, timeout) and is only counted; anything else is a
|
||||
# bug on our side and earns a traceback (the first of its kind per poll).
|
||||
except RssError as exc:
|
||||
log.warning("RSS poll skipped for channel %s: %s", channel.id, exc)
|
||||
failed += 1
|
||||
first_error = first_error or f"{channel.id}: {exc}"
|
||||
log.debug("RSS poll skipped for channel %s: %s", channel.id, exc)
|
||||
fail(exc.reason, f"{channel.id}: {exc}")
|
||||
except Exception as exc:
|
||||
log.exception("RSS fetch failed for channel %s", channel.id)
|
||||
failed += 1
|
||||
first_error = first_error or f"{channel.id}: {exc}"
|
||||
fail(
|
||||
"error", f"{channel.id}: {exc}", unexpected=f"RSS fetch failed for {channel.id}"
|
||||
)
|
||||
else:
|
||||
try:
|
||||
new += apply_rss_feed(db, channel, entries)
|
||||
@@ -107,14 +131,27 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
|
||||
# A write failure (read-only DB, failover) leaves the channel just as un-polled
|
||||
# as an unreachable feed — count it, or a wholly broken run reports "failed=0".
|
||||
db.rollback()
|
||||
failed += 1
|
||||
first_error = first_error or f"{channel.id} (apply): {exc}"
|
||||
log.exception("RSS apply failed for channel %s", channel.id)
|
||||
fail(
|
||||
"apply",
|
||||
f"{channel.id} (apply): {exc}",
|
||||
unexpected=f"RSS apply failed for {channel.id}",
|
||||
)
|
||||
# One place that advances the counter, whatever happened above.
|
||||
done += 1
|
||||
progress.report(done, total, "rss_poll")
|
||||
if failed:
|
||||
log.warning("RSS poll: %d/%d channels could not be polled", failed, total)
|
||||
log.warning(
|
||||
"RSS poll: %d/%d channels could not be polled",
|
||||
failed,
|
||||
total,
|
||||
extra=logs.event(
|
||||
"sync.rss.failed",
|
||||
failed=failed,
|
||||
total=total,
|
||||
reasons=dict(reasons),
|
||||
examples=examples,
|
||||
),
|
||||
)
|
||||
if total and failed == total:
|
||||
# Nothing was polled at all — an egress outage (the tinyproxy-after-reboot incident), a
|
||||
# dead DNS, a read-only database. Returning normally would record the run as `ok` with the
|
||||
@@ -174,7 +211,11 @@ def run_recent_backfill(
|
||||
return {"channels_processed": 0, "videos_added": 0, "reason": "no service user"}
|
||||
if channels is None:
|
||||
channels = (
|
||||
db.execute(select(Channel).where(Channel.recent_synced_at.is_(None))).scalars().all()
|
||||
db.execute(
|
||||
select(Channel).where(Channel.recent_synced_at.is_(None), followed_channel())
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
if max_channels:
|
||||
channels = channels[:max_channels]
|
||||
@@ -204,7 +245,8 @@ def run_deep_backfill(db: Session, max_channels: int = 10, max_pages: int = 5) -
|
||||
if user is None:
|
||||
return {"channels_processed": 0, "videos_added": 0, "reason": "no service user"}
|
||||
# Demand-driven: only deep-backfill channels at least one user has opted into
|
||||
# (Subscription.deep_requested). Recent backfill stays unconditional (cheap).
|
||||
# (Subscription.deep_requested). Recent backfill needs no opt-in: it covers every followed channel
|
||||
# (`followed_channel`).
|
||||
deep_requested = (
|
||||
select(Subscription.id)
|
||||
.where(
|
||||
|
||||
+48
-19
@@ -14,7 +14,7 @@ from dataclasses import dataclass
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app import security
|
||||
from app.config import settings
|
||||
from app.config import Settings, canonical_choice, choices_of, settings
|
||||
from app.models import SystemConfig
|
||||
|
||||
log = logging.getLogger("siftlode.config")
|
||||
@@ -27,7 +27,7 @@ _decrypt_warned: set[str] = set()
|
||||
@dataclass(frozen=True)
|
||||
class ConfigSpec:
|
||||
key: str
|
||||
type: str # "int" | "str" | "bool"
|
||||
type: str # "int" | "str" | "bool" | "choice"
|
||||
group: str # admin-UI grouping (logically related keys stay together)
|
||||
default_attr: str # attribute on `settings` holding the env/config default
|
||||
secret: bool = False
|
||||
@@ -35,16 +35,33 @@ class ConfigSpec:
|
||||
max: int | None = None
|
||||
allow_empty: bool = False # str keys: a stored "" is an explicit empty value, not "unset"
|
||||
|
||||
@property
|
||||
def choices(self) -> tuple[str, ...]:
|
||||
"""A choice key's allowed values, read from its setting's Literal type in config.py — the one
|
||||
place the list is written. The admin page renders them as a picker."""
|
||||
return choices_of(Settings.model_fields[self.default_attr].annotation)
|
||||
|
||||
|
||||
# Registry of DB-overridable keys. Add a key here + wire its read through get_*() to move it
|
||||
# off env into the admin UI; the admin page renders it automatically from this list.
|
||||
# off env into the admin UI; the admin page renders it automatically from this list. The ORDER is the
|
||||
# admin page's tab order (first appearance of each group), so keep a group's keys together.
|
||||
SPECS: tuple[ConfigSpec, ...] = (
|
||||
# --- Access / registration ---
|
||||
ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
|
||||
# --- Email / SMTP (one logical group; the password is the only secret) ---
|
||||
ConfigSpec("smtp_host", "str", "email", "smtp_host"),
|
||||
ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535),
|
||||
ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True),
|
||||
ConfigSpec("smtp_from", "str", "email", "smtp_from"),
|
||||
ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True),
|
||||
# --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
|
||||
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
|
||||
# Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
|
||||
ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
|
||||
# --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
|
||||
# install wizard / Configuration page so no restart is needed when the creds change. ---
|
||||
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
|
||||
ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
|
||||
# --- Quota (shared YouTube Data API daily budget) ---
|
||||
ConfigSpec("quota_daily_budget", "int", "quota", "quota_daily_budget", min=1, max=1_000_000),
|
||||
ConfigSpec(
|
||||
@@ -59,7 +76,7 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
max=100_000,
|
||||
),
|
||||
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
|
||||
ConfigSpec("search_source", "str", "quota", "search_source"),
|
||||
ConfigSpec("search_source", "choice", "quota", "search_source"),
|
||||
# --- Backfill (recent-first first pass per channel) ---
|
||||
ConfigSpec(
|
||||
"backfill_recent_max_videos",
|
||||
@@ -96,16 +113,6 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
ConfigSpec("explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650),
|
||||
# Days an un-kept live-search result video is kept before the cleanup job removes it.
|
||||
ConfigSpec("search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650),
|
||||
# --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
|
||||
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
|
||||
# Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
|
||||
ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
|
||||
# --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
|
||||
# install wizard / Configuration page so no restart is needed when the creds change. ---
|
||||
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
|
||||
ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
|
||||
# --- Access / registration ---
|
||||
ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
|
||||
# --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
|
||||
ConfigSpec("download_total_max_bytes", "int", "downloads", "download_total_max_bytes", min=0),
|
||||
ConfigSpec(
|
||||
@@ -137,7 +144,7 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
ConfigSpec(
|
||||
"download_gc_grace_days", "int", "downloads", "download_gc_grace_days", min=0, max=3_650
|
||||
),
|
||||
ConfigSpec("download_layout", "str", "downloads", "download_layout"),
|
||||
ConfigSpec("download_layout", "choice", "downloads", "download_layout"),
|
||||
ConfigSpec(
|
||||
"download_worker_concurrency",
|
||||
"int",
|
||||
@@ -150,6 +157,13 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
# --- Audit log ---
|
||||
# Days to keep admin audit-log rows before the audit_gc job prunes them (0 = keep forever).
|
||||
ConfigSpec("audit_retention_days", "int", "audit", "audit_retention_days", min=0, max=3650),
|
||||
# --- Diagnostics (E22) ---
|
||||
# Runtime log level for the API process. Applied without a restart — see routes/config.py and
|
||||
# main.py's lifespan.
|
||||
ConfigSpec("log_level", "choice", "diagnostics", "log_level"),
|
||||
# Days the log_events table keeps a row (the log_gc job prunes). No "0 = forever": stored logs
|
||||
# carry account-linked request history, and the privacy policy names a limit.
|
||||
ConfigSpec("log_retention_days", "int", "diagnostics", "log_retention_days", min=1, max=90),
|
||||
# --- Plex integration (optional; local-file playback + Plex metadata) ---
|
||||
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled"),
|
||||
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url"),
|
||||
@@ -159,10 +173,10 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
|
||||
# (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
|
||||
ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
|
||||
# Transcode re-encode tuning (E6 S1). hw = "auto"|"on"|"off" (free-text, like search_source); the
|
||||
# libx264 knobs apply only to the software fallback. crf 0–51 (x264 range); max_height 0 = source.
|
||||
ConfigSpec("plex_transcode_hw", "str", "plex", "plex_transcode_hw"),
|
||||
ConfigSpec("plex_transcode_preset", "str", "plex", "plex_transcode_preset"),
|
||||
# Transcode re-encode tuning (E6 S1). The libx264 knobs apply only to the software fallback.
|
||||
# crf 0–51 (x264 range); max_height 0 = source.
|
||||
ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw"),
|
||||
ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset"),
|
||||
ConfigSpec("plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51),
|
||||
ConfigSpec(
|
||||
"plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
|
||||
@@ -194,11 +208,23 @@ def _default(s: ConfigSpec):
|
||||
return getattr(settings, s.default_attr)
|
||||
|
||||
|
||||
def _choice(s: ConfigSpec, raw: object) -> str:
|
||||
"""The canonical spelling of `raw` among the key's choices (any letter case), or ValueError."""
|
||||
option = canonical_choice(s.choices, raw)
|
||||
if option is None:
|
||||
raise ValueError(f"{s.key} must be one of: {', '.join(s.choices)}")
|
||||
return option
|
||||
|
||||
|
||||
def _coerce(s: ConfigSpec, raw: str):
|
||||
if s.type == "int":
|
||||
return int(raw)
|
||||
if s.type == "bool":
|
||||
return raw == "true"
|
||||
if s.type == "choice":
|
||||
# A stored value that is no longer a choice (the list shrank) reads as the default, like a
|
||||
# non-numeric int override does — see get().
|
||||
return _choice(s, raw)
|
||||
return raw
|
||||
|
||||
|
||||
@@ -274,6 +300,8 @@ def set_value(db: Session, key: str, value) -> None:
|
||||
raw = str(v)
|
||||
elif s.type == "bool":
|
||||
raw = "true" if value else "false"
|
||||
elif s.type == "choice":
|
||||
raw = _choice(s, value)
|
||||
else:
|
||||
raw = "" if value is None else str(value)
|
||||
|
||||
@@ -316,6 +344,7 @@ def describe(db: Session) -> dict:
|
||||
"max": s.max,
|
||||
"is_set": is_set, # an override row exists
|
||||
"allow_empty": s.allow_empty, # if true, the UI stores "" rather than deleting the row
|
||||
"choices": list(s.choices), # the picker's options; [] for any other type
|
||||
}
|
||||
if s.secret:
|
||||
item["default_is_set"] = bool(_default(s))
|
||||
|
||||
+17
-6
@@ -28,6 +28,8 @@ from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from app import logs
|
||||
from app import logsink
|
||||
from app import sysconfig
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
@@ -491,15 +493,20 @@ def _worker_loop(idx: int) -> None:
|
||||
if job_id is None:
|
||||
_stop.wait(settings.download_poll_seconds)
|
||||
continue
|
||||
try:
|
||||
_process_job(job_id)
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("processing job %s crashed", job_id)
|
||||
_fail_running(job_id, error="Worker error", error_code=None)
|
||||
# Tag every line this job produces (including the crash below) with the job id: a worker runs
|
||||
# N jobs at once on N threads, so without it the interleaved output is unreadable. The bind
|
||||
# wraps the handler too — bound inside the `try`, it was already reset by the time the crash
|
||||
# line was logged.
|
||||
with logs.bind(job_id=job_id):
|
||||
try:
|
||||
_process_job(job_id)
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("processing job %s crashed", job_id)
|
||||
_fail_running(job_id, error="Worker error", error_code=None)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logs.configure_logging("worker")
|
||||
signal.signal(signal.SIGTERM, _handle_signal)
|
||||
signal.signal(signal.SIGINT, _handle_signal)
|
||||
|
||||
@@ -515,6 +522,10 @@ def main() -> None:
|
||||
if _stop.is_set():
|
||||
return # asked to shut down while waiting — that IS a clean exit
|
||||
raise SystemExit(1)
|
||||
# After the wait, not before: the DB is reachable now. The wait checks `download_jobs`, not
|
||||
# `log_events`, so on the deploy that first adds the table the worker may start before the API
|
||||
# has migrated — those few seconds of batches fail and say so on stderr; nothing else breaks.
|
||||
logsink.start()
|
||||
_recover_orphans()
|
||||
n = max(1, settings.download_worker_concurrency)
|
||||
log.info("Download worker started (concurrency=%d, root=%s).", n, settings.download_root)
|
||||
|
||||
@@ -14,7 +14,14 @@ class RssError(Exception):
|
||||
|
||||
Deliberately NOT the same as an empty feed: a failed poll used to return `[]`, which the
|
||||
caller stamped as a successful `last_rss_at` — so a channel whose feed 404s permanently
|
||||
(deleted, renamed id) looked like it was being polled fine forever, and nothing surfaced it."""
|
||||
(deleted, renamed id) looked like it was being polled fine forever, and nothing surfaced it.
|
||||
|
||||
`reason` is a short stable code (`http_404`, `transport`) so a poll over many channels can report
|
||||
WHAT failed as counts instead of one line per channel."""
|
||||
|
||||
def __init__(self, message: str, reason: str):
|
||||
super().__init__(message)
|
||||
self.reason = reason
|
||||
|
||||
|
||||
def fetch_channel_feed(channel_id: str) -> list[dict]:
|
||||
@@ -24,9 +31,12 @@ def fetch_channel_feed(channel_id: str) -> list[dict]:
|
||||
try:
|
||||
resp = httpx.get(url, timeout=20.0, headers={"User-Agent": "Siftlode/1.0"})
|
||||
except httpx.HTTPError as exc:
|
||||
raise RssError(f"RSS fetch failed for {channel_id}: {exc}") from exc
|
||||
raise RssError(f"RSS fetch failed for {channel_id}: {exc}", "transport") from exc
|
||||
if resp.status_code != 200:
|
||||
raise RssError(f"RSS fetch for {channel_id} returned HTTP {resp.status_code}")
|
||||
raise RssError(
|
||||
f"RSS fetch for {channel_id} returned HTTP {resp.status_code}",
|
||||
f"http_{resp.status_code}",
|
||||
)
|
||||
|
||||
parsed = feedparser.parse(resp.content)
|
||||
out: list[dict] = []
|
||||
|
||||
@@ -15,4 +15,8 @@ echo "Starting Siftlode API..."
|
||||
# X-Forwarded-For only when the TRUE socket peer is a configured proxy (TRUSTED_PROXY_IPS); those
|
||||
# flags would make uvicorn overwrite request.client from the forgeable XFF, defeating that check and
|
||||
# reopening the rate-limit bypass (SA3).
|
||||
exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --log-config log_config.json --timeout-keep-alive 75
|
||||
#
|
||||
# No --log-config: app/logs.py configures logging for every Siftlode process (api, worker, download
|
||||
# child, alembic) in one place, and it silences uvicorn's access log in favour of our own
|
||||
# `http.request` line (route template, no query string, request id).
|
||||
exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --timeout-keep-alive 75
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
{
|
||||
"version": 1,
|
||||
"disable_existing_loggers": false,
|
||||
"formatters": {
|
||||
"default": {
|
||||
"()": "uvicorn.logging.DefaultFormatter",
|
||||
"fmt": "%(asctime)s %(levelprefix)s [%(name)s] %(message)s"
|
||||
},
|
||||
"access": {
|
||||
"()": "uvicorn.logging.AccessFormatter",
|
||||
"fmt": "%(asctime)s %(levelprefix)s %(client_addr)s - \"%(request_line)s\" %(status_code)s"
|
||||
},
|
||||
"plain": {
|
||||
"format": "%(asctime)s %(levelname)-5s [%(name)s] %(message)s"
|
||||
}
|
||||
},
|
||||
"handlers": {
|
||||
"default": {
|
||||
"class": "logging.StreamHandler",
|
||||
"formatter": "default",
|
||||
"stream": "ext://sys.stdout"
|
||||
},
|
||||
"access": {
|
||||
"class": "logging.StreamHandler",
|
||||
"formatter": "access",
|
||||
"stream": "ext://sys.stdout"
|
||||
},
|
||||
"plain": {
|
||||
"class": "logging.StreamHandler",
|
||||
"formatter": "plain",
|
||||
"stream": "ext://sys.stdout"
|
||||
}
|
||||
},
|
||||
"loggers": {
|
||||
"uvicorn": { "handlers": ["default"], "level": "INFO", "propagate": false },
|
||||
"uvicorn.error": { "handlers": ["default"], "level": "INFO", "propagate": false },
|
||||
"uvicorn.access": { "handlers": ["access"], "level": "INFO", "propagate": false },
|
||||
"siftlode": { "handlers": ["plain"], "level": "INFO", "propagate": false }
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,9 @@
|
||||
# Test + lint tooling — NEVER in the prod image. Installed only by backend/Dockerfile.test (the
|
||||
# throwaway test image), which the prod build (root Dockerfile) never references. Pinned so the
|
||||
# suite is reproducible: pytest for the pure-logic tests, and ruff at the SAME version the gate's
|
||||
# backend lint lane currently runs on the host (see the run_lint note in siftlode.sh). httpx is
|
||||
# already an app dependency (requirements.txt), so its TestClient needs nothing extra here.
|
||||
# backend lint lane runs on the host (the gate fails when the host's ruff is not this one). httpx2:
|
||||
# Starlette 1.x's TestClient is built on it (`import httpx2 as httpx`) and only falls back to the
|
||||
# app's own httpx with a deprecation warning on every run.
|
||||
pytest==8.4.2
|
||||
ruff==0.15.21
|
||||
httpx2==2.13.0
|
||||
|
||||
@@ -57,6 +57,10 @@ def _db_reachable() -> bool:
|
||||
|
||||
_DB_REACHABLE = _db_reachable()
|
||||
|
||||
# No background log writer in tests: every TestClient would otherwise start one on app startup and
|
||||
# write into tables the fixtures truncate between tests. The sink's own tests drive it explicitly.
|
||||
settings.log_db_sink = False
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def _schema():
|
||||
@@ -210,3 +214,15 @@ def _reset_rate_limiters():
|
||||
for obj in vars(mod).values():
|
||||
if isinstance(obj, RateLimiter):
|
||||
obj.reset()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_log_erasures():
|
||||
"""`logs._erased` is module-level and remembers an erased account for an hour of monotonic time
|
||||
— longer than any suite run — while the `db` fixture hands serial ids back out from 1 via
|
||||
TRUNCATE … RESTART IDENTITY. So a test that purges user 1 would otherwise make a LATER test's
|
||||
rows for user 1 store with `user_id = None`, failing somewhere nowhere near the cause and only
|
||||
in some file orders. Same reasoning as the limiter sweep above, different global."""
|
||||
from app import logs
|
||||
|
||||
logs._erased.clear()
|
||||
|
||||
+92
@@ -0,0 +1,92 @@
|
||||
{
|
||||
"_comment": "One redaction contract for both logs: backend/app/logs.py redact() and frontend/src/lib/log.ts redact() must turn every `value` (under `key`) into `expected`. Both test suites run this file (backend tests/test_logs.py, frontend src/lib/log.contract.test.ts), so the two implementations cannot drift apart. Some cases prove a secret goes; others prove ordinary content (a search term, a Plex title, a chat line) is not mangled — what a LEVEL may carry is decided at the call site, not here.",
|
||||
"cases": [
|
||||
{
|
||||
"name": "a password field is dropped whole",
|
||||
"key": "",
|
||||
"value": {
|
||||
"email": "a@b.co",
|
||||
"password": "hunter2",
|
||||
"current_password": "x"
|
||||
},
|
||||
"expected": {
|
||||
"email": "[email]",
|
||||
"password": "[redacted]",
|
||||
"current_password": "[redacted]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "a token under any spelling of the key, nested and in lists",
|
||||
"key": "",
|
||||
"value": {
|
||||
"plex": { "plex_server_token": "abc", "Authorization": "Bearer z" },
|
||||
"items": [{ "access_token": "t1" }, { "id": 3 }]
|
||||
},
|
||||
"expected": {
|
||||
"plex": {
|
||||
"plex_server_token": "[redacted]",
|
||||
"Authorization": "[redacted]"
|
||||
},
|
||||
"items": [{ "access_token": "[redacted]" }, { "id": 3 }]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "E2EE material and cookies",
|
||||
"key": "",
|
||||
"value": {
|
||||
"ciphertext": "AAAA",
|
||||
"passphrase": "p",
|
||||
"private_key": "k",
|
||||
"cookies": "SID=1"
|
||||
},
|
||||
"expected": {
|
||||
"ciphertext": "[redacted]",
|
||||
"passphrase": "[redacted]",
|
||||
"private_key": "[redacted]",
|
||||
"cookies": "[redacted]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "a reset link and a verify fragment",
|
||||
"key": "msg",
|
||||
"value": "open https://app.example/#reset=QWErty123 or /?verify=abc&lang=hu",
|
||||
"expected": "open https://app.example/#reset=[redacted] or /?verify=[redacted]&lang=hu"
|
||||
},
|
||||
{
|
||||
"name": "an OAuth callback URL",
|
||||
"key": "url",
|
||||
"value": "/auth/callback?code=4/0Ab-xyz&state=s3cr3t&scope=email",
|
||||
"expected": "/auth/callback?code=[redacted]&state=[redacted]&scope=email"
|
||||
},
|
||||
{
|
||||
"name": "an email address inside a sentence",
|
||||
"key": "message",
|
||||
"value": "Invitation sent to Some.One+tag@mail.example.org.",
|
||||
"expected": "Invitation sent to [email]."
|
||||
},
|
||||
{
|
||||
"name": "a search term is kept (TRACE decides whether it is logged at all)",
|
||||
"key": "q",
|
||||
"value": "lo-fi beats to relax",
|
||||
"expected": "lo-fi beats to relax"
|
||||
},
|
||||
{
|
||||
"name": "a Plex title with an at-sign is not an address",
|
||||
"key": "title",
|
||||
"value": "Live @ Wembley 1986",
|
||||
"expected": "Live @ Wembley 1986"
|
||||
},
|
||||
{
|
||||
"name": "a chat line is text like any other; its only secret part is scrubbed",
|
||||
"key": "body",
|
||||
"value": "see you at 8, my mail is me@home.example",
|
||||
"expected": "see you at 8, my mail is [email]"
|
||||
},
|
||||
{
|
||||
"name": "non-text values pass through",
|
||||
"key": "",
|
||||
"value": { "status": 404, "ok": false, "ratio": 0.5, "none": null },
|
||||
"expected": { "status": 404, "ok": false, "ratio": 0.5, "none": null }
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -7,14 +7,14 @@ of the caller's already-translated fallback, so from that commit on the demo use
|
||||
this gate is UI-gated on `is_demo` either — the comfort shelf is not, so the pin button is live.
|
||||
|
||||
A structured `{code: "demo"}` is what `localizeDetail` can turn into `errors.demo` in both locales.
|
||||
Pinned at all FOUR sites: the shared dependency, the two auth routes that hand-roll the same check,
|
||||
and the account-delete route in `routes/me.py` — which round 6 missed while its docstring claimed
|
||||
Pinned at every site: the shared dependency, the two auth routes that hand-roll the same check,
|
||||
the account-delete route in `routes/me.py` — which round 6 missed while its docstring claimed
|
||||
all the sites were covered, so the next reader had no reason to look (round 7). An asymmetric fix is
|
||||
the smell this repo bans, and a docstring asserting symmetry it does not have is worse than none.
|
||||
|
||||
The list is derived from the source, not typed out: `test_every_is_demo_refusal_is_on_the_list`
|
||||
below fails when a fifth `is_demo` 403 appears anywhere in `app/`, so this file cannot fall behind
|
||||
the code the way round 6's did. Needs the DB.
|
||||
the code the way round 6's did. (E22 S5 added the fifth: raising the diagnostics level.) Needs the DB.
|
||||
"""
|
||||
|
||||
import re
|
||||
@@ -33,6 +33,8 @@ def test_every_demo_refusal_carries_the_same_structured_reason(db, make_user, as
|
||||
("post", "/auth/logout-others", None),
|
||||
("post", "/auth/set-password", {"password": "a long enough password"}),
|
||||
("delete", "/api/me/account", None),
|
||||
# E22 S5: the shared account's log would interleave every visitor's session.
|
||||
("put", "/api/me/diagnostics", {"level": "DEBUG"}),
|
||||
]:
|
||||
r = getattr(client, method)(url, **({"json": body} if body is not None else {}))
|
||||
assert r.status_code == 403, (url, r.text)
|
||||
@@ -55,9 +57,9 @@ def test_every_is_demo_refusal_is_on_the_list():
|
||||
for path in sorted(app_dir.rglob("*.py")):
|
||||
lines = path.read_text(encoding="utf-8").splitlines()
|
||||
for i, line in enumerate(lines):
|
||||
# `if user.is_demo:` / `if u.is_demo:` — the hand-rolled gate, not the ORM filters
|
||||
# (`User.is_demo.is_(False)`) and not the column definition.
|
||||
if not re.search(r"^\s*if\s+\w+\.is_demo\s*:", line):
|
||||
# `if user.is_demo:` / `if u.is_demo and …:` — the hand-rolled gate, not the ORM
|
||||
# filters (`User.is_demo.is_(False)`) and not the column definition.
|
||||
if not re.search(r"^\s*if\s+\w+\.is_demo\b[^:]*:", line):
|
||||
continue
|
||||
block = "\n".join(lines[i : i + 6])
|
||||
if "403" not in block:
|
||||
@@ -67,9 +69,9 @@ def test_every_is_demo_refusal_is_on_the_list():
|
||||
f"{path.relative_to(app_dir)}:{i + 1} refuses the demo account with an "
|
||||
f"unstructured detail — the frontend cannot localize it:\n{block}"
|
||||
)
|
||||
# auth.py's shared gate + routes/auth.py's two + routes/me.py's delete. A new one is welcome,
|
||||
# but it has to be added to the loop above in the same commit.
|
||||
assert len(sites) == 4, sites
|
||||
# auth.py's shared gate + routes/auth.py's two + routes/me.py's delete and diagnostics level. A
|
||||
# new one is welcome, but it has to be added to the loop above in the same commit.
|
||||
assert len(sites) == 5, sites
|
||||
|
||||
|
||||
def test_a_real_account_is_not_refused_by_the_gate(db, make_user, as_user):
|
||||
|
||||
@@ -0,0 +1,492 @@
|
||||
"""E22 S5 — a user's consented diagnostics level, and the browser log it lets in.
|
||||
|
||||
Three parts:
|
||||
* the level itself: the server computes its expiry, a lapsed one reads as the base, the demo
|
||||
account cannot raise it (DB lane, through the real routes);
|
||||
* `POST /api/diagnostics/events`: refused at the base level, and nothing in a batch is trusted —
|
||||
the account comes from the session, everything is redacted again, events under the consented
|
||||
level are dropped, sizes and pace are capped (DB lane);
|
||||
* the server side of the elevation: ONE user's requests are logged in detail while everyone
|
||||
else's stay at the base level, and the account reaches a handler's own lines at all (pure, an
|
||||
in-process app).
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends, FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import diagnostics, logs, logsink, sysconfig
|
||||
from app.config import settings
|
||||
from app.models import LogEvent, User
|
||||
from app.routes import diagnostics as diagnostics_routes
|
||||
|
||||
_PW = "correct horse battery"
|
||||
|
||||
|
||||
# --- the level ------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_raised_level_counts_until_it_lapses_and_not_a_moment_longer():
|
||||
now = datetime.now(timezone.utc)
|
||||
assert diagnostics.effective_level(User()) == "INFO"
|
||||
raised = User(diagnostics_level="TRACE", diagnostics_until=now + timedelta(minutes=1))
|
||||
assert diagnostics.effective_level(raised, now) == "TRACE"
|
||||
assert diagnostics.effective_level(raised, now + timedelta(minutes=2)) == "INFO"
|
||||
# A level without an expiry is not a level that never ends — it is no level at all.
|
||||
assert diagnostics.effective_level(User(diagnostics_level="DEBUG")) == "INFO"
|
||||
|
||||
|
||||
def _signed_in(client, make_user, **kw):
|
||||
"""A real sign-in: `/api/me` resolves through `optional_current_user`, which the `as_user`
|
||||
override does not cover (see test_me_email_flag)."""
|
||||
user = make_user(email="diag@example.com", password=_PW, **kw)
|
||||
r = client.post("/auth/password-login", json={"email": "diag@example.com", "password": _PW})
|
||||
assert r.status_code == 200, r.text
|
||||
return user
|
||||
|
||||
|
||||
def test_raising_the_level_sets_the_servers_expiry_whatever_the_client_sends(db, client, make_user):
|
||||
user = _signed_in(client, make_user)
|
||||
r = client.put("/api/me/diagnostics", json={"level": "DEBUG", "until": "2099-01-01T00:00:00Z"})
|
||||
assert r.status_code == 200, r.text
|
||||
db.refresh(user)
|
||||
expected = datetime.now(timezone.utc) + timedelta(hours=diagnostics.ELEVATION_HOURS)
|
||||
assert abs((user.diagnostics_until - expected).total_seconds()) < 60
|
||||
assert r.json()["diagnostics"]["level"] == "DEBUG"
|
||||
assert client.get("/api/me").json()["diagnostics"]["level"] == "DEBUG"
|
||||
|
||||
assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
|
||||
db.refresh(user)
|
||||
assert (user.diagnostics_level, user.diagnostics_until) == (None, None)
|
||||
assert client.get("/api/me").json()["diagnostics"] == {
|
||||
"level": "INFO",
|
||||
"until": None,
|
||||
"retention_days": 7,
|
||||
}
|
||||
|
||||
|
||||
def test_a_lapsed_level_reads_as_the_base_in_me(db, client, make_user):
|
||||
_signed_in(
|
||||
client,
|
||||
make_user,
|
||||
diagnostics_level="TRACE",
|
||||
diagnostics_until=datetime.now(timezone.utc) - timedelta(minutes=1),
|
||||
)
|
||||
assert client.get("/api/me").json()["diagnostics"] == {
|
||||
"level": "INFO",
|
||||
"until": None,
|
||||
"retention_days": 7,
|
||||
}
|
||||
|
||||
|
||||
def test_only_the_three_levels_can_be_chosen(db, make_user, as_user):
|
||||
client = as_user(make_user())
|
||||
for bogus in ("WARNING", "debug", "EVERYTHING"):
|
||||
assert client.put("/api/me/diagnostics", json={"level": bogus}).status_code == 422
|
||||
|
||||
|
||||
def test_the_demo_account_can_return_to_the_base_but_never_raise_it(db, make_user, as_user):
|
||||
client = as_user(make_user(is_demo=True))
|
||||
r = client.put("/api/me/diagnostics", json={"level": "TRACE"})
|
||||
assert r.status_code == 403 and r.json()["detail"]["code"] == "demo"
|
||||
assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
|
||||
|
||||
|
||||
def test_preferences_must_be_a_json_object(db, make_user, as_user):
|
||||
client = as_user(make_user())
|
||||
assert client.put("/api/me/preferences", json=["not", "an", "object"]).status_code == 422
|
||||
r = client.put("/api/me/preferences", json={"theme": "dark"})
|
||||
assert r.status_code == 200 and r.json()["preferences"]["theme"] == "dark"
|
||||
|
||||
|
||||
# --- the browser's batches ------------------------------------------------------------------
|
||||
|
||||
|
||||
def _event(**over) -> dict:
|
||||
return {
|
||||
"ts": datetime.now(timezone.utc).isoformat(),
|
||||
"level": "INFO",
|
||||
"event": "nav.view",
|
||||
"msg": "view",
|
||||
"tab_id": "a1b2c3",
|
||||
"version": "0.88.0",
|
||||
**over,
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def elevated(db, make_user, as_user):
|
||||
"""A signed-in client whose user consented to `level` a minute ago, and a reader for its rows."""
|
||||
diagnostics_routes._batches.reset()
|
||||
|
||||
def _make(level: str = "DEBUG"):
|
||||
user = make_user(
|
||||
diagnostics_level=level,
|
||||
diagnostics_until=datetime.now(timezone.utc) + timedelta(hours=23),
|
||||
)
|
||||
|
||||
def rows() -> list[LogEvent]:
|
||||
db.expire_all()
|
||||
return list(db.execute(select(LogEvent).order_by(LogEvent.id)).scalars())
|
||||
|
||||
return user, as_user(user), rows
|
||||
|
||||
yield _make
|
||||
diagnostics_routes._batches.reset()
|
||||
|
||||
|
||||
def test_nothing_is_accepted_at_the_base_level(db, make_user, as_user):
|
||||
client = as_user(make_user())
|
||||
r = client.post("/api/diagnostics/events", json={"events": [_event()]})
|
||||
assert r.status_code == 409 and r.json()["detail"]["code"] == "diagnostics_off"
|
||||
assert not db.execute(select(LogEvent)).first()
|
||||
|
||||
|
||||
def test_an_anonymous_caller_is_refused(client):
|
||||
assert client.post("/api/diagnostics/events", json={"events": [_event()]}).status_code == 401
|
||||
|
||||
|
||||
def test_a_batch_is_stored_as_the_sessions_web_events_and_redacted_again(elevated):
|
||||
user, client, rows = elevated("DEBUG")
|
||||
r = client.post(
|
||||
"/api/diagnostics/events",
|
||||
json={
|
||||
"events": [
|
||||
_event(
|
||||
level="WARNING",
|
||||
event="ui.notify",
|
||||
msg="sent to someone@example.com",
|
||||
attrs={"password": "hunter2", "url": "/?token=abc123", "user_id": 999},
|
||||
request_id="0123456789abcdef",
|
||||
),
|
||||
_event(level="DEBUG", event="ui.click", attrs={"control": "feed.refresh"}),
|
||||
],
|
||||
"user_id": 999, # ignored: the account is the session's
|
||||
},
|
||||
)
|
||||
assert r.json() == {"ok": True, "stored": 2}
|
||||
warning, click = rows()
|
||||
assert (warning.source, warning.logger, warning.user_id) == ("web", "web", user.id)
|
||||
assert warning.msg == "sent to [email]"
|
||||
assert warning.attrs["password"] == logs.REDACTED
|
||||
assert "abc123" not in json.dumps(warning.attrs)
|
||||
assert warning.attrs["tab_id"] == "a1b2c3"
|
||||
assert warning.request_id == "0123456789abcdef"
|
||||
assert (warning.sev, click.level, click.version) == (13, "DEBUG", "0.88.0")
|
||||
|
||||
|
||||
def test_an_erased_accounts_batch_is_stored_without_the_account(elevated):
|
||||
"""The sink's writer is not the only path into `log_events` — this route inserts too, and an
|
||||
account erased while its browser was mid-flush must not get its rows back through it. Faked at
|
||||
the `forget_user` seam the purge itself calls: once the User row is actually gone the session
|
||||
no longer resolves, so the reachable window is exactly a request already past `current_user`.
|
||||
(`logs._erased` is cleared before every test by conftest's `_reset_log_erasures`.)"""
|
||||
user, client, rows = elevated("DEBUG")
|
||||
logs.forget_user(user.id)
|
||||
r = client.post("/api/diagnostics/events", json={"events": [_event(level="WARNING")]})
|
||||
assert r.json() == {"ok": True, "stored": 1}
|
||||
(stored,) = rows()
|
||||
assert (stored.source, stored.user_id) == ("web", None)
|
||||
|
||||
|
||||
def test_an_event_under_the_consented_level_is_dropped(elevated):
|
||||
_user, client, rows = elevated("DEBUG")
|
||||
r = client.post(
|
||||
"/api/diagnostics/events",
|
||||
json={"events": [_event(level="TRACE", msg="typed: lofi"), _event(level="DEBUG")]},
|
||||
)
|
||||
assert r.json()["stored"] == 1
|
||||
assert [row.level for row in rows()] == ["DEBUG"]
|
||||
|
||||
|
||||
def test_a_timestamp_outside_the_consent_window_is_clamped(elevated):
|
||||
_user, client, rows = elevated("DEBUG")
|
||||
now = datetime.now(timezone.utc)
|
||||
future = (now + timedelta(days=365)).isoformat()
|
||||
past = (now - timedelta(days=30)).isoformat()
|
||||
client.post("/api/diagnostics/events", json={"events": [_event(ts=future), _event(ts=past)]})
|
||||
late, early = rows()
|
||||
assert late.ts <= datetime.now(timezone.utc)
|
||||
assert early.ts >= now - timedelta(hours=diagnostics.ELEVATION_HOURS, minutes=1)
|
||||
|
||||
|
||||
def test_sizes_are_capped(elevated):
|
||||
_user, client, rows = elevated("DEBUG")
|
||||
too_many = [_event() for _ in range(diagnostics_routes.EVENTS_MAX + 1)]
|
||||
assert client.post("/api/diagnostics/events", json={"events": too_many}).status_code == 422
|
||||
|
||||
huge = "x" * (diagnostics_routes.BODY_MAX_BYTES + 1)
|
||||
r = client.post(
|
||||
"/api/diagnostics/events",
|
||||
content=json.dumps({"events": [_event(attrs={"blob": huge})]}),
|
||||
headers={"content-type": "application/json"},
|
||||
)
|
||||
assert r.status_code == 413
|
||||
|
||||
big_attrs = {"blob": "y" * (diagnostics_routes.ATTRS_MAX_BYTES + 10)}
|
||||
assert (
|
||||
client.post("/api/diagnostics/events", json={"events": [_event(attrs=big_attrs)]}).json()[
|
||||
"stored"
|
||||
]
|
||||
== 1
|
||||
)
|
||||
(row,) = rows()
|
||||
assert "blob" not in row.attrs and row.attrs["attrs_dropped_bytes"] > 0
|
||||
|
||||
bad_code = client.post("/api/diagnostics/events", json={"events": [_event(event="Nav View")]})
|
||||
assert bad_code.status_code == 422
|
||||
|
||||
|
||||
def test_the_pace_is_capped_per_user(elevated):
|
||||
_user, client, _rows = elevated("DEBUG")
|
||||
codes = [
|
||||
client.post("/api/diagnostics/events", json={"events": []}).status_code for _ in range(31)
|
||||
]
|
||||
assert codes[:30] == [200] * 30 and codes[30] == 429
|
||||
|
||||
|
||||
def test_what_an_account_stores_per_hour_is_capped_and_said_once(elevated, monkeypatch):
|
||||
user, client, rows = elevated("DEBUG")
|
||||
monkeypatch.setattr(
|
||||
diagnostics_routes, "_stored_bytes", diagnostics_routes.RateLimiter(3_000, 3600)
|
||||
)
|
||||
batch = {"events": [_event(msg="m" * 1_000)]}
|
||||
stored = [client.post("/api/diagnostics/events", json=batch).json()["stored"] for _ in range(4)]
|
||||
assert stored == [1, 1, 0, 0]
|
||||
events = [row.event for row in rows()]
|
||||
# Two batches, then ONE notice for the hour, not one per refused batch.
|
||||
assert events == ["nav.view", "nav.view", "diagnostics.budget_spent"]
|
||||
assert rows()[-1].user_id == user.id
|
||||
|
||||
|
||||
def test_only_a_json_body_is_read(elevated):
|
||||
_user, client, rows = elevated("DEBUG")
|
||||
# What a same-site `<form enctype="text/plain">` can send without a preflight.
|
||||
r = client.post(
|
||||
"/api/diagnostics/events",
|
||||
content=json.dumps({"events": [_event()]}),
|
||||
headers={"content-type": "text/plain"},
|
||||
)
|
||||
assert r.status_code == 415 and r.json()["detail"]["code"] == "json_required"
|
||||
assert rows() == []
|
||||
ok = client.post(
|
||||
"/api/diagnostics/events",
|
||||
content=json.dumps({"events": [_event()]}),
|
||||
headers={"content-type": "application/json; charset=utf-8"},
|
||||
)
|
||||
assert ok.json()["stored"] == 1
|
||||
|
||||
|
||||
# --- the server side of an elevation ---------------------------------------------------------
|
||||
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
"""The stdout handler's filters (context + gate), capturing records."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(level=logging.NOTSET)
|
||||
self.records: list[logging.LogRecord] = []
|
||||
self.addFilter(logs.ContextFilter())
|
||||
self.addFilter(logs.LevelGate())
|
||||
|
||||
def emit(self, record):
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gated():
|
||||
"""A process configured at INFO with a gated handler on the ROOT (where the real one sits), and
|
||||
every piece of module state the elevation touches restored afterwards."""
|
||||
root = logging.getLogger()
|
||||
saved = (logs._base_level, dict(logs._elevations), logs._root_applied, root.level)
|
||||
handler = _Capture()
|
||||
root.addHandler(handler)
|
||||
logs._elevations.clear()
|
||||
logs.set_level("INFO")
|
||||
try:
|
||||
yield handler
|
||||
finally:
|
||||
root.removeHandler(handler)
|
||||
logs._base_level, logs._root_applied = saved[0], saved[2]
|
||||
logs._elevations.clear()
|
||||
logs._elevations.update(saved[1])
|
||||
root.setLevel(saved[3])
|
||||
|
||||
|
||||
def _app() -> TestClient:
|
||||
"""What a real request does: the request middleware, then a SYNC auth dependency that resolves
|
||||
the account (run by FastAPI in a threadpool, on a copy of the context), then a handler."""
|
||||
app = FastAPI()
|
||||
app.middleware("http")(logs.log_requests)
|
||||
handler_log = logging.getLogger("siftlode.difftest")
|
||||
|
||||
def account(user_id: int, level: str = "INFO"):
|
||||
until = datetime.now(timezone.utc) + timedelta(hours=1)
|
||||
logs.set_user(user_id, level, until)
|
||||
|
||||
@app.get("/api/thing/{user_id}")
|
||||
def _thing(user_id: int, _=Depends(account)):
|
||||
handler_log.debug("detail only an elevated request keeps")
|
||||
handler_log.info("an ordinary line")
|
||||
return {}
|
||||
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_the_account_reaches_the_handlers_own_lines(gated):
|
||||
_app().get("/api/thing/41")
|
||||
line = next(r for r in gated.records if r.getMessage() == "an ordinary line")
|
||||
assert getattr(line, "user_id", None) == 41
|
||||
|
||||
|
||||
def test_one_users_elevation_logs_their_requests_in_detail_and_nobody_elses(gated):
|
||||
client = _app()
|
||||
client.get("/api/thing/7", params={"level": "DEBUG"})
|
||||
client.get("/api/thing/8")
|
||||
detail = [r for r in gated.records if r.getMessage().startswith("detail only")]
|
||||
assert [r.user_id for r in detail] == [7]
|
||||
assert getattr(detail[0], "elevated", False) is True
|
||||
# Every line of the elevated request is marked for storage — the sink keeps it at any level.
|
||||
ordinary = [r for r in gated.records if r.getMessage() == "an ordinary line"]
|
||||
assert [logsink.should_store(r) for r in ordinary] == [True, False]
|
||||
# The process's configured level is still what it reports, however low the root went.
|
||||
assert logs.current_level() == "INFO" and logging.getLogger().level == logging.DEBUG
|
||||
|
||||
|
||||
def test_the_root_level_returns_to_the_base_when_the_elevation_ends(gated):
|
||||
client = _app()
|
||||
client.get("/api/thing/7", params={"level": "TRACE"})
|
||||
assert logging.getLogger().level == logs.TRACE
|
||||
client.get("/api/thing/7") # the same user, back at the base
|
||||
assert logging.getLogger().level == logging.INFO
|
||||
logs._track_elevation(9, logging.DEBUG, datetime.now(timezone.utc) - timedelta(seconds=1))
|
||||
assert logging.getLogger().level == logging.INFO # an already-expired one is pruned at once
|
||||
|
||||
|
||||
def test_the_query_string_is_logged_only_for_a_traced_request_and_redacted(gated):
|
||||
client = _app()
|
||||
client.get("/api/thing/7", params={"level": "DEBUG", "q": "lofi"})
|
||||
client.get("/api/thing/7", params={"level": "TRACE", "q": "lofi", "token": "SECRET"})
|
||||
lines = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
|
||||
assert "query" not in lines[0].attrs
|
||||
assert "q=lofi" in lines[1].attrs["query"] and "SECRET" not in lines[1].attrs["query"]
|
||||
|
||||
|
||||
def test_an_address_typed_into_a_search_is_redacted_though_the_browser_encodes_it(gated):
|
||||
_app().get("/api/thing/7?level=TRACE&q=someone%40example.com&new_password=hunter2")
|
||||
(line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
|
||||
assert line.attrs["query"] == "level=TRACE&q=[email]&new_password=[redacted]"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"raw",
|
||||
["state=a%20b", "code=%20SECRET", "sig=a%22b", "key=a%26b", "reset=a+b"],
|
||||
ids=["space", "leading-space", "quote", "ampersand", "plus"],
|
||||
)
|
||||
def test_a_credential_param_goes_whole_however_its_value_decodes(raw):
|
||||
"""Decoding is what lets `_EMAIL_RE` see `%40`, but it also puts spaces and separators into a
|
||||
value that `_SECRET_PARAM_RE`'s value class stops at — so the name rule decides, not the text."""
|
||||
name = raw.split("=", 1)[0]
|
||||
assert logs.redact_query(f"{raw}&q=lofi") == f"{name}={logs.REDACTED}&q=lofi"
|
||||
|
||||
|
||||
def test_an_admin_level_of_trace_does_not_log_other_users_query_strings(gated):
|
||||
logs.set_level("TRACE")
|
||||
_app().get("/api/thing/7", params={"q": "something private"})
|
||||
(line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
|
||||
assert "query" not in line.attrs
|
||||
|
||||
|
||||
def test_the_admin_level_applies_to_the_running_process_when_set_and_when_reset(
|
||||
gated, db, make_user, as_user
|
||||
):
|
||||
admin = as_user(make_user(role="admin"))
|
||||
r = admin.patch("/api/admin/config/log_level", json={"value": "DEBUG"})
|
||||
assert r.status_code == 200, r.text
|
||||
assert logs.current_level() == "DEBUG"
|
||||
assert admin.delete("/api/admin/config/log_level").status_code == 200
|
||||
assert logs.current_level() == settings.log_level
|
||||
|
||||
|
||||
def test_erasing_the_only_elevated_account_puts_the_process_level_back(gated):
|
||||
"""The root level is the lowest of the base and every unexpired elevation. `forget_user` drops
|
||||
an elevation like any other path that drops one, so it owes the same recompute — or the process
|
||||
keeps building every TRACE record for an account that no longer exists, to throw them away."""
|
||||
until = datetime.now(timezone.utc) + timedelta(hours=1)
|
||||
try:
|
||||
logs.set_user(7, "TRACE", until)
|
||||
assert logging.getLogger().level == logs.TRACE
|
||||
logs.forget_user(7)
|
||||
assert logging.getLogger().level == logging.INFO
|
||||
finally:
|
||||
# `set_user` writes the request scope, and outside a request that scope is this process's
|
||||
# for good — a later test would store its INFO lines as an elevated session's.
|
||||
logs.set_user(None)
|
||||
|
||||
|
||||
def test_the_stored_admin_level_is_applied_at_boot(gated, db, monkeypatch):
|
||||
import asyncio
|
||||
|
||||
import app.main as app_main
|
||||
|
||||
sysconfig.set_value(db, "log_level", "DEBUG")
|
||||
db.commit()
|
||||
monkeypatch.setattr(app_main.state, "is_configured", lambda db: True)
|
||||
monkeypatch.setattr(app_main, "start_scheduler", lambda: None)
|
||||
monkeypatch.setattr(app_main, "shutdown_scheduler", lambda: None)
|
||||
monkeypatch.setattr(app_main.plex_subtitles, "shutdown", lambda: None)
|
||||
|
||||
async def _no_sweep():
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(app_main, "_sweep_hls_orphans", _no_sweep)
|
||||
levels: list[str] = []
|
||||
|
||||
async def _cycle():
|
||||
async with app_main.lifespan(app_main.app):
|
||||
levels.append(logs.current_level())
|
||||
|
||||
asyncio.run(_cycle())
|
||||
assert levels == ["DEBUG"]
|
||||
|
||||
|
||||
def test_the_root_level_is_applied_in_the_order_the_elevations_changed(gated, monkeypatch):
|
||||
"""Two requests change the elevations at once: A's user leaves (the root may go back to INFO),
|
||||
B's user arrives (it must go to DEBUG). B's change is the later one, so DEBUG must be what stays
|
||||
— even when A is paused between working out its level and applying it."""
|
||||
root = logging.getLogger()
|
||||
logs._track_elevation(1, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1))
|
||||
real_set = root.setLevel
|
||||
other: list[threading.Thread] = []
|
||||
|
||||
def paused_set(level):
|
||||
if level == logging.INFO and not other:
|
||||
b = threading.Thread(
|
||||
target=logs._track_elevation,
|
||||
args=(2, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1)),
|
||||
)
|
||||
other.append(b)
|
||||
b.start()
|
||||
b.join(0.3) # B runs to completion here unless A's level change holds it off
|
||||
real_set(level)
|
||||
|
||||
monkeypatch.setattr(root, "setLevel", paused_set)
|
||||
logs._track_elevation(1, None, None) # A
|
||||
other[0].join(5)
|
||||
assert root.level == logging.DEBUG and 2 in logs._elevations
|
||||
|
||||
|
||||
def test_a_real_sign_in_carries_the_consented_level_into_the_process(gated, client, db, make_user):
|
||||
user = _signed_in(client, make_user)
|
||||
assert client.put("/api/me/diagnostics", json={"level": "DEBUG"}).status_code == 200
|
||||
logs._elevations.clear() # only a request resolving the account through auth may put it back
|
||||
client.get("/api/me")
|
||||
assert logs._elevations[user.id][0] == logging.DEBUG
|
||||
assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
|
||||
assert user.id not in logs._elevations
|
||||
@@ -0,0 +1,63 @@
|
||||
"""The scheduled RSS poll and recent backfill cover FOLLOWED channels only (owner decision, 2026-09-17).
|
||||
|
||||
A channel row also comes from a search result, a playlist item or a subscription somebody dropped,
|
||||
and nothing deletes those rows. Polling all of them was 582 of 979 channels on prod with no follower,
|
||||
three of them dead feeds answering 404 every 20 minutes. Followed = subscribed by someone, or explored
|
||||
by someone. The admin card counting "channels to sync" must use the same selection, or it counts
|
||||
channels the backfill will never touch. DB lane.
|
||||
"""
|
||||
|
||||
from app.models import Channel, ExploredChannel, Subscription
|
||||
from app.sync import runner
|
||||
|
||||
|
||||
def _channels(db, make_user):
|
||||
user = make_user()
|
||||
for cid in ("UC_sub", "UC_explored", "UC_search", "UC_playlist"):
|
||||
db.add(Channel(id=cid, title=cid, from_search=cid == "UC_search"))
|
||||
db.flush()
|
||||
db.add(Subscription(user_id=user.id, channel_id="UC_sub"))
|
||||
db.add(ExploredChannel(user_id=user.id, channel_id="UC_explored"))
|
||||
db.commit()
|
||||
return user
|
||||
|
||||
|
||||
def test_the_rss_poll_skips_channels_nobody_follows(db, make_user, monkeypatch):
|
||||
_channels(db, make_user)
|
||||
polled: list[str] = []
|
||||
monkeypatch.setattr(runner, "fetch_channel_feed", lambda cid: polled.append(cid) or [])
|
||||
monkeypatch.setattr(runner, "apply_rss_feed", lambda db, ch, entries: 0)
|
||||
assert runner.run_rss_poll(db) == {"new": 0, "failed": 0}
|
||||
assert sorted(polled) == ["UC_explored", "UC_sub"]
|
||||
|
||||
|
||||
def test_the_recent_backfill_skips_channels_nobody_follows(db, make_user, monkeypatch):
|
||||
user = _channels(db, make_user)
|
||||
done: list[str] = []
|
||||
|
||||
class _Client:
|
||||
def __init__(self, *a):
|
||||
pass
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
monkeypatch.setattr(runner, "get_service_user", lambda db: user)
|
||||
monkeypatch.setattr(runner, "YouTubeClient", _Client)
|
||||
monkeypatch.setattr(runner.quota, "backfill_reserve_reached", lambda: False)
|
||||
monkeypatch.setattr(
|
||||
runner, "backfill_channel_recent", lambda db, yt, ch: done.append(ch.id) or 0
|
||||
)
|
||||
monkeypatch.setattr(runner, "reconcile_full_history", lambda db: 0)
|
||||
runner.run_recent_backfill(db)
|
||||
assert sorted(done) == ["UC_explored", "UC_sub"]
|
||||
|
||||
|
||||
def test_the_dashboard_counts_the_channels_the_backfill_will_process(db, make_user, as_user):
|
||||
_channels(db, make_user)
|
||||
admin = make_user(role="admin")
|
||||
queue = as_user(admin).get("/api/admin/scheduler").json()["queue"]
|
||||
assert queue["channels_recent_pending"] == 2
|
||||
@@ -0,0 +1,97 @@
|
||||
"""`users.last_login_at` (E22 S1), DB-lane.
|
||||
|
||||
The roadmap assumed stamping it in `_establish_session` would cover Google and password alike. It
|
||||
would not have: only password login called it, and the Google callback carried its own copy of the
|
||||
same four lines. So these pin BOTH doors through the real routes — plus the two that must not stamp
|
||||
(a refused password, the shared demo account) and the admin list that shows it.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from app.config import settings
|
||||
from app.models import DemoWhitelist
|
||||
from app.routes import auth as auth_routes
|
||||
|
||||
_PW = "correct horse battery"
|
||||
|
||||
|
||||
def _recent(ts: datetime | None) -> bool:
|
||||
return ts is not None and datetime.now(timezone.utc) - ts < timedelta(minutes=1)
|
||||
|
||||
|
||||
def test_a_password_sign_in_stamps_last_login(client, db, make_user):
|
||||
user = make_user(email="pw@example.com", password=_PW)
|
||||
assert user.last_login_at is None
|
||||
|
||||
r = client.post("/auth/password-login", json={"email": "pw@example.com", "password": _PW})
|
||||
|
||||
assert r.status_code == 200
|
||||
db.refresh(user)
|
||||
assert _recent(user.last_login_at)
|
||||
|
||||
|
||||
def test_a_refused_password_does_not_stamp(client, db, make_user):
|
||||
user = make_user(email="wrong@example.com", password=_PW)
|
||||
|
||||
r = client.post(
|
||||
"/auth/password-login", json={"email": "wrong@example.com", "password": "not it"}
|
||||
)
|
||||
|
||||
assert r.status_code == 401
|
||||
db.refresh(user)
|
||||
assert user.last_login_at is None
|
||||
|
||||
|
||||
class _FakeGoogle:
|
||||
"""authlib's client, reduced to the one call the callback makes after Google redirects back."""
|
||||
|
||||
def __init__(self, userinfo: dict):
|
||||
self._userinfo = userinfo
|
||||
|
||||
async def authorize_access_token(self, _request):
|
||||
return {"userinfo": self._userinfo, "access_token": "at", "scope": "openid email profile"}
|
||||
|
||||
|
||||
def test_a_google_sign_in_stamps_last_login(client, db, make_user, monkeypatch):
|
||||
from cryptography.fernet import Fernet
|
||||
|
||||
from app import security
|
||||
|
||||
# The callback stores the OAuth grant encrypted; a throwaway key stands in for TOKEN_ENCRYPTION_KEY.
|
||||
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
|
||||
user = make_user(email="g@example.com", password=None, google_sub="sub-g")
|
||||
monkeypatch.setattr(settings, "admin_emails", "g@example.com") # the bootstrap allow-list
|
||||
fake = _FakeGoogle({"sub": "sub-g", "email": "g@example.com", "email_verified": True})
|
||||
monkeypatch.setattr(auth_routes, "google_oauth", lambda _db: fake)
|
||||
|
||||
r = client.get("/auth/callback", follow_redirects=False)
|
||||
|
||||
assert r.status_code in (302, 307) and r.headers["location"] == "/"
|
||||
db.refresh(user)
|
||||
assert _recent(user.last_login_at)
|
||||
# …and it is a real sign-in, not only a stamp: the session now answers as this account.
|
||||
assert client.get("/api/me").json()["id"] == user.id
|
||||
|
||||
|
||||
def test_the_demo_entry_does_not_stamp_the_shared_account(client, db):
|
||||
db.add(DemoWhitelist(email="visitor@example.com"))
|
||||
db.commit()
|
||||
|
||||
r = client.post("/auth/demo", json={"email": "visitor@example.com"})
|
||||
|
||||
assert r.json() == {"authenticated": True}
|
||||
demo = auth_routes.get_or_create_demo_user(db)
|
||||
db.refresh(demo)
|
||||
assert demo.last_login_at is None
|
||||
|
||||
|
||||
def test_the_admin_user_list_carries_last_login(as_user, db, make_user):
|
||||
admin = make_user(email="boss@example.com", role="admin")
|
||||
never = make_user(email="never@example.com")
|
||||
admin.last_login_at = datetime(2026, 9, 16, 8, 30, tzinfo=timezone.utc)
|
||||
db.commit()
|
||||
|
||||
rows = {u["email"]: u for u in as_user(admin).get("/api/admin/users").json()}
|
||||
|
||||
assert rows["boss@example.com"]["last_login_at"].startswith("2026-09-16T08:30")
|
||||
assert rows[never.email]["last_login_at"] is None
|
||||
@@ -0,0 +1,320 @@
|
||||
"""E22 S2 — the one logging configuration, its redaction, and the request line that replaced
|
||||
uvicorn's access log.
|
||||
|
||||
Redaction is the part worth guarding hardest: it is applied at the HANDLER, so these cases are the
|
||||
only thing standing between a future call site and a token in a log we attach to a Gitea issue.
|
||||
The request-line cases pin what uvicorn's access log got wrong — the query string (`?account=1`,
|
||||
`?token=…`) was printed on every request, and nothing correlated two lines of the same request.
|
||||
|
||||
Pure logic + an in-process ASGI app: no database.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app import logs
|
||||
from app.config import LogLevel, choices_of
|
||||
|
||||
|
||||
class _Sink(logging.Handler):
|
||||
"""Captures records AND their formatted JSON, with the same filter/formatter the real stdout
|
||||
handler uses — so a test sees exactly the bytes a deployment would write."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(level=logging.NOTSET)
|
||||
self.lines: list[str] = []
|
||||
self.records: list[logging.LogRecord] = []
|
||||
self.setFormatter(logs.JsonFormatter())
|
||||
self.addFilter(logs.ContextFilter())
|
||||
|
||||
def emit(self, record):
|
||||
self.lines.append(self.format(record))
|
||||
self.records.append(record)
|
||||
|
||||
def objects(self) -> list[dict]:
|
||||
return [json.loads(line) for line in self.lines]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sink():
|
||||
"""Attached to `siftlode` (not root) so a `configure_logging` call inside a test — which
|
||||
replaces the ROOT handlers — can't silently detach it mid-test."""
|
||||
logger = logging.getLogger("siftlode")
|
||||
handler = _Sink()
|
||||
previous = logger.level
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(logs.TRACE)
|
||||
try:
|
||||
yield handler
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
logger.setLevel(previous)
|
||||
|
||||
|
||||
# --- redaction ------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text, must_not_contain",
|
||||
[
|
||||
("user signed in: someone@example.com", "someone@example.com"),
|
||||
("callback https://app/auth/callback?code=4/0AY&state=xyz", "4/0AY"),
|
||||
("mail link https://app/#verify=abc123def", "abc123def"),
|
||||
("share https://app/s?token=SHARETOKEN", "SHARETOKEN"),
|
||||
("reset https://app/#reset=RESETTOKEN", "RESETTOKEN"),
|
||||
],
|
||||
)
|
||||
def test_free_text_secrets_never_reach_the_line(text, must_not_contain):
|
||||
assert must_not_contain not in logs.redact_text(text)
|
||||
|
||||
|
||||
def test_a_key_that_names_a_secret_is_dropped_whole():
|
||||
# Not truncated to a prefix: a prefix is still a leak and is useless for triage anyway.
|
||||
out = logs.redact(
|
||||
{
|
||||
"password": "hunter2",
|
||||
"current_password": "hunter2",
|
||||
"plex_server_token": "abc",
|
||||
"Cookie": "session=xyz",
|
||||
"authorization": "Bearer abc",
|
||||
"nested": {"refresh_token": "rt", "keep": "visible"},
|
||||
"list": [{"api_key": "k"}],
|
||||
"count": 7,
|
||||
}
|
||||
)
|
||||
assert out["password"] == logs.REDACTED
|
||||
assert out["current_password"] == logs.REDACTED
|
||||
assert out["plex_server_token"] == logs.REDACTED
|
||||
assert out["Cookie"] == logs.REDACTED
|
||||
assert out["authorization"] == logs.REDACTED
|
||||
assert out["nested"]["refresh_token"] == logs.REDACTED
|
||||
assert out["nested"]["keep"] == "visible" # only the named keys go
|
||||
assert out["list"][0]["api_key"] == logs.REDACTED
|
||||
assert out["count"] == 7
|
||||
|
||||
|
||||
def test_an_email_in_an_attribute_value_is_scrubbed_too():
|
||||
assert logs.redact({"to": "a.b+tag@mail.example.org"})["to"] == logs.REDACTED_EMAIL
|
||||
|
||||
|
||||
# The SAME cases frontend/src/lib/log.contract.test.ts runs against the browser's copy of redact():
|
||||
# the one thing that keeps the two logs' idea of "safe to keep" from drifting apart.
|
||||
_REDACTION_CASES = json.loads(
|
||||
(Path(__file__).parent / "fixtures" / "log_redaction_cases.json").read_text(encoding="utf-8")
|
||||
)["cases"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case", _REDACTION_CASES, ids=[c["name"] for c in _REDACTION_CASES])
|
||||
def test_the_shared_redaction_cases(case):
|
||||
assert logs.redact(case["value"], key=case["key"]) == case["expected"]
|
||||
|
||||
|
||||
def test_redaction_runs_on_the_message_the_attrs_and_the_traceback(sink):
|
||||
log = logging.getLogger("siftlode.test")
|
||||
try:
|
||||
raise RuntimeError("boom for someone@example.com")
|
||||
except RuntimeError:
|
||||
log.exception(
|
||||
"failed for %s",
|
||||
"someone@example.com",
|
||||
extra=logs.event("test.failed", password="hunter2", url="https://x/?token=T0K3N"),
|
||||
)
|
||||
line = sink.lines[-1]
|
||||
assert "someone@example.com" not in line
|
||||
assert "hunter2" not in line
|
||||
assert "T0K3N" not in line
|
||||
payload = json.loads(line)
|
||||
assert payload["attrs"]["password"] == logs.REDACTED
|
||||
assert "RuntimeError" in payload["exc"] # the diagnosis survives the scrubbing
|
||||
|
||||
|
||||
# --- event shape ----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_every_line_carries_the_agreed_fields(sink):
|
||||
logging.getLogger("siftlode.test").info("hello", extra=logs.event("test.hello", n=1))
|
||||
payload = sink.objects()[-1]
|
||||
for field in ("ts", "level", "sev", "source", "logger", "msg", "event", "version"):
|
||||
assert field in payload, field
|
||||
assert payload["level"] == "INFO"
|
||||
assert payload["sev"] == 9
|
||||
assert payload["event"] == "test.hello"
|
||||
assert payload["attrs"] == {"n": 1}
|
||||
|
||||
|
||||
def test_a_plain_log_call_still_produces_a_well_formed_line(sink):
|
||||
"""An event code is optional — 125 existing call sites have none yet (E22.5 adds them), and they
|
||||
must not be worse off than before this module."""
|
||||
logging.getLogger("siftlode.test").warning("just a message")
|
||||
payload = sink.objects()[-1]
|
||||
assert payload["msg"] == "just a message"
|
||||
assert payload["level"] == "WARNING"
|
||||
assert "event" not in payload
|
||||
|
||||
|
||||
def test_a_broken_format_call_reports_itself_instead_of_raising():
|
||||
"""A record is formatted directly rather than logged: a genuinely broken call ALSO breaks
|
||||
pytest's own capture handler (stdlib formatting raises there too), which would fail the test for
|
||||
the wrong reason. What is under test is our formatter's refusal to raise."""
|
||||
record = logging.LogRecord(
|
||||
"siftlode.test", logging.INFO, __file__, 0, "two args %s %s", ("only-one",), None
|
||||
)
|
||||
payload = json.loads(logs.JsonFormatter().format(record))
|
||||
assert "log formatting failed" in payload["msg"]
|
||||
|
||||
|
||||
def test_bound_correlation_appears_and_is_released(sink):
|
||||
log = logging.getLogger("siftlode.test")
|
||||
with logs.bind(job_id=42):
|
||||
log.info("inside")
|
||||
log.info("outside")
|
||||
assert sink.objects()[-2]["job_id"] == 42
|
||||
assert "job_id" not in sink.objects()[-1]
|
||||
|
||||
|
||||
def test_every_selectable_level_resolves_to_a_real_logging_level():
|
||||
# The picker's options come from `LogLevel`; each must be a level `logging` knows, TRACE included.
|
||||
assert [logs.resolve_level(n) for n in choices_of(LogLevel)] == [
|
||||
logs.TRACE,
|
||||
logging.DEBUG,
|
||||
logging.INFO,
|
||||
logging.WARNING,
|
||||
logging.ERROR,
|
||||
]
|
||||
with pytest.raises(ValueError):
|
||||
logs.resolve_level("nonsense")
|
||||
|
||||
|
||||
def test_configure_logging_silences_the_uvicorn_access_log_and_keeps_ours_propagating():
|
||||
logs.configure_logging("api")
|
||||
assert logging.getLogger("uvicorn.access").getEffectiveLevel() >= logging.CRITICAL
|
||||
# propagate stays on so every line reaches the ONE root handler — and so pytest's caplog works.
|
||||
assert logging.getLogger("siftlode.anything").propagate is True
|
||||
assert len(logging.getLogger().handlers) == 1
|
||||
|
||||
|
||||
# --- the request line -----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _client() -> TestClient:
|
||||
app = FastAPI()
|
||||
app.middleware("http")(logs.log_requests)
|
||||
|
||||
@app.get("/api/videos/{video_id}")
|
||||
def _video(video_id: str, q: str | None = None):
|
||||
return {"ok": video_id}
|
||||
|
||||
@app.get("/healthz")
|
||||
def _health():
|
||||
return {"ok": True}
|
||||
|
||||
@app.get("/boom")
|
||||
def _boom():
|
||||
raise RuntimeError("kaboom")
|
||||
|
||||
return TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
|
||||
def test_the_request_line_has_the_route_template_and_no_query_string(sink):
|
||||
logging.getLogger("siftlode.http").setLevel(logs.TRACE)
|
||||
r = _client().get("/api/videos/abc123?q=cats&token=SECRET")
|
||||
assert r.status_code == 200
|
||||
payload = sink.objects()[-1]
|
||||
assert payload["event"] == "http.request"
|
||||
# The TEMPLATE, so every video request groups into one queryable path...
|
||||
assert payload["attrs"]["path"] == "/api/videos/{video_id}"
|
||||
# ...and the query string — uvicorn's access log printed it, `?account=1` and all — is gone.
|
||||
assert "SECRET" not in sink.lines[-1]
|
||||
assert "cats" not in sink.lines[-1]
|
||||
assert payload["attrs"]["status"] == 200
|
||||
assert isinstance(payload["attrs"]["duration_ms"], float)
|
||||
|
||||
|
||||
def test_anonymous_request_lines_are_capped_per_client_for_storage_but_still_logged(sink):
|
||||
# A scanner's 404s are WARNINGs, and WARNINGs are stored. Past the cap they stay on stdout but are
|
||||
# marked not-for-storage; a signed-in request is never capped. No address appears in any line.
|
||||
logs._anonymous_lines.reset()
|
||||
client = _client()
|
||||
for _ in range(logs.ANONYMOUS_LINES_PER_MINUTE + 3):
|
||||
client.get("/wp-login.php")
|
||||
lines = sink.objects()[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
|
||||
assert len(lines) == logs.ANONYMOUS_LINES_PER_MINUTE + 3 # stdout keeps every one
|
||||
records = sink.records[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
|
||||
stored = [getattr(r, logs.PERSIST, True) for r in records]
|
||||
assert stored.count(False) == 3 and all(stored[: logs.ANONYMOUS_LINES_PER_MINUTE])
|
||||
assert "testclient" not in "".join(sink.lines) # the client key never reaches the line
|
||||
logs._anonymous_lines.reset()
|
||||
|
||||
|
||||
def test_a_signed_in_request_is_never_capped_and_names_its_account(sink):
|
||||
logs._anonymous_lines.reset()
|
||||
app = FastAPI()
|
||||
app.middleware("http")(logs.log_requests)
|
||||
|
||||
@app.get("/api/me")
|
||||
def _me(request: Request):
|
||||
request.state.user_id = 42 # what the auth dependency does
|
||||
return {}
|
||||
|
||||
client = TestClient(app)
|
||||
for _ in range(logs.ANONYMOUS_LINES_PER_MINUTE + 3):
|
||||
client.get("/api/me")
|
||||
records = sink.records[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
|
||||
assert all(getattr(r, logs.PERSIST, True) for r in records)
|
||||
assert sink.objects()[-1]["user_id"] == 42 and "user_id" not in sink.objects()[-1]["attrs"]
|
||||
|
||||
|
||||
def test_the_request_id_is_echoed_and_shared_by_every_line_of_that_request(sink):
|
||||
client = _client()
|
||||
r = client.get("/api/videos/abc123")
|
||||
rid = r.headers["X-Request-Id"]
|
||||
assert rid and sink.objects()[-1]["request_id"] == rid
|
||||
|
||||
|
||||
def test_the_route_template_goes_back_to_the_browser_but_never_a_raw_path(sink):
|
||||
# The browser logs its own copy of each request, grouped by this template (frontend lib/http.ts).
|
||||
client = _client()
|
||||
assert client.get("/api/videos/SeCrEt-VaLuE").headers["X-Route"] == "/api/videos/{video_id}"
|
||||
# Unmatched: the only template there is the raw path, which may carry a value — so none is sent.
|
||||
assert "X-Route" not in client.get("/api/share/SeCrEt-VaLuE").headers
|
||||
|
||||
|
||||
def test_a_client_supplied_id_is_reused_only_when_it_is_well_formed(sink):
|
||||
client = _client()
|
||||
good = client.get("/api/videos/x", headers={"X-Request-Id": "web-0123456789ab"})
|
||||
assert good.headers["X-Request-Id"] == "web-0123456789ab"
|
||||
# An unbounded header would be a way to write arbitrary text into the log; short/odd ones are
|
||||
# replaced with our own id rather than trusted.
|
||||
for bogus in ("short", "has spaces and \n newline", "x" * 200):
|
||||
r = client.get("/api/videos/x", headers={"X-Request-Id": bogus})
|
||||
assert r.headers["X-Request-Id"] != bogus
|
||||
|
||||
|
||||
def test_the_healthcheck_is_logged_at_debug_so_it_cannot_drown_the_log(sink):
|
||||
http = logging.getLogger("siftlode.http")
|
||||
http.setLevel(logs.TRACE)
|
||||
_client().get("/healthz")
|
||||
assert sink.objects()[-1]["level"] == "DEBUG"
|
||||
# ...but it is still logged when something is wrong with it: the floor drops, nothing is hidden.
|
||||
http.setLevel(logging.INFO)
|
||||
before = len(sink.lines)
|
||||
_client().get("/healthz")
|
||||
assert len(sink.lines) == before
|
||||
|
||||
|
||||
def test_a_failing_request_is_logged_by_status_and_an_unhandled_error_is_logged_at_all(sink):
|
||||
client = _client()
|
||||
client.get("/api/videos/missing/deeper") # 404 from the router
|
||||
assert sink.objects()[-1]["level"] == "WARNING"
|
||||
client.get("/boom")
|
||||
payload = sink.objects()[-1]
|
||||
# The request that most needs a line is the one that raised: uvicorn's access log wrote it, but
|
||||
# our middleware sees the exception instead of a response, so it has to log it explicitly.
|
||||
assert payload["level"] == "ERROR"
|
||||
assert payload["event"] == "http.request"
|
||||
assert "RuntimeError" in payload["exc"]
|
||||
@@ -0,0 +1,260 @@
|
||||
"""E22 S3 — the `log_events` sink: what reaches the table, in what shape, and what it refuses to do.
|
||||
|
||||
The sink is driven synchronously (`write_pending()` on the test's own thread) so nothing here waits on
|
||||
a timer. It is attached to a private logger rather than the root, for the same reason test_logs.py's
|
||||
capture handler is: a `configure_logging` call elsewhere replaces the root handlers.
|
||||
DB lane (`db` fixture): the rows are read back from a real Postgres.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from fastapi import BackgroundTasks
|
||||
from sqlalchemy import create_engine, func, select
|
||||
|
||||
from app import logs, logsink, scheduler
|
||||
from app.auth import purge_user
|
||||
from app.db import engine
|
||||
from app.models import LogEvent
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stored(db):
|
||||
"""A LogSink on the logger `siftlode.sinktest`, and a reader for the rows it wrote."""
|
||||
sink = logsink.LogSink(engine)
|
||||
logger = logging.getLogger("siftlode.sinktest")
|
||||
logger.addHandler(sink)
|
||||
logger.setLevel(logs.TRACE)
|
||||
|
||||
def rows() -> list[LogEvent]:
|
||||
sink.write_pending()
|
||||
db.expire_all()
|
||||
return list(db.execute(select(LogEvent).order_by(LogEvent.id)).scalars())
|
||||
|
||||
try:
|
||||
yield sink, logger, rows
|
||||
finally:
|
||||
logger.removeHandler(sink)
|
||||
|
||||
|
||||
def test_warnings_and_request_lines_are_stored_and_everything_else_is_not(stored):
|
||||
_sink, log, rows = stored
|
||||
log.info("an ordinary info line")
|
||||
log.debug("GET /healthz 200", extra=logs.event("http.request", path="/healthz", status=200))
|
||||
log.info("GET /api/feed 200", extra=logs.event("http.request", path="/api/feed", status=200))
|
||||
log.warning("scanner", extra={**logs.event("http.request", status=404), logs.PERSIST: False})
|
||||
log.warning("disk nearly full")
|
||||
assert [r.msg for r in rows()] == ["GET /api/feed 200", "disk nearly full"]
|
||||
|
||||
|
||||
def test_a_row_is_redacted_exactly_like_the_stdout_line(stored):
|
||||
_sink, log, rows = stored
|
||||
try:
|
||||
raise ValueError("reset link https://x.test/#reset=abc123")
|
||||
except ValueError:
|
||||
log.exception(
|
||||
"sign-in failed for someone@example.com",
|
||||
extra=logs.event("auth.failed", password="hunter2", reason="bad_password"),
|
||||
)
|
||||
(row,) = rows()
|
||||
assert row.level == "ERROR" and row.sev == logs.SEVERITY["ERROR"]
|
||||
assert row.event == "auth.failed"
|
||||
assert "someone@example.com" not in row.msg and logs.REDACTED_EMAIL in row.msg
|
||||
assert row.attrs == {"password": logs.REDACTED, "reason": "bad_password"}
|
||||
assert "abc123" not in row.exc and "ValueError" in row.exc
|
||||
assert row.source == logs.current_source() and row.version
|
||||
|
||||
|
||||
def test_correlation_is_captured_when_the_line_is_logged_not_when_it_is_written(stored):
|
||||
_sink, log, rows = stored
|
||||
with logs.bind(job_id=7, request_id="req-0123456789"):
|
||||
log.warning("inside the job")
|
||||
# The writer runs later, outside the bind — the row still belongs to job 7.
|
||||
(row,) = rows()
|
||||
assert (row.job_id, row.request_id) == (7, "req-0123456789")
|
||||
|
||||
|
||||
def test_a_flood_of_one_message_is_stored_up_to_the_limit_and_then_summarised(stored):
|
||||
sink, log, rows = stored
|
||||
limit = logsink.REPEATS_PER_MINUTE
|
||||
for i in range(limit + 70):
|
||||
log.warning("RSS poll skipped for channel %s: HTTP 404", f"UC{i}")
|
||||
log.warning("a different message")
|
||||
assert len(rows()) == limit + 1 # the flood capped; the other message untouched
|
||||
sink._repeats.started -= logsink.REPEAT_WINDOW_SECONDS # the minute is over
|
||||
notice = rows()[-1]
|
||||
assert notice.event == "log.repeats_suppressed"
|
||||
assert notice.attrs["suppressed"] == 70
|
||||
assert notice.attrs["template"] == "RSS poll skipped for channel %s: HTTP 404"
|
||||
|
||||
|
||||
def test_request_lines_are_never_treated_as_a_flood(stored):
|
||||
_sink, log, rows = stored
|
||||
for _ in range(logsink.REPEATS_PER_MINUTE + 5):
|
||||
log.info("%s %s %s", "GET", "/api/feed", 200, extra=logs.event("http.request"))
|
||||
assert len(rows()) == logsink.REPEATS_PER_MINUTE + 5
|
||||
|
||||
|
||||
def test_a_full_queue_drops_and_says_how_many(db, monkeypatch):
|
||||
monkeypatch.setattr(logsink, "QUEUE_MAX", 3)
|
||||
sink = logsink.LogSink(engine)
|
||||
log = logging.getLogger("siftlode.sinktest.small")
|
||||
log.addHandler(sink)
|
||||
try:
|
||||
for i in range(5):
|
||||
log.warning("message number %d", i)
|
||||
finally:
|
||||
log.removeHandler(sink)
|
||||
assert sink.write_pending() == 4
|
||||
notice = db.execute(select(LogEvent).where(LogEvent.event == "log.sink.dropped")).scalar_one()
|
||||
assert notice.attrs == {"dropped": 2}
|
||||
|
||||
|
||||
def test_an_unreachable_database_costs_the_batch_not_the_caller(capsys):
|
||||
dead = create_engine(
|
||||
"postgresql+psycopg://x:y@127.0.0.1:1/none", connect_args={"connect_timeout": 1}
|
||||
)
|
||||
sink = logsink.LogSink(dead)
|
||||
log = logging.getLogger("siftlode.sinktest.dead")
|
||||
log.addHandler(sink)
|
||||
try:
|
||||
log.error("this has nowhere to go") # must not raise
|
||||
finally:
|
||||
log.removeHandler(sink)
|
||||
assert sink.write_pending() == 0
|
||||
assert "could not store 1 log events" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_the_writer_thread_does_not_store_its_own_records(stored):
|
||||
sink, log, rows = stored
|
||||
sink._writer = threading.current_thread() # as if this line came from inside the writer
|
||||
log.error("a driver warning raised while inserting")
|
||||
sink._writer = None
|
||||
assert rows() == []
|
||||
|
||||
|
||||
def test_gc_keeps_the_retention_window_and_never_keeps_forever(db):
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
def add(age: timedelta) -> None:
|
||||
db.add(
|
||||
LogEvent(ts=now - age, level="WARNING", sev=13, source="api", logger="t", msg=str(age))
|
||||
)
|
||||
|
||||
for age in (timedelta(hours=1), timedelta(days=2), timedelta(days=8)):
|
||||
add(age)
|
||||
db.commit()
|
||||
assert logsink.gc(db, 7) == 1
|
||||
assert logsink.gc(db, 0) == 1 # 0 is not "forever": stored logs always expire (floor: 1 day)
|
||||
db.commit()
|
||||
assert [r.msg for r in db.execute(select(LogEvent)).scalars()] == [str(timedelta(hours=1))]
|
||||
|
||||
|
||||
def test_log_gc_is_a_scheduled_job():
|
||||
assert "log_gc" in scheduler.JOB_FUNCS and scheduler.JOB_INTERVALS["log_gc"] >= 1
|
||||
|
||||
|
||||
def test_deleting_an_account_deletes_its_stored_log_events(db, make_user):
|
||||
gone, kept = make_user(), make_user()
|
||||
for uid in (gone.id, kept.id, gone.id):
|
||||
db.add(
|
||||
LogEvent(
|
||||
ts=datetime.now(timezone.utc),
|
||||
level="INFO",
|
||||
sev=9,
|
||||
source="api",
|
||||
logger="t",
|
||||
msg="x",
|
||||
user_id=uid,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
purge_user(db, gone, BackgroundTasks())
|
||||
assert [r.user_id for r in db.execute(select(LogEvent)).scalars()] == [kept.id]
|
||||
|
||||
|
||||
def test_an_erased_accounts_late_lines_are_stored_without_it(stored, db, make_user):
|
||||
"""The DELETE in purge_user cannot reach a row still on its way: one queued before the erasure,
|
||||
or the erasing request's own access line, logged after the handler committed."""
|
||||
_sink, log, rows = stored
|
||||
user = make_user()
|
||||
# `logs._erased` outlives the test; conftest's `_reset_log_erasures` clears it before each one.
|
||||
log.warning("queued before the erasure", extra={"user_id": user.id})
|
||||
purge_user(db, user, BackgroundTasks())
|
||||
log.info("DELETE /api/me/account 200", extra={**logs.event("http.request"), "user_id": user.id})
|
||||
assert [(r.msg, r.user_id) for r in rows()] == [
|
||||
("queued before the erasure", None),
|
||||
("DELETE /api/me/account 200", None),
|
||||
]
|
||||
|
||||
|
||||
def test_every_path_into_the_table_drops_an_erased_account():
|
||||
"""The two behavioural tests above cover the two writers there are today (this sink and
|
||||
`POST /api/diagnostics/events`). This one covers the THIRD one nobody has written yet: a new
|
||||
`insert(LogEvent)` that forgets the filter would leave an erased account's rows named, and no
|
||||
test of an existing path can notice. Source-level on purpose — it is a rule about call sites."""
|
||||
import pathlib
|
||||
|
||||
root = pathlib.Path(__file__).resolve().parents[1] / "app"
|
||||
unfiltered = [
|
||||
f"{path.relative_to(root)}:{n}"
|
||||
for path in root.rglob("*.py")
|
||||
for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1)
|
||||
if "insert(LogEvent)" in line and "without_erased" not in line
|
||||
]
|
||||
assert unfiltered == [], f"insert(LogEvent) without logsink.without_erased at {unfiltered}"
|
||||
|
||||
|
||||
def test_a_nul_character_costs_nothing_but_itself(stored):
|
||||
"""Postgres refuses NUL in text and JSONB, and a refused row fails its whole batch — every other
|
||||
user's rows with it. `GET /assets/%00` logs exactly that."""
|
||||
_sink, log, rows = stored
|
||||
log.warning("before")
|
||||
log.warning("GET /assets/\x00 404", extra=logs.event("http.request", path="/assets/\x00"))
|
||||
log.warning("after")
|
||||
before, nul, after = rows()
|
||||
assert (before.msg, after.msg) == ("before", "after")
|
||||
assert nul.msg == "GET /assets/� 404" and nul.attrs["path"] == "/assets/�"
|
||||
|
||||
|
||||
def test_the_writer_drains_what_was_queued_when_it_is_stopped(db):
|
||||
sink = logsink.LogSink(engine)
|
||||
log = logging.getLogger("siftlode.sinktest.drain")
|
||||
log.addHandler(sink)
|
||||
try:
|
||||
for i in range(logsink.BATCH_MAX + 5):
|
||||
log.warning(f"line {i}") # distinct templates: no flood limiter involved
|
||||
finally:
|
||||
log.removeHandler(sink)
|
||||
sink._stopping.set()
|
||||
sink._run() # stopped before its first wait: everything goes through the drain
|
||||
assert (
|
||||
db.execute(select(func.count()).select_from(LogEvent)).scalar_one() == logsink.BATCH_MAX + 5
|
||||
)
|
||||
|
||||
|
||||
def test_the_running_sink_keeps_the_base_level_for_everyone_the_elevation_is_not_for(
|
||||
db, monkeypatch
|
||||
):
|
||||
"""`logsink.start()` as production runs it. One user's elevation lowers the ROOT logger, so an
|
||||
ordinary request line below the base level reaches every handler — the sink's own level gate is
|
||||
what keeps it out of the table, as stdout's keeps it off the terminal."""
|
||||
root = logging.getLogger()
|
||||
saved = (logs._base_level, dict(logs._elevations), logs._root_applied, root.level)
|
||||
monkeypatch.setattr(logsink.settings, "log_db_sink", True)
|
||||
logs.set_level("WARNING")
|
||||
logs._track_elevation(1, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1))
|
||||
log = logging.getLogger("siftlode.sinktest.live")
|
||||
try:
|
||||
logsink.start()
|
||||
log.info("GET /api/feed 200", extra={**logs.event("http.request"), "user_id": 2})
|
||||
log.warning("a warning")
|
||||
finally:
|
||||
logsink.stop()
|
||||
logs._base_level, logs._root_applied = saved[0], saved[2]
|
||||
logs._elevations.clear()
|
||||
logs._elevations.update(saved[1])
|
||||
root.setLevel(saved[3])
|
||||
assert [r.msg for r in db.execute(select(LogEvent)).scalars()] == ["a warning"]
|
||||
@@ -1,6 +1,8 @@
|
||||
"""Guards that stop one bad YouTube answer from cascading: the subscription prune guard, the
|
||||
playlist-push circuit breaker, and RSS telling a failure apart from an empty feed."""
|
||||
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
@@ -120,7 +122,7 @@ class TestRssPollResult:
|
||||
# One dead channel among healthy ones: counted, but the run itself is still a run.
|
||||
def fetch(cid):
|
||||
if cid == "UC_dead":
|
||||
raise rss.RssError("404")
|
||||
raise rss.RssError("404", "http_404")
|
||||
return [{"id": "v"}]
|
||||
|
||||
out = self._run(
|
||||
@@ -143,6 +145,54 @@ class TestRssPollResult:
|
||||
)
|
||||
assert out == {"new": 1, "failed": 1}, "an unstored channel must not read as ok"
|
||||
|
||||
def test_many_failures_write_one_warning_with_counts_not_one_per_channel(
|
||||
self, monkeypatch, caplog
|
||||
):
|
||||
# YouTube's RSS outage of 2026-09-16: ~900 of 979 feeds failed on every 20-minute poll and
|
||||
# each wrote its own WARNING — 11k lines in four hours, which the log_events table would
|
||||
# have stored too. One summary per poll, per-channel detail at DEBUG.
|
||||
channels = [f"UC{i}" for i in range(40)]
|
||||
|
||||
def fetch(cid):
|
||||
n = int(cid[2:])
|
||||
if n < 30:
|
||||
raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404", "http_404")
|
||||
if n < 35:
|
||||
raise rss.RssError(f"RSS fetch failed for {cid}: timed out", "transport")
|
||||
return [{"id": "v"}]
|
||||
|
||||
caplog.set_level(logging.DEBUG, logger="siftlode.sync")
|
||||
out = self._run(monkeypatch, fetch=fetch, apply_=lambda db, ch, e: 1, channels=channels)
|
||||
assert out == {"new": 5, "failed": 35}
|
||||
warnings = [r for r in caplog.records if r.levelno >= logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert warnings[0].event == "sync.rss.failed"
|
||||
assert warnings[0].attrs["reasons"] == {"http_404": 30, "transport": 5}
|
||||
assert len(warnings[0].attrs["examples"]) == runner.RSS_FAILURE_EXAMPLES
|
||||
assert sum(r.levelno == logging.DEBUG for r in caplog.records) == 35
|
||||
|
||||
def test_a_failure_on_our_side_gets_one_traceback_per_poll_not_one_per_channel(
|
||||
self, monkeypatch, caplog
|
||||
):
|
||||
# A read-only database fails every channel's apply identically. The first gets a traceback
|
||||
# at ERROR; repeating it for each of 979 channels would be the RSS flood all over again.
|
||||
def apply_(db, ch, entries):
|
||||
raise RuntimeError("read-only transaction")
|
||||
|
||||
caplog.set_level(logging.DEBUG, logger="siftlode.sync")
|
||||
with pytest.raises(RuntimeError):
|
||||
self._run(
|
||||
monkeypatch,
|
||||
fetch=lambda cid: [{"id": "v"}],
|
||||
apply_=apply_,
|
||||
channels=("A", "B", "C"),
|
||||
)
|
||||
errors = [r for r in caplog.records if r.levelno == logging.ERROR]
|
||||
assert len(errors) == 1 and errors[0].exc_info is not None
|
||||
assert (
|
||||
sum(r.levelno == logging.DEBUG and r.exc_info is not None for r in caplog.records) == 2
|
||||
)
|
||||
|
||||
def test_a_quiet_hour_is_distinguishable_from_an_outage(self, monkeypatch):
|
||||
quiet = self._run(monkeypatch, fetch=lambda cid: [], apply_=lambda db, ch, e: 0)
|
||||
assert quiet == {"new": 0, "failed": 0} # same "new", different "failed" — that's the point
|
||||
@@ -152,7 +202,7 @@ class TestRssPollResult:
|
||||
# here recorded the scheduler run as "ok", with the outage only visible inside the summary
|
||||
# text — a green pill for a run that did nothing at all.
|
||||
def boom(cid):
|
||||
raise rss.RssError("Connection refused")
|
||||
raise rss.RssError("Connection refused", "transport")
|
||||
|
||||
with pytest.raises(RuntimeError, match="all 2 channels"):
|
||||
self._run(monkeypatch, fetch=boom, apply_=lambda db, ch, e: 1)
|
||||
@@ -162,7 +212,7 @@ class TestRssPollResult:
|
||||
# The card has to say which: `HTTP 404` = fix the subscription, `Connection refused` =
|
||||
# the egress is down.
|
||||
def dead(cid):
|
||||
raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404")
|
||||
raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404", "http_404")
|
||||
|
||||
with pytest.raises(RuntimeError, match="HTTP 404"):
|
||||
self._run(monkeypatch, fetch=dead, apply_=lambda db, ch, e: 1, channels=("UC_dead",))
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
"""The sysconfig registry: its order is the admin page's tab order, and a closed-set knob is a CHOICE.
|
||||
|
||||
The admin Configuration page used to keep its own hand-written group order, which had already drifted
|
||||
(`explore` was missing, so its tab sorted last). The page now follows SPECS, so SPECS has to read in
|
||||
tab order: one run of keys per group.
|
||||
|
||||
A closed-set admin knob is a CHOICE, not free text (E22 S2 follow-up).
|
||||
|
||||
`log_level`, `search_source`, `download_layout` and the transcode `hw`/`preset` used to be free-text
|
||||
fields: the admin could type anything and every consumer carried its own `or "auto").lower()` guess
|
||||
at what a typo meant. Now the value set is a `Literal` in config.py, pydantic holds the env to it at
|
||||
boot, sysconfig holds a stored override to it, and the admin page renders a picker from the same list.
|
||||
The DB lane (`db` fixture) covers the store; the rest is pure logic.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app import sysconfig
|
||||
from app.config import Settings, choices_of
|
||||
|
||||
CHOICE_KEYS = {s.key for s in sysconfig.SPECS if s.type == "choice"}
|
||||
|
||||
|
||||
def test_each_group_is_one_run_of_keys_so_the_registry_reads_in_tab_order():
|
||||
runs = [
|
||||
s.group
|
||||
for i, s in enumerate(sysconfig.SPECS)
|
||||
if i == 0 or sysconfig.SPECS[i - 1].group != s.group
|
||||
]
|
||||
assert len(runs) == len(set(runs)), runs
|
||||
assert runs[:2] == ["access", "email"]
|
||||
|
||||
|
||||
def test_every_closed_set_knob_is_a_choice_whose_options_come_from_config():
|
||||
assert CHOICE_KEYS == {
|
||||
"search_source",
|
||||
"download_layout",
|
||||
"log_level",
|
||||
"plex_transcode_hw",
|
||||
"plex_transcode_preset",
|
||||
}
|
||||
for s in sysconfig.SPECS:
|
||||
field = Settings.model_fields[s.default_attr]
|
||||
if s.type == "choice":
|
||||
assert s.choices, s.key
|
||||
assert field.default in s.choices, s.key
|
||||
else:
|
||||
# A Literal-typed setting exposed as "str" would be a free-text field again.
|
||||
assert s.choices == () and choices_of(field.annotation) == (), s.key
|
||||
|
||||
|
||||
def test_the_env_accepts_any_letter_case_and_rejects_an_unknown_value():
|
||||
assert Settings(log_level="debug").log_level == "DEBUG"
|
||||
assert Settings(search_source=" API ").search_source == "api"
|
||||
with pytest.raises(ValidationError):
|
||||
Settings(log_level="verbose")
|
||||
|
||||
|
||||
class _Row:
|
||||
def __init__(self, value):
|
||||
self.value, self.encrypted = value, False
|
||||
|
||||
|
||||
class _FakeDb:
|
||||
"""Just enough of a Session for sysconfig's reads: one stored override row per key."""
|
||||
|
||||
def __init__(self, **rows):
|
||||
self.rows = {k: _Row(v) for k, v in rows.items()}
|
||||
|
||||
def get(self, _model, key):
|
||||
return self.rows.get(key)
|
||||
|
||||
|
||||
def test_a_stored_value_reads_back_canonical_and_a_stale_one_falls_back_to_the_default():
|
||||
assert sysconfig.get_str(_FakeDb(plex_transcode_hw="OFF"), "plex_transcode_hw") == "off"
|
||||
# A value that is no longer (or never was) a choice is not passed through to the consumer.
|
||||
assert sysconfig.get_str(_FakeDb(plex_transcode_hw="maybe"), "plex_transcode_hw") == "auto"
|
||||
|
||||
|
||||
def test_describe_hands_the_picker_its_options():
|
||||
items = {i["key"]: i for g in sysconfig.describe(_FakeDb())["groups"].values() for i in g}
|
||||
assert items["log_level"]["choices"] == ["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"]
|
||||
assert items["log_level"]["value"] == "INFO"
|
||||
assert items["quota_daily_budget"]["choices"] == []
|
||||
|
||||
|
||||
def test_set_value_stores_the_canonical_spelling_and_refuses_anything_else(db):
|
||||
sysconfig.set_value(db, "search_source", "API")
|
||||
assert sysconfig.get_str(db, "search_source") == "api"
|
||||
with pytest.raises(ValueError, match="scrape, api"):
|
||||
sysconfig.set_value(db, "search_source", "bing")
|
||||
db.rollback()
|
||||
@@ -236,3 +236,37 @@ def test_stop_child_noop_if_already_exited(killpg_rec):
|
||||
proc.returncode = 0 # already gone
|
||||
worker._stop_child(proc, grace_s=0.01)
|
||||
assert killpg_rec == []
|
||||
|
||||
|
||||
def test_a_crashed_jobs_crash_line_and_cleanup_carry_the_job_id(monkeypatch):
|
||||
"""The job id is bound for the WHOLE job, the crash handler included: bound inside the `try`, it
|
||||
was already reset when the one line that says the job crashed was logged."""
|
||||
from app import logs
|
||||
|
||||
seen: dict[str, object] = {}
|
||||
claims = iter([42])
|
||||
|
||||
def claim():
|
||||
job = next(claims, None)
|
||||
if job is None:
|
||||
worker._stop.set()
|
||||
return job
|
||||
|
||||
def crash(_job_id):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
monkeypatch.setattr(worker, "_claim_job", claim)
|
||||
monkeypatch.setattr(worker, "_process_job", crash)
|
||||
monkeypatch.setattr(
|
||||
worker.log, "exception", lambda *a, **k: seen.setdefault("crash", logs._job_id.get())
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
worker, "_fail_running", lambda *a, **k: seen.setdefault("cleanup", logs._job_id.get())
|
||||
)
|
||||
monkeypatch.setattr(worker.settings, "download_poll_seconds", 0)
|
||||
worker._stop.clear()
|
||||
try:
|
||||
worker._worker_loop(0)
|
||||
finally:
|
||||
worker._stop.clear()
|
||||
assert seen == {"crash": 42, "cleanup": 42}
|
||||
|
||||
@@ -11,6 +11,15 @@
|
||||
# applies local migrations to THIS local DB only. Seed the catalog once with a pg_dump from
|
||||
# another instance if you want real data (see RUNBOOK).
|
||||
|
||||
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
|
||||
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
|
||||
# busy instance and a hard ceiling on the disk it can take.
|
||||
x-logging: &logging
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "50m"
|
||||
max-file: "3"
|
||||
|
||||
services:
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
@@ -27,6 +36,7 @@ services:
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
api:
|
||||
@@ -63,6 +73,7 @@ services:
|
||||
- ./downloads:/downloads
|
||||
# Plex media, read-only (see the plex_media volume note below). Maps to plex_path_map.
|
||||
- plex_media:/plex-media:ro
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# Dedicated download worker: same image, runs the yt-dlp job loop instead of the API.
|
||||
@@ -96,6 +107,7 @@ services:
|
||||
volumes:
|
||||
- ./downloads:/downloads
|
||||
- plex_media:/plex-media:ro
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# PO-token provider: mints YouTube Proof-of-Origin tokens on demand so the worker bypasses
|
||||
@@ -106,6 +118,7 @@ services:
|
||||
init: true
|
||||
security_opt:
|
||||
- apparmor:unconfined
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
|
||||
@@ -10,6 +10,15 @@
|
||||
# Everything else — the admin account, Google sign-in, SMTP — is set in the web wizard on
|
||||
# first run. See docs/self-hosting.md.
|
||||
|
||||
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
|
||||
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
|
||||
# busy instance and a hard ceiling on the disk it can take.
|
||||
x-logging: &logging
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "50m"
|
||||
max-file: "3"
|
||||
|
||||
services:
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
@@ -27,6 +36,7 @@ services:
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
api:
|
||||
@@ -71,6 +81,7 @@ services:
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
|
||||
@@ -109,6 +120,7 @@ services:
|
||||
- ALL
|
||||
tmpfs:
|
||||
- /tmp
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
|
||||
@@ -119,6 +131,7 @@ services:
|
||||
networks: [internal]
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
|
||||
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
|
||||
# busy instance and a hard ceiling on the disk it can take.
|
||||
x-logging: &logging
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "50m"
|
||||
max-file: "3"
|
||||
|
||||
services:
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
@@ -12,6 +21,7 @@ services:
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
api:
|
||||
@@ -49,6 +59,7 @@ services:
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 25s
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
|
||||
@@ -83,6 +94,7 @@ services:
|
||||
- ${DOWNLOAD_HOST_PATH:-downloads}:/downloads
|
||||
# Same read-only Plex library as the API — the worker resolves library paths too.
|
||||
- ${PLEX_MEDIA_HOST_PATH:-plex_media}:/plex-media:ro
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
# PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
|
||||
@@ -90,6 +102,7 @@ services:
|
||||
bgutil-pot:
|
||||
image: brainicism/bgutil-ytdlp-pot-provider:1.3.1
|
||||
init: true
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
|
||||
@@ -14,6 +14,36 @@ import globals from "globals";
|
||||
//
|
||||
// reportUnusedDisableDirectives: a `// eslint-disable` that no longer suppresses anything is itself
|
||||
// an error — that is what keeps inert disable comments from re-accumulating (the S1b mandate).
|
||||
|
||||
// E22 S4: every way out of the browser goes through the one module that LOGS it — a request line
|
||||
// with a request id (lib/http.ts), the upload XHR (lib/api/core.ts), the messages socket
|
||||
// (lib/messagesSocket.ts). A bare `fetch` elsewhere is a request the client log never sees, which is
|
||||
// exactly how the baseline survey found four. Each owner is exempt from its OWN global only.
|
||||
const NETWORK_OWNERS = {
|
||||
fetch: "src/lib/http.ts",
|
||||
XMLHttpRequest: "src/lib/api/core.ts",
|
||||
WebSocket: "src/lib/messagesSocket.ts",
|
||||
EventSource: null,
|
||||
};
|
||||
const networkGlobals = (exempt) =>
|
||||
Object.entries(NETWORK_OWNERS)
|
||||
.filter(([name]) => name !== exempt)
|
||||
.map(([name, owner]) => ({
|
||||
name,
|
||||
message: owner
|
||||
? `Use the logged request layer: \`${name}\` belongs to ${owner} only (E22 S4 — a request outside it never reaches the client log).`
|
||||
: `No \`${name}\` without a logged owner module (E22 S4).`,
|
||||
}));
|
||||
const networkProperties = [
|
||||
{ object: "window", property: "fetch", message: "Use lib/http.ts `send` (E22 S4)." },
|
||||
{ object: "globalThis", property: "fetch", message: "Use lib/http.ts `send` (E22 S4)." },
|
||||
{
|
||||
object: "navigator",
|
||||
property: "sendBeacon",
|
||||
message: "Use lib/api/core.ts `beacon` (E22 S4).",
|
||||
},
|
||||
];
|
||||
|
||||
export default tseslint.config(
|
||||
{ ignores: ["dist", "node_modules", "playwright-report", "test-results", "e2e/.auth"] },
|
||||
{
|
||||
@@ -42,6 +72,11 @@ export default tseslint.config(
|
||||
// types; i18n `t` props → TFunction), so the rule is now enforced to keep new ones from
|
||||
// creeping back in. Prefer `unknown` + narrowing, or a typed boundary, over `any`.
|
||||
"@typescript-eslint/no-explicit-any": "error",
|
||||
// E22 S4: the client log (lib/log.ts) is the one place diagnostics go — redacted, levelled,
|
||||
// bounded. A `console.*` call is none of those, and it is invisible to the user's own log.
|
||||
"no-console": "error",
|
||||
"no-restricted-globals": ["error", ...networkGlobals(null)],
|
||||
"no-restricted-properties": ["error", ...networkProperties],
|
||||
// Side-effect ternaries/short-circuits are an established idiom here (`cond ? a() : b()`,
|
||||
// `v.paused ? v.play() : v.pause()`) — allow them rather than rewrite working code to if/else.
|
||||
"@typescript-eslint/no-unused-expressions": [
|
||||
@@ -131,5 +166,17 @@ export default tseslint.config(
|
||||
],
|
||||
},
|
||||
},
|
||||
// The network owners (see NETWORK_OWNERS), each exempt from its own global only.
|
||||
...Object.entries(NETWORK_OWNERS)
|
||||
.filter(([, owner]) => owner)
|
||||
.map(([name, owner]) => ({
|
||||
files: [owner],
|
||||
rules: { "no-restricted-globals": ["error", ...networkGlobals(name)] },
|
||||
})),
|
||||
{
|
||||
// Tests stand doubles in for the network (`vi.stubGlobal("fetch", …)`) and may reach it directly.
|
||||
files: ["src/**/*.test.{ts,tsx}"],
|
||||
rules: { "no-restricted-globals": "off", "no-restricted-properties": "off" },
|
||||
},
|
||||
prettier,
|
||||
);
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
"build": "npm run typecheck:app && npm run typecheck:node && vite build",
|
||||
"preview": "vite preview",
|
||||
"knip": "knip",
|
||||
"lint": "eslint .",
|
||||
"lint": "eslint . --max-warnings 0",
|
||||
"format": "prettier --write .",
|
||||
"format:check": "prettier --check .",
|
||||
"test": "vitest run",
|
||||
|
||||
+13
-4
@@ -18,10 +18,10 @@ import { notify, removeByMetaKind } from "./lib/notifications";
|
||||
import { LS, setAccountRaw } from "./lib/storage";
|
||||
import { usePref } from "./lib/usePref";
|
||||
import { useVerifyEntry } from "./lib/useVerifyEntry";
|
||||
import { useNavigation, useNavigationActions } from "./components/NavigationProvider";
|
||||
import { useFeedFilters, useFeedFiltersActions } from "./components/FeedFiltersProvider";
|
||||
import { useChannelLayout } from "./components/ChannelsProvider";
|
||||
import { useFeedView, useFeedViewActions } from "./components/FeedViewProvider";
|
||||
import { useNavigation, useNavigationActions } from "./components/navigationContext";
|
||||
import { useFeedFilters, useFeedFiltersActions } from "./components/feedFiltersContext";
|
||||
import { useChannelLayout } from "./components/channelsContext";
|
||||
import { useFeedView, useFeedViewActions } from "./components/feedViewContext";
|
||||
import { moduleDef, moduleOrder } from "./lib/modules";
|
||||
import { PAGE_CONTENT, PAGE_RAIL, preloadPages } from "./components/moduleRegistry";
|
||||
import { lazyWithPreload } from "./lib/lazyWithPreload";
|
||||
@@ -48,6 +48,7 @@ import { focusAccessRequestsTab } from "./lib/adminUsersTab";
|
||||
// pulls its whole ~900-line changelog into the eager App chunk and defeats the lazy ReleaseNotes
|
||||
// split. FRONTEND_VERSION equals RELEASE_NOTES[0].version by release convention (both bump together).
|
||||
import { FRONTEND_VERSION } from "./lib/version";
|
||||
import { applyDiagnostics } from "./lib/diagnostics";
|
||||
|
||||
// The page modules (and their side rails) are code-split per route via the module registry
|
||||
// (components/moduleRegistry). App keeps only the persistent-shell chunks + the channel page and the
|
||||
@@ -151,6 +152,14 @@ export default function App() {
|
||||
meQuery.data ? [...moduleOrder(meQuery.data), "settings"] : [],
|
||||
);
|
||||
|
||||
// The account's consented diagnostics level (E22 S5), applied to this tab's log: raised, the log is
|
||||
// kept in more detail and streamed to the server until the level lapses. A lapse (its own timer, or
|
||||
// the server saying it is off) refetches `me`, so Settings shows the base level again.
|
||||
const diagnostics = meQuery.data?.diagnostics;
|
||||
useEffect(() => {
|
||||
applyDiagnostics(diagnostics, () => void queryClient.invalidateQueries({ queryKey: qk.me() }));
|
||||
}, [diagnostics?.level, diagnostics?.until]); // eslint-disable-line react-hooks/exhaustive-deps
|
||||
|
||||
// Fetch the code of every module this account can open — and the channel page — once the first page
|
||||
// is on screen, so a module's first visit renders at once instead of flashing the chunk fallback.
|
||||
// Idle-scheduled so it never competes with the first page's own requests (Safari has no
|
||||
|
||||
@@ -4,13 +4,14 @@ import { useTranslation } from "react-i18next";
|
||||
import { Info, Sparkles } from "lucide-react";
|
||||
import { api } from "../lib/api";
|
||||
import { FRONTEND_VERSION, FRONTEND_SHA, FRONTEND_BUILD_DATE } from "../lib/version";
|
||||
import { formatDateTime } from "../lib/format";
|
||||
import { Button } from "./ui/Button";
|
||||
import Modal from "./ui/Modal";
|
||||
|
||||
function fmtDate(d?: string | null): string {
|
||||
if (!d) return "—";
|
||||
const t = new Date(d);
|
||||
return isNaN(t.getTime()) ? d : t.toLocaleString();
|
||||
return isNaN(t.getTime()) ? d : formatDateTime(d);
|
||||
}
|
||||
|
||||
// About dialog: app + build + schema versions (frontend/backend/database), plus a
|
||||
|
||||
@@ -3,7 +3,8 @@ import { useQuery } from "@tanstack/react-query";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { api } from "../lib/api";
|
||||
import ChannelAboutContent from "./ChannelAbout";
|
||||
import { PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
|
||||
import { PopoverSurface } from "./ui/AnchoredPopover";
|
||||
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
|
||||
|
||||
/**
|
||||
* A channel-description cell: one truncated line in the table, revealing the FULL channel "About"
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Check, Clock, Copy, Globe, KeyRound, Mail, Send, ShieldAlert } from "lucide-react";
|
||||
import { Check, Clock, Copy, Globe, KeyRound, LogIn, Mail, Send, ShieldAlert } from "lucide-react";
|
||||
import Modal from "./ui/Modal";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import { api } from "../lib/api";
|
||||
@@ -9,6 +9,7 @@ import { qk } from "../lib/queryKeys";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { notifyActionError } from "../lib/actionErrors";
|
||||
import { copyText } from "../lib/clipboard";
|
||||
import { formatDateTime } from "../lib/format";
|
||||
import { invalidateUserAccess } from "../lib/adminUsers";
|
||||
import type { AdminUserAccess, AdminUserRow } from "../lib/api/admin";
|
||||
|
||||
@@ -168,6 +169,20 @@ export default function AccountPanel({
|
||||
}
|
||||
tone={a.has_google ? "ok" : "muted"}
|
||||
/>
|
||||
{/* The exact moment; the list row beside it only says how long ago. Null has two different
|
||||
meanings, so it says which: nothing recorded yet, or the shared demo account, which is
|
||||
deliberately never stamped. */}
|
||||
<AccessRow
|
||||
icon={LogIn}
|
||||
label={t("users.account.lastLogin")}
|
||||
value={
|
||||
a.last_login_at
|
||||
? formatDateTime(a.last_login_at)
|
||||
: a.is_demo
|
||||
? t("users.account.lastLoginDemo")
|
||||
: t("users.account.lastLoginNone")
|
||||
}
|
||||
/>
|
||||
{/* The gate that explains a Google sign-in silently becoming a fresh access request. */}
|
||||
{!a.is_allowed && (
|
||||
<AccessRow
|
||||
|
||||
@@ -4,7 +4,8 @@ import { Filter, X } from "lucide-react";
|
||||
import type { ActiveFilter } from "../lib/useActiveFilters";
|
||||
import { decideChipLayout } from "../lib/chipOverflow";
|
||||
import { chipLaneClass, useChipOverflowFit } from "../lib/useChipOverflowFit";
|
||||
import { PopoverItem, PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
|
||||
import { PopoverItem, PopoverSurface } from "./ui/AnchoredPopover";
|
||||
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
|
||||
|
||||
// The applied-filter row above the feed: every narrowing currently in force, each removable on its
|
||||
// own. It exists because the filter panel now sits collapsed as a tab — without this you'd have to
|
||||
|
||||
@@ -7,7 +7,8 @@ import { api, type Playlist } from "../lib/api";
|
||||
import { notifyActionError } from "../lib/actionErrors";
|
||||
import { useScrollFade } from "../lib/useScrollFade";
|
||||
import { playlistName } from "../lib/playlistName";
|
||||
import { PopoverItem, PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
|
||||
import { PopoverItem, PopoverSurface } from "./ui/AnchoredPopover";
|
||||
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
|
||||
|
||||
// A small anchored menu for toggling a video's membership in the user's playlists, with inline
|
||||
// "new playlist" creation. Same shape as the channel tag editor (R8 S1's pilot): radio-style
|
||||
|
||||
@@ -24,7 +24,7 @@ vi.mock("../lib/useMe", () => ({
|
||||
pending_activations: 0,
|
||||
}),
|
||||
}));
|
||||
vi.mock("./ConfirmProvider", () => ({ useConfirm: () => async () => true }));
|
||||
vi.mock("./confirmContext", () => ({ useConfirm: () => async () => true }));
|
||||
const { notify } = vi.hoisted(() => ({
|
||||
notify: vi.fn((_n: { level: string; message: string }) => 0),
|
||||
}));
|
||||
@@ -36,6 +36,7 @@ vi.mock("../lib/errorDialog", () => ({ reportError: vi.fn() }));
|
||||
|
||||
import AdminUsers from "./AdminUsers";
|
||||
import { api, type AdminUserRow } from "../lib/api";
|
||||
import { relativeTime } from "../lib/format";
|
||||
import i18n from "../i18n";
|
||||
|
||||
const row = {
|
||||
@@ -76,6 +77,27 @@ function mount() {
|
||||
return users;
|
||||
}
|
||||
|
||||
describe("the last sign-in column (E22 S1)", () => {
|
||||
it("says how long ago each account signed in, and a dash where nothing is recorded", async () => {
|
||||
const at = new Date(Date.now() - 3 * 86_400_000).toISOString();
|
||||
vi.spyOn(api, "adminUsers").mockResolvedValue([
|
||||
{ ...row, id: 3, email: "seen@x", last_login_at: at },
|
||||
{ ...row, id: 4, email: "never@x", last_login_at: null },
|
||||
{ ...row, id: 5, email: "demo@x", is_demo: true, last_login_at: null },
|
||||
] as never);
|
||||
render(
|
||||
<QueryClientProvider
|
||||
client={new QueryClient({ defaultOptions: { queries: { retry: false } } })}
|
||||
>
|
||||
<AdminUsers />
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
|
||||
expect(await screen.findByText(relativeTime(at))).toBeTruthy();
|
||||
expect(screen.getAllByText("—")).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("a user action that fails", () => {
|
||||
it("says a user who is gone, and refetches the list", async () => {
|
||||
const users = mount();
|
||||
|
||||
@@ -24,6 +24,7 @@ import {
|
||||
type PendingAccessRow,
|
||||
} from "../lib/api";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { formatDateTime, relativeTime } from "../lib/format";
|
||||
import { invalidateUserAccess } from "../lib/adminUsers";
|
||||
import { ADMIN_USERS_TAB_KEY } from "../lib/adminUsersTab";
|
||||
import { Button } from "./ui/Button";
|
||||
@@ -34,8 +35,9 @@ import SegmentedToggle from "./ui/SegmentedToggle";
|
||||
import { LANGUAGES, type LangCode } from "../i18n";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import AccountPanel from "./AccountPanel";
|
||||
import { useConfirm } from "./ConfirmProvider";
|
||||
import Tabs, { usePersistedTab } from "./ui/Tabs";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import Tabs from "./ui/Tabs";
|
||||
import { useAccountPersistedState } from "../lib/storage";
|
||||
import { PageToolbar } from "./ui/PageShell";
|
||||
import { useMe } from "../lib/useMe";
|
||||
import { notifyActionError, notifyKnownFailure } from "../lib/actionErrors";
|
||||
@@ -49,7 +51,7 @@ import { notifyActionError, notifyKnownFailure } from "../lib/actionErrors";
|
||||
export default function AdminUsers() {
|
||||
const me = useMe();
|
||||
const { t } = useTranslation();
|
||||
const [tab, setTab] = usePersistedTab(ADMIN_USERS_TAB_KEY, "roles");
|
||||
const [tab, setTab] = useAccountPersistedState(ADMIN_USERS_TAB_KEY, "roles");
|
||||
const tabs = [
|
||||
{ id: "roles", label: t("users.tabs.roles") },
|
||||
// Both kinds of "waiting on the admin": a request needing a decision, and a sign-up whose
|
||||
@@ -310,6 +312,23 @@ function UsersRoles({ me }: { me: Me }) {
|
||||
icon={<Trash2 className="w-4 h-4" />}
|
||||
/>
|
||||
</Tooltip>
|
||||
{/* How long ago this account last signed in. LAST in the row on purpose: the badges and
|
||||
the role button before it vary in width row to row, so only the row's own right
|
||||
edge lines the values up into one column. The exact moment is the hint here and a
|
||||
line in the account panel; below `sm` the row has no room, and the panel still says it. */}
|
||||
<Tooltip
|
||||
hint={
|
||||
u.last_login_at
|
||||
? t("users.roles.lastLogin", { time: formatDateTime(u.last_login_at) })
|
||||
: u.is_demo
|
||||
? t("users.roles.lastLoginDemo")
|
||||
: t("users.roles.lastLoginNone")
|
||||
}
|
||||
>
|
||||
<span className="hidden sm:inline-block w-20 shrink-0 text-right text-2xs text-muted tabular-nums">
|
||||
{u.last_login_at ? relativeTime(u.last_login_at) : "—"}
|
||||
</span>
|
||||
</Tooltip>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
@@ -413,7 +432,7 @@ function PendingAccessRowView({
|
||||
{row.created_at && (
|
||||
<span>
|
||||
{t("users.pendingAccess.signedUp", {
|
||||
time: new Date(row.created_at).toLocaleString(),
|
||||
time: formatDateTime(row.created_at),
|
||||
})}
|
||||
</span>
|
||||
)}
|
||||
@@ -601,7 +620,10 @@ function AdminInvites() {
|
||||
inv.invited_at
|
||||
? "settings.invites.inviteConfirmAgain"
|
||||
: "settings.invites.inviteConfirm",
|
||||
{ email: inv.email, time: new Date(inv.invited_at ?? 0).toLocaleString() },
|
||||
{
|
||||
email: inv.email,
|
||||
time: formatDateTime(inv.invited_at ?? new Date(0).toISOString()),
|
||||
},
|
||||
)}
|
||||
</p>
|
||||
<InviteLangPicker initial={lang} onPick={(v) => (lang = v)} />
|
||||
@@ -700,7 +722,7 @@ function AdminInvites() {
|
||||
hint={
|
||||
i.invited_at
|
||||
? t("settings.invites.invitedAt", {
|
||||
time: new Date(i.invited_at).toLocaleString(),
|
||||
time: formatDateTime(i.invited_at),
|
||||
})
|
||||
: t("settings.invites.inviteHint")
|
||||
}
|
||||
@@ -898,7 +920,7 @@ function InviteRow({
|
||||
{inv.requested_at && (
|
||||
<div className="text-2xs text-muted">
|
||||
{t("settings.invites.requested", {
|
||||
time: new Date(inv.requested_at).toLocaleString(),
|
||||
time: formatDateTime(inv.requested_at),
|
||||
})}
|
||||
</div>
|
||||
)}
|
||||
|
||||
@@ -10,10 +10,10 @@ import { moduleDef, moduleOrder } from "../lib/modules";
|
||||
import { SCHEMES, type Scheme } from "../lib/theme";
|
||||
import { closePalette, toggleKeyboardHelp, usePaletteOpen } from "../lib/keymap";
|
||||
import { requestSurprise } from "../lib/homeIntent";
|
||||
import { useNavigationActions } from "./NavigationProvider";
|
||||
import { usePlaylistsActions } from "./PlaylistsProvider";
|
||||
import { usePrefs } from "./PrefsProvider";
|
||||
import { useSavedViews } from "./SavedViewsProvider";
|
||||
import { useNavigationActions } from "./navigationContext";
|
||||
import { usePlaylistsActions } from "./playlistsContext";
|
||||
import { usePrefs } from "./prefsContext";
|
||||
import { useSavedViews } from "./savedViewsContext";
|
||||
|
||||
// What Ctrl/Cmd+K can reach. The generic list/filter/keyboard shell is `ui/CommandPalette`; this is
|
||||
// the domain half — the layer boundary says a primitive may not import `lib/api` or a module.
|
||||
|
||||
@@ -8,7 +8,7 @@ import { notify } from "../lib/notifications";
|
||||
import { LS } from "../lib/storage";
|
||||
import { Button } from "./ui/Button";
|
||||
import { Section } from "./ui/form";
|
||||
import { useConfirm } from "./ConfirmProvider";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import DataTable from "./ui/DataTable";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import { auditColumns } from "./auditColumns";
|
||||
|
||||
@@ -14,7 +14,8 @@ import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { useEffect } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import BootGate from "./BootGate";
|
||||
import { NavigationProvider, useNavigation } from "./NavigationProvider";
|
||||
import { NavigationProvider } from "./NavigationProvider";
|
||||
import { useNavigation } from "./navigationContext";
|
||||
import { api, type Me } from "../lib/api";
|
||||
import { ACTIVE_ACCOUNT_KEY } from "../lib/activeAccount";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
|
||||
@@ -21,8 +21,8 @@ import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import ColumnSortControl from "./ui/ColumnSortControl";
|
||||
import Pager from "./ui/Pager";
|
||||
import { PageToolbar } from "./ui/PageShell";
|
||||
import { useConfirm } from "./ConfirmProvider";
|
||||
import { useChannelLayout } from "./ChannelsProvider";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import { useChannelLayout } from "./channelsContext";
|
||||
|
||||
// The Channel manager's "Discovery" tab: channels that turn up in the user's playlists but
|
||||
// that they don't subscribe to. Subscribing here only creates the subscription + enriches
|
||||
|
||||
@@ -16,11 +16,11 @@ vi.mock("react-i18next", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("react-i18next")>()),
|
||||
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
|
||||
}));
|
||||
vi.mock("./NavigationProvider", () => ({
|
||||
vi.mock("./navigationContext", () => ({
|
||||
useNavigationActions: () => ({ closeChannel: vi.fn(), openChannel: vi.fn() }),
|
||||
}));
|
||||
vi.mock("./ConfirmProvider", () => ({ useConfirm: () => async () => true }));
|
||||
vi.mock("./FeedFiltersProvider", () => ({
|
||||
vi.mock("./confirmContext", () => ({ useConfirm: () => async () => true }));
|
||||
vi.mock("./feedFiltersContext", () => ({
|
||||
useFeedFilters: () => ({ includeNormal: true, includeShorts: false, includeLive: false }),
|
||||
}));
|
||||
vi.mock("./Feed", () => ({ default: () => null }));
|
||||
|
||||
@@ -14,11 +14,11 @@ vi.mock("react-i18next", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("react-i18next")>()),
|
||||
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
|
||||
}));
|
||||
vi.mock("./NavigationProvider", () => ({
|
||||
vi.mock("./navigationContext", () => ({
|
||||
useNavigationActions: () => ({ closeChannel: vi.fn(), openChannel: vi.fn() }),
|
||||
}));
|
||||
vi.mock("./ConfirmProvider", () => ({ useConfirm: () => async () => true }));
|
||||
vi.mock("./FeedFiltersProvider", () => ({
|
||||
vi.mock("./confirmContext", () => ({ useConfirm: () => async () => true }));
|
||||
vi.mock("./feedFiltersContext", () => ({
|
||||
useFeedFilters: () => ({ includeNormal: true, includeShorts: false, includeLive: false }),
|
||||
}));
|
||||
vi.mock("./Feed", () => ({ default: () => null }));
|
||||
|
||||
@@ -4,7 +4,7 @@ import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { useLiveQuery } from "../lib/useLiveQuery";
|
||||
import { channelDetailPollMs, useBackfillFeedRefresh } from "../lib/channelDetailPoll";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useNavigationActions } from "./NavigationProvider";
|
||||
import { useNavigationActions } from "./navigationContext";
|
||||
import { PriorityStepper } from "./ui/PriorityStepper";
|
||||
import { useChannelPriority } from "../lib/useChannelPriority";
|
||||
import {
|
||||
@@ -24,9 +24,10 @@ import Avatar from "./ui/Avatar";
|
||||
import Tabs from "./ui/Tabs";
|
||||
import Feed from "./Feed";
|
||||
import ChannelAboutContent from "./ChannelAbout";
|
||||
import { PageToolbar, PageToolbarSlot } from "./ui/PageShell";
|
||||
import { useFeedFilters } from "./FeedFiltersProvider";
|
||||
import { useConfirm } from "./ConfirmProvider";
|
||||
import { PageToolbar } from "./ui/PageShell";
|
||||
import { PageToolbarSlot } from "./ui/pageToolbarSlot";
|
||||
import { useFeedFilters } from "./feedFiltersContext";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { notifyYouTubeActionError } from "../lib/youtubeErrors";
|
||||
import { notifyActionError } from "../lib/actionErrors";
|
||||
|
||||
@@ -19,7 +19,7 @@ vi.mock("react-i18next", async (importOriginal) => {
|
||||
|
||||
import ChannelStatsPopover from "./ChannelStatsPopover";
|
||||
import { type MyStatus } from "../lib/api";
|
||||
import { formatEta } from "../lib/format";
|
||||
import { formatEta, formatNumber } from "../lib/format";
|
||||
|
||||
beforeAll(() => {
|
||||
class Stub {
|
||||
@@ -74,7 +74,7 @@ describe("ChannelStatsPopover", () => {
|
||||
expect(screen.getByText("channels.stats.recentSynced")).toBeTruthy();
|
||||
expect(screen.getByText("3/10")).toBeTruthy(); // recent-synced ratio
|
||||
expect(screen.getByText("1/4")).toBeTruthy(); // full-history ratio
|
||||
expect(screen.getByText((1234).toLocaleString())).toBeTruthy(); // my_videos, locale-agnostic
|
||||
expect(screen.getByText(formatNumber(1234))).toBeTruthy(); // my_videos, locale-agnostic
|
||||
});
|
||||
|
||||
it("shows the 'left' ETA row only while a backfill is pending, with the formatted ETA value", () => {
|
||||
|
||||
@@ -11,8 +11,9 @@ import {
|
||||
} from "lucide-react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { type MyStatus } from "../lib/api";
|
||||
import { formatEta } from "../lib/format";
|
||||
import { PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
|
||||
import { formatEta, formatNumber } from "../lib/format";
|
||||
import { PopoverSurface } from "./ui/AnchoredPopover";
|
||||
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
|
||||
|
||||
// The channel-manager catalog stats, collapsed from a crowded inline row into a compact "Stats" chip
|
||||
// that reveals the numbers on hover (reuses the AnchoredPopover hover mode). One row per stat, each
|
||||
@@ -63,13 +64,13 @@ export default function ChannelStatsPopover({ status }: { status: MyStatus }) {
|
||||
{
|
||||
icon: Film,
|
||||
label: t("channels.stats.myVideos"),
|
||||
value: status.my_videos.toLocaleString(),
|
||||
value: formatNumber(status.my_videos),
|
||||
hint: t("channels.stats.myVideosHint"),
|
||||
},
|
||||
{
|
||||
icon: Zap,
|
||||
label: t("channels.stats.quotaLeft"),
|
||||
value: status.quota_remaining_today.toLocaleString(),
|
||||
value: formatNumber(status.quota_remaining_today),
|
||||
hint: t("channels.stats.quotaLeftHint"),
|
||||
},
|
||||
];
|
||||
|
||||
@@ -2,7 +2,8 @@ import { useEffect, useRef, useState } from "react";
|
||||
import { Check, Plus, Sparkles, X } from "lucide-react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { type Tag } from "../lib/api";
|
||||
import { useAnchoredPopover, PopoverSurface, PopoverItem } from "./ui/AnchoredPopover";
|
||||
import { PopoverSurface, PopoverItem } from "./ui/AnchoredPopover";
|
||||
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
|
||||
import { useScrollFade } from "../lib/useScrollFade";
|
||||
import { fitChipsWithPill } from "../lib/chipOverflow";
|
||||
import { chipLaneClass, useChipOverflowFit } from "../lib/useChipOverflowFit";
|
||||
|
||||
@@ -36,11 +36,11 @@ import { useCardPager } from "../lib/useCardPager";
|
||||
import { PageToolbar } from "./ui/PageShell";
|
||||
import ChannelDiscovery from "./ChannelDiscovery";
|
||||
import TagManager from "./TagManager";
|
||||
import { useConfirm } from "./ConfirmProvider";
|
||||
import { useChannelLayout, useChannels, useChannelsActions } from "./ChannelsProvider";
|
||||
import { useNavigationActions } from "./NavigationProvider";
|
||||
import { useFeedFilters, useFeedFiltersActions } from "./FeedFiltersProvider";
|
||||
import { useWizard } from "./WizardProvider";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import { useChannelLayout, useChannels, useChannelsActions } from "./channelsContext";
|
||||
import { useNavigationActions } from "./navigationContext";
|
||||
import { useFeedFilters, useFeedFiltersActions } from "./feedFiltersContext";
|
||||
import { useWizard } from "./wizardContext";
|
||||
import { useMe } from "../lib/useMe";
|
||||
|
||||
export type ChannelsView = "subscribed" | "discovery" | "hidden" | "blocked";
|
||||
@@ -292,7 +292,7 @@ export default function Channels() {
|
||||
if (filtersResetToken === resetRef.current) return;
|
||||
resetRef.current = filtersResetToken;
|
||||
setSearch("");
|
||||
}, [filtersResetToken]);
|
||||
}, [filtersResetToken, setSearch]);
|
||||
|
||||
// Tag/priority/text filters stay controlled COLUMN filters (`filters.tags` etc.) shared by the top
|
||||
// chips, the column funnels and the cards. The sync-status filter (`filters.sync`) is no longer a
|
||||
|
||||
@@ -2,21 +2,22 @@
|
||||
//
|
||||
// ChannelsProvider's tab-history behaviour: setView pushes a `_chTab` history entry + persists the tab,
|
||||
// and a genuine browser Back (popstate carrying `_chTab`) restores + re-persists it. This wiring was
|
||||
// re-routed through usePersistedTab in R13 S2b, so it's exercised here rather than only by hand E2E.
|
||||
// re-routed through useAccountPersistedState in R13 S2b, so it's exercised here rather than only by hand E2E.
|
||||
//
|
||||
// Uses REAL storage (jsdom localStorage/sessionStorage) with the tab's account pinned to 1, so
|
||||
// usePersistedTab / useSortState / useFilterState all resolve their account-scoped keys; only
|
||||
// NavigationProvider (the sole runtime dependency) is mocked, pinned to the channels page.
|
||||
// useAccountPersistedState / useSortState / useFilterState all resolve their account-scoped keys; only
|
||||
// navigationContext (the sole runtime dependency) is mocked, pinned to the channels page.
|
||||
import { describe, it, expect, beforeEach, vi } from "vitest";
|
||||
import { renderHook, act } from "@testing-library/react";
|
||||
import type { ReactNode } from "react";
|
||||
|
||||
vi.mock("./NavigationProvider", () => ({
|
||||
vi.mock("./navigationContext", () => ({
|
||||
useNavigation: () => ({ page: "channels" }),
|
||||
useNavigationActions: () => ({ setPage: vi.fn() }),
|
||||
}));
|
||||
|
||||
import { ChannelsProvider, useChannels, useChannelsActions } from "./ChannelsProvider";
|
||||
import { ChannelsProvider } from "./ChannelsProvider";
|
||||
import { useChannels, useChannelsActions } from "./channelsContext";
|
||||
|
||||
const wrapper = ({ children }: { children: ReactNode }) => (
|
||||
<ChannelsProvider>{children}</ChannelsProvider>
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
import {
|
||||
createContext,
|
||||
useCallback,
|
||||
useContext,
|
||||
useEffect,
|
||||
useLayoutEffect,
|
||||
useMemo,
|
||||
@@ -9,9 +7,9 @@ import {
|
||||
useState,
|
||||
type ReactNode,
|
||||
} from "react";
|
||||
import { getAccountRaw, LS } from "../lib/storage";
|
||||
import { parseChannelLayout, type ChannelLayout } from "../lib/channelLayout";
|
||||
import { hydrateSort, type SortState } from "../lib/columnSort";
|
||||
import { getAccountRaw, LS, useAccountPersistedState } from "../lib/storage";
|
||||
import { parseChannelLayout } from "../lib/channelLayout";
|
||||
import { hydrateSort } from "../lib/columnSort";
|
||||
import { useSortState } from "../lib/useSortState";
|
||||
import { useFilterState } from "../lib/useFilterState";
|
||||
import { useUrlMirror } from "../lib/useUrlMirror";
|
||||
@@ -22,10 +20,16 @@ import {
|
||||
hasChannelsParams,
|
||||
type ChannelsUrlState,
|
||||
} from "../lib/channelsUrlState";
|
||||
import { usePersistedTab } from "./ui/Tabs";
|
||||
import type { FilterMap } from "../lib/tableColumns";
|
||||
import { useNavigation, useNavigationActions } from "./NavigationProvider";
|
||||
import { useNavigation, useNavigationActions } from "./navigationContext";
|
||||
import type { ChannelsView } from "./Channels";
|
||||
import {
|
||||
ChannelLayoutContext,
|
||||
ChannelsActionsContext,
|
||||
ChannelsStateContext,
|
||||
type ChannelsActions,
|
||||
type ChannelsState,
|
||||
} from "./channelsContext";
|
||||
|
||||
// Owns the Channels module's shared local state — the rail-less manager's name filter, column-filter
|
||||
// map, active tab, the "drop a stale column filter" token, and the focus-a-channel intent — lifted out of
|
||||
@@ -38,6 +42,8 @@ import type { ChannelsView } from "./Channels";
|
||||
// goToFullHistory both jump to the Channels page (and force the subscriptions tab). filters / view are
|
||||
// per-account persisted (localStorage); search / focus / the reset token are ephemeral. An account
|
||||
// switch does a full reload, so init-from-storage is all the hydration the persisted ones need.
|
||||
//
|
||||
// The contexts, their value types and the hooks live in channelsContext.ts.
|
||||
|
||||
function clampView(v: string | null): ChannelsView {
|
||||
return v === "discovery" || v === "blocked" || v === "hidden" ? v : "subscribed";
|
||||
@@ -85,71 +91,6 @@ function readChannelsUrlSeed(): ChannelsUrlState | null {
|
||||
});
|
||||
}
|
||||
|
||||
type ChannelsState = {
|
||||
search: string;
|
||||
// The subscribed table's column filters, lifted out of DataTable so the top status chips, the Sync
|
||||
// column funnel, and the card layout share ONE filter state (the sync chip lives at `filters.sync`).
|
||||
filters: FilterMap;
|
||||
view: ChannelsView;
|
||||
// The subscribed table's sort, lifted out of DataTable so the card layout shares it.
|
||||
sort: SortState;
|
||||
// Bumped to tell the manager to drop a stale column filter when an intent sends the user to a
|
||||
// specific set (the header's "without full history" link).
|
||||
filtersResetToken: number;
|
||||
focusName: string | null;
|
||||
// Bumped every time a focus intent is (re-)issued, so re-focusing the same channel re-seeds the
|
||||
// manager's name filter even when the name is unchanged.
|
||||
focusToken: number;
|
||||
};
|
||||
|
||||
interface ChannelsActions {
|
||||
setSearch: (q: string) => void;
|
||||
setFilters: (f: FilterMap) => void;
|
||||
setView: (v: ChannelsView) => void;
|
||||
setLayout: (l: ChannelLayout) => void;
|
||||
setSort: (s: SortState) => void;
|
||||
// "Focus this channel in the manager": jump to Channels (subscriptions tab) and seed its name
|
||||
// filter so the channel is isolated. Called from the feed sidebar and the notifications inbox.
|
||||
focusChannel: (name: string) => void;
|
||||
// Jump to the manager filtered to channels still missing full history (the header sync chip).
|
||||
goToFullHistory: () => void;
|
||||
}
|
||||
|
||||
const StateContext = createContext<ChannelsState>({
|
||||
search: "",
|
||||
filters: {},
|
||||
view: "subscribed",
|
||||
sort: null,
|
||||
filtersResetToken: 0,
|
||||
focusName: null,
|
||||
focusToken: 0,
|
||||
});
|
||||
// The table/cards density gets its OWN context, deliberately narrow: App reads it (to decide the
|
||||
// shell's top fade) and would otherwise re-render the whole app on every keystroke in the manager's
|
||||
// search box, since the state context above changes with it.
|
||||
const LayoutContext = createContext<ChannelLayout>("table");
|
||||
const ActionsContext = createContext<ChannelsActions>({
|
||||
setSearch: () => {},
|
||||
setFilters: () => {},
|
||||
setView: () => {},
|
||||
setLayout: () => {},
|
||||
setSort: () => {},
|
||||
focusChannel: () => {},
|
||||
goToFullHistory: () => {},
|
||||
});
|
||||
|
||||
export function useChannels(): ChannelsState {
|
||||
return useContext(StateContext);
|
||||
}
|
||||
/** The manager's table/cards density on its own — subscribe to this instead of the whole state when
|
||||
* the layout is all you need (App does, for the shell's top fade). */
|
||||
export function useChannelLayout(): ChannelLayout {
|
||||
return useContext(LayoutContext);
|
||||
}
|
||||
export function useChannelsActions(): ChannelsActions {
|
||||
return useContext(ActionsContext);
|
||||
}
|
||||
|
||||
export function ChannelsProvider({ children }: { children: ReactNode }) {
|
||||
const { page } = useNavigation();
|
||||
const { setPage } = useNavigationActions();
|
||||
@@ -158,12 +99,12 @@ export function ChannelsProvider({ children }: { children: ReactNode }) {
|
||||
// The subscribed table's column-filter map, owned here (lifted out of DataTable) so the top status
|
||||
// chips, the Sync column funnel, and the card layout share one state; useFilterState persists it.
|
||||
const [filters, setFilters] = useFilterState(LS.channelsFilterMap, migrateSyncChip);
|
||||
// view + layout are persisted tabs (usePersistedTab = the shared account-scoped tab-state hook).
|
||||
// view + layout are persisted tabs (useAccountPersistedState = the shared account-scoped tab-state hook).
|
||||
// Stored un-clamped; clampView/parseChannelLayout heal them for display. setView layers the history
|
||||
// logic (below) on top of the raw setter.
|
||||
const [viewRaw, setViewRaw] = usePersistedTab(LS.channelsView, "subscribed");
|
||||
const [viewRaw, setViewRaw] = useAccountPersistedState(LS.channelsView, "subscribed");
|
||||
const view = clampView(viewRaw);
|
||||
const [layoutRaw, setLayout] = usePersistedTab(LS.channelLayout, "table");
|
||||
const [layoutRaw, setLayout] = useAccountPersistedState(LS.channelLayout, "table");
|
||||
const layout = parseChannelLayout(layoutRaw);
|
||||
// Sort is owned here (lifted out of DataTable) so the card layout shares it; ColumnSortControl heals
|
||||
// it against the live columns for display, so the provider stores it un-healed (no options passed).
|
||||
@@ -203,7 +144,7 @@ export function ChannelsProvider({ children }: { children: ReactNode }) {
|
||||
// Re-selecting the tab you are already on is not a navigation — bail before pushing, or clicking
|
||||
// the active tab (and goToFullHistory forcing "subscribed") stacks Back entries that do nothing.
|
||||
if (v === viewRef.current) return;
|
||||
setViewRaw(v); // persists (via usePersistedTab)
|
||||
setViewRaw(v); // persists (via useAccountPersistedState)
|
||||
// Only own an entry while the manager is already on screen. Arriving from elsewhere
|
||||
// (focusChannel / goToFullHistory) lets setPage push the entry — the stamp effect labels it — so
|
||||
// one navigation never costs two Back presses.
|
||||
@@ -282,10 +223,10 @@ export function ChannelsProvider({ children }: { children: ReactNode }) {
|
||||
useUrlMirror(channelsParamUpdates(page === "channels" ? { view, sort, filters } : null));
|
||||
|
||||
return (
|
||||
<ActionsContext.Provider value={actions}>
|
||||
<LayoutContext.Provider value={layout}>
|
||||
<StateContext.Provider value={state}>{children}</StateContext.Provider>
|
||||
</LayoutContext.Provider>
|
||||
</ActionsContext.Provider>
|
||||
<ChannelsActionsContext.Provider value={actions}>
|
||||
<ChannelLayoutContext.Provider value={layout}>
|
||||
<ChannelsStateContext.Provider value={state}>{children}</ChannelsStateContext.Provider>
|
||||
</ChannelLayoutContext.Provider>
|
||||
</ChannelsActionsContext.Provider>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
// @vitest-environment jsdom
|
||||
//
|
||||
// A closed-set config key is picked, not typed: a short list renders as a segmented bar, a long one as
|
||||
// a dropdown, and the pick is what Save sends. A cleared draft (reset to the default) shows the default
|
||||
// as the active option rather than no option at all.
|
||||
//
|
||||
// i18n is a key-passthrough, except for an option's label: a full key would distort the
|
||||
// bar-or-dropdown choice, which weighs the real label length, so a choice id resolves to its value
|
||||
// under a one-character marker. The marker is what makes a lost lookup visible — rendering the bare
|
||||
// value instead of the id would be indistinguishable from `label: c`. The admin config API is stubbed.
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import type { ConfigItem } from "../lib/api";
|
||||
|
||||
vi.mock("react-i18next", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("react-i18next")>();
|
||||
return {
|
||||
...actual,
|
||||
useTranslation: () => ({
|
||||
// Only an id the fixture really defines resolves: a dropped lookup renders the bare value and
|
||||
// a malformed id renders the key (long enough to turn every bar into a dropdown), so the
|
||||
// assertions below see both.
|
||||
t: (key: string) => CHOICE_LABELS[key] ?? key,
|
||||
i18n: { language: "en" },
|
||||
}),
|
||||
};
|
||||
});
|
||||
|
||||
const choice = (key: string, choices: string[], value: string, is_set = false): ConfigItem => ({
|
||||
key,
|
||||
type: "choice",
|
||||
group: "g",
|
||||
secret: false,
|
||||
min: null,
|
||||
max: null,
|
||||
is_set,
|
||||
allow_empty: false,
|
||||
choices,
|
||||
value,
|
||||
default: choices[0],
|
||||
});
|
||||
|
||||
const PRESETS = ["ultrafast", "superfast", "veryfast", "faster", "fast", "medium", "slow"];
|
||||
const LEVELS = ["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"];
|
||||
const CHOICE_LABELS: Record<string, string> = Object.fromEntries(
|
||||
(
|
||||
[
|
||||
["plex_transcode_hw", ["auto", "on", "off"]],
|
||||
["plex_transcode_preset", PRESETS],
|
||||
["log_level", LEVELS],
|
||||
] as const
|
||||
).flatMap(([key, choices]) =>
|
||||
choices.map((c) => [`config.choices.${key}.${c}`, `~${c}`] as const),
|
||||
),
|
||||
);
|
||||
const setConfig = vi.fn((_key: string, _value: unknown) => Promise.resolve({}));
|
||||
const resetConfig = vi.fn((_key: string) => Promise.resolve({}));
|
||||
|
||||
vi.mock("../lib/api", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../lib/api")>();
|
||||
return {
|
||||
...actual,
|
||||
api: {
|
||||
...actual.api,
|
||||
adminConfig: () =>
|
||||
Promise.resolve({
|
||||
groups: {
|
||||
g: [
|
||||
choice("plex_transcode_hw", ["auto", "on", "off"], "on", true),
|
||||
choice("plex_transcode_preset", PRESETS, "veryfast"),
|
||||
choice("log_level", LEVELS, "INFO"),
|
||||
],
|
||||
},
|
||||
secrets_manageable: true,
|
||||
}),
|
||||
setConfig: (key: string, value: unknown) => setConfig(key, value),
|
||||
resetConfig: (key: string) => resetConfig(key),
|
||||
},
|
||||
};
|
||||
});
|
||||
vi.mock("../lib/useMe", () => ({ useMe: () => ({ email_configured: true }) }));
|
||||
|
||||
import ConfigPanel from "./ConfigPanel";
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
const mount = () =>
|
||||
render(
|
||||
<QueryClientProvider client={new QueryClient()}>
|
||||
<ConfigPanel />
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
|
||||
describe("ConfigPanel choice keys", () => {
|
||||
it("renders a short list as a segmented bar and saves the pick", async () => {
|
||||
mount();
|
||||
const bar = await screen.findByRole("group", { name: "config.fields.plex_transcode_hw.label" });
|
||||
const pressed = () =>
|
||||
[...bar.querySelectorAll('[aria-pressed="true"]')].map((b) => b.textContent);
|
||||
expect(pressed()).toEqual(["~on"]);
|
||||
expect(screen.queryByRole("textbox")).toBeNull(); // no free-text field for a choice key
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: "~off" }));
|
||||
fireEvent.click(await screen.findByRole("button", { name: "config.save" }));
|
||||
await waitFor(() => expect(setConfig).toHaveBeenCalledWith("plex_transcode_hw", "off"));
|
||||
});
|
||||
|
||||
it("shows the default as active once the override is reset, and Save resets it", async () => {
|
||||
mount();
|
||||
await screen.findByRole("group", { name: "config.fields.plex_transcode_hw.label" });
|
||||
fireEvent.click(screen.getByRole("button", { name: /config\.reset$/ }));
|
||||
expect(screen.getByRole("button", { name: "~auto" }).getAttribute("aria-pressed")).toBe("true");
|
||||
fireEvent.click(await screen.findByRole("button", { name: "config.save" }));
|
||||
await waitFor(() => expect(resetConfig).toHaveBeenCalledWith("plex_transcode_hw"));
|
||||
});
|
||||
|
||||
it("renders a long list as a dropdown showing only the value", async () => {
|
||||
mount();
|
||||
const trigger = await screen.findByRole("button", {
|
||||
name: "config.fields.plex_transcode_preset.label: ~veryfast",
|
||||
});
|
||||
expect(trigger.textContent).toBe("~veryfast");
|
||||
fireEvent.click(trigger);
|
||||
// The popover rows are queried as FilterSelect.dom.test does (`hidden: true`, picked by text).
|
||||
const rows = screen.getAllByRole("menuitemradio", { hidden: true });
|
||||
fireEvent.click(rows.find((r) => r.textContent === "~slow")!);
|
||||
fireEvent.click(await screen.findByRole("button", { name: "config.save" }));
|
||||
await waitFor(() => expect(setConfig).toHaveBeenCalledWith("plex_transcode_preset", "slow"));
|
||||
});
|
||||
|
||||
it("a few options with long labels are a dropdown too: a bar that wide does not fit a phone", async () => {
|
||||
mount();
|
||||
await screen.findByRole("button", { name: "config.fields.log_level.label: ~INFO" });
|
||||
expect(screen.queryByRole("group", { name: "config.fields.log_level.label" })).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
// @vitest-environment jsdom
|
||||
//
|
||||
// E22 round-2 review: locks the stacking that makes an admin config row usable on a phone. The
|
||||
// control column cannot shrink, so beside a wide control it left the label and its hint a sliver of
|
||||
// the ~250 px a phone leaves next to the navigation rail. The sibling fix — a long-labelled choice
|
||||
// becoming a dropdown — is covered in ConfigPanel.choice.dom.test.tsx, but that one does nothing for
|
||||
// a text, number or secret key, whose `w-52` input is the wide control: this is their only fix.
|
||||
//
|
||||
// jsdom does no layout, so what is asserted is the class contract those rules rest on, the same
|
||||
// trade SearchBar.dom.test.tsx already makes for the narrow-header rules.
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { cleanup, render, screen } from "@testing-library/react";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import type { ConfigItem } from "../lib/api";
|
||||
|
||||
vi.mock("react-i18next", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("react-i18next")>();
|
||||
return {
|
||||
...actual,
|
||||
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
|
||||
};
|
||||
});
|
||||
|
||||
const text = (key: string): ConfigItem => ({
|
||||
key,
|
||||
type: "str",
|
||||
group: "g",
|
||||
secret: false,
|
||||
min: null,
|
||||
max: null,
|
||||
is_set: false,
|
||||
allow_empty: true,
|
||||
choices: [],
|
||||
value: "",
|
||||
default: "",
|
||||
});
|
||||
|
||||
vi.mock("../lib/api", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../lib/api")>();
|
||||
return {
|
||||
...actual,
|
||||
api: {
|
||||
...actual.api,
|
||||
adminConfig: () =>
|
||||
Promise.resolve({ groups: { g: [text("site_name")] }, secrets_manageable: true }),
|
||||
},
|
||||
};
|
||||
});
|
||||
vi.mock("../lib/useMe", () => ({ useMe: () => ({ email_configured: true }) }));
|
||||
|
||||
import ConfigPanel from "./ConfigPanel";
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
describe("ConfigPanel field row (E22 round 2)", () => {
|
||||
it("stacks the label above the control below `sm`, and lets the control fit the screen", async () => {
|
||||
render(
|
||||
<QueryClientProvider client={new QueryClient()}>
|
||||
<ConfigPanel />
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
const label = await screen.findByText("config.fields.site_name.label");
|
||||
const row = label.parentElement!.parentElement!; // label → the label/hint column → the row
|
||||
// One column on a phone, the two-column row only from `sm` up:
|
||||
expect(row.className).toContain("flex-col");
|
||||
expect(row.className).toContain("sm:flex-row");
|
||||
// The control column may not out-width the screen, and only stops shrinking from `sm` up:
|
||||
const control = row.lastElementChild as HTMLElement;
|
||||
expect(control.className).toContain("max-w-full");
|
||||
expect(control.className).toContain("sm:shrink-0");
|
||||
expect(control.className).not.toMatch(/(^|\s)shrink-0(\s|$)/);
|
||||
});
|
||||
});
|
||||
@@ -8,34 +8,21 @@ import { notify } from "../lib/notifications";
|
||||
import Tooltip from "./ui/Tooltip";
|
||||
import InlineNotice from "./ui/InlineNotice";
|
||||
import { useMe } from "../lib/useMe";
|
||||
import Tabs, { usePersistedTab } from "./ui/Tabs";
|
||||
import Tabs from "./ui/Tabs";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import { PageToolbar } from "./ui/PageShell";
|
||||
import { Switch } from "./ui/form";
|
||||
import SegmentedToggle from "./ui/SegmentedToggle";
|
||||
import FilterSelect from "./ui/FilterSelect";
|
||||
import { DraftSaveBar, type SaveState } from "./ui/DraftSaveBar";
|
||||
import { LS } from "../lib/storage";
|
||||
import { LS, useAccountPersistedState } from "../lib/storage";
|
||||
import { notifyActionError } from "../lib/actionErrors";
|
||||
|
||||
// Admin Configuration page: edit DB-overridable operational settings (registry-driven from the
|
||||
// backend — adding a key there makes it appear here automatically). Edits are drafted and
|
||||
// applied together via one Save/Discard bar; nothing hits
|
||||
// the server until Save. Group order is fixed where known so logically related settings (e.g.
|
||||
// Email/SMTP) stay together.
|
||||
const GROUP_ORDER = [
|
||||
"access",
|
||||
"email",
|
||||
"youtube",
|
||||
"google",
|
||||
"quota",
|
||||
"backfill",
|
||||
"shorts",
|
||||
"batch",
|
||||
"maintenance",
|
||||
"downloads",
|
||||
"audit",
|
||||
"plex",
|
||||
"tmdb",
|
||||
];
|
||||
// applied together via one Save/Discard bar; nothing hits the server until Save. The tabs follow the
|
||||
// registry's own order (`sysconfig.SPECS`), so a new group lands where the backend puts it instead of
|
||||
// needing a second, hand-kept order list here.
|
||||
|
||||
export default function ConfigPanel() {
|
||||
const { t } = useTranslation();
|
||||
@@ -64,7 +51,7 @@ export default function ConfigPanel() {
|
||||
const [reseedToken, setReseedToken] = useState(0);
|
||||
// One tab per config group (persisted). Called before the loading guard so hook order is
|
||||
// stable; the active id is clamped to a real group at render time once data has loaded.
|
||||
const [tab, setTab] = usePersistedTab(LS.configTab);
|
||||
const [tab, setTab] = useAccountPersistedState(LS.configTab);
|
||||
|
||||
// A stable signature of the config SHAPE — changes only when keys are added/removed, not when an
|
||||
// unchanged refetch produces a new baseline object. Re-seed the draft on a shape change (initial
|
||||
@@ -171,9 +158,7 @@ export default function ConfigPanel() {
|
||||
return <LoadingState label={t("config.loading")} />;
|
||||
}
|
||||
|
||||
const groupKeys = Object.keys(data.groups).sort(
|
||||
(a, b) => (GROUP_ORDER.indexOf(a) + 1 || 99) - (GROUP_ORDER.indexOf(b) + 1 || 99),
|
||||
);
|
||||
const groupKeys = Object.keys(data.groups);
|
||||
// Clamp the persisted tab to a real group (the registry can change between sessions). The
|
||||
// tab pill carries a dot when that group has unsaved edits, so a draft in a hidden tab is
|
||||
// never silently lost behind the global Save bar.
|
||||
@@ -330,6 +315,7 @@ function Field({
|
||||
const { t } = useTranslation();
|
||||
const isNum = item.type === "int";
|
||||
const isBool = item.type === "bool";
|
||||
const label = t(`config.fields.${item.key}.label`, item.key);
|
||||
const secretDisabled = item.secret && !secretsManageable;
|
||||
|
||||
// Status line: never reveal a secret's value.
|
||||
@@ -354,22 +340,26 @@ function Field({
|
||||
const showReset = item.is_set && !isBool;
|
||||
|
||||
return (
|
||||
<div className="flex items-start justify-between gap-3 py-3">
|
||||
// Stacked below `sm`: the control column cannot shrink, and beside a wide control it left the
|
||||
// label and hint a sliver of a phone's width.
|
||||
<div className="flex flex-col gap-2 py-3 sm:flex-row sm:items-start sm:justify-between sm:gap-3">
|
||||
<div className="min-w-0 flex-1">
|
||||
<div className="text-sm font-medium">{t(`config.fields.${item.key}.label`, item.key)}</div>
|
||||
<div className="text-sm font-medium">{label}</div>
|
||||
<p className="text-xs text-muted leading-relaxed mt-0.5">
|
||||
{t(`config.fields.${item.key}.hint`, "")}
|
||||
</p>
|
||||
<p className="text-2xs text-muted mt-1">{status}</p>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col items-end gap-1.5 shrink-0">
|
||||
<div className="flex max-w-full flex-col items-start gap-1.5 sm:shrink-0 sm:items-end">
|
||||
{isBool ? (
|
||||
<Switch
|
||||
label={t(`config.fields.${item.key}.label`, item.key)}
|
||||
label={label}
|
||||
checked={value === "true"}
|
||||
onChange={(v) => onChange(v ? "true" : "false")}
|
||||
/>
|
||||
) : item.type === "choice" ? (
|
||||
<ChoiceField item={item} label={label} value={value} onChange={onChange} />
|
||||
) : (
|
||||
<input
|
||||
type={item.secret ? "password" : isNum ? "number" : "text"}
|
||||
@@ -379,7 +369,7 @@ function Field({
|
||||
min={isNum && item.min != null ? item.min : undefined}
|
||||
max={isNum && item.max != null ? item.max : undefined}
|
||||
placeholder={item.secret ? "••••••••" : String(item.default ?? "")}
|
||||
aria-label={t(`config.fields.${item.key}.label`, item.key)}
|
||||
aria-label={label}
|
||||
className="w-52 bg-surface border border-border rounded-xl px-3 py-1.5 text-sm focus:outline-hidden focus:ring-2 focus:ring-accent/40 disabled:opacity-50"
|
||||
/>
|
||||
)}
|
||||
@@ -400,3 +390,53 @@ function Field({
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
// A closed-set key picks from its options instead of taking free text: a segmented bar for a short
|
||||
// list, a dropdown past six (the house rule for chip vs. select). A cleared draft ("" = reset to the
|
||||
// default on Save) shows the default as the active option, as a text field shows it as placeholder.
|
||||
//
|
||||
// "Short" is also about the words, not just their number: a bar cannot shrink or wrap, and five level
|
||||
// names (TRACE … WARNING, ~300 px) are wider than a phone leaves beside the navigation rail (~250 px).
|
||||
// Past MAX_SEGMENT_CHARS of labels, in the language shown, it is a dropdown too.
|
||||
const MAX_SEGMENTS = 6;
|
||||
const MAX_SEGMENT_CHARS = 22;
|
||||
|
||||
function ChoiceField({
|
||||
item,
|
||||
label,
|
||||
value,
|
||||
onChange,
|
||||
}: {
|
||||
item: ConfigItem;
|
||||
label: string;
|
||||
value: string;
|
||||
onChange: (v: string) => void;
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
const options = item.choices.map((c) => ({
|
||||
value: c,
|
||||
label: t(`config.choices.${item.key}.${c}`, c),
|
||||
}));
|
||||
const current = value || String(item.default ?? "");
|
||||
const labelChars = options.reduce((sum, o) => sum + o.label.length, 0);
|
||||
if (options.length <= MAX_SEGMENTS && labelChars <= MAX_SEGMENT_CHARS) {
|
||||
return (
|
||||
<SegmentedToggle
|
||||
options={options}
|
||||
value={current}
|
||||
onChange={onChange}
|
||||
ariaLabel={label}
|
||||
size="md"
|
||||
/>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<FilterSelect
|
||||
label={label}
|
||||
hideLabel
|
||||
options={options}
|
||||
selected={[current]}
|
||||
onChange={([next]) => next && onChange(next)}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,65 +1,10 @@
|
||||
import { createContext, useCallback, useContext, useState, type ReactNode } from "react";
|
||||
import { useCallback, useState, type ReactNode } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import Modal from "./ui/Modal";
|
||||
import { Button } from "./ui/Button";
|
||||
import { ChoiceContext, type ChoiceFn, type ChoiceOptions } from "./confirmContext";
|
||||
|
||||
// App-styled, promise-based replacement for window.confirm. Wrap the tree in
|
||||
// <ConfirmProvider> once, then anywhere: const confirm = useConfirm();
|
||||
// if (await confirm({ message: "…", danger: true })) { … }
|
||||
//
|
||||
// For a decision with more than two outcomes use `useChoice()` instead — never chain two
|
||||
// confirms, because the second dialog's Cancel/Escape/backdrop is then indistinguishable from
|
||||
// a deliberate answer (the playlist delete used to read a dismissal as "delete here only" and
|
||||
// went ahead and deleted).
|
||||
interface ConfirmOptions {
|
||||
title?: string;
|
||||
message: ReactNode;
|
||||
confirmLabel?: string;
|
||||
cancelLabel?: string;
|
||||
danger?: boolean;
|
||||
}
|
||||
|
||||
interface Choice {
|
||||
id: string;
|
||||
label: string;
|
||||
danger?: boolean;
|
||||
}
|
||||
|
||||
export interface ChoiceOptions {
|
||||
title?: string;
|
||||
message: ReactNode;
|
||||
cancelLabel?: string;
|
||||
/** Rendered left→right after Cancel. Resolves to the clicked choice's `id`. */
|
||||
choices: Choice[];
|
||||
}
|
||||
|
||||
type ChoiceFn = (opts: ChoiceOptions) => Promise<string | null>;
|
||||
type ConfirmFn = (opts: ConfirmOptions) => Promise<boolean>;
|
||||
|
||||
const ChoiceContext = createContext<ChoiceFn>(() => Promise.resolve(null));
|
||||
|
||||
/** Two outcomes: resolves true on confirm, false on cancel/Escape/backdrop. */
|
||||
export function useConfirm(): ConfirmFn {
|
||||
const ask = useContext(ChoiceContext);
|
||||
return useCallback(
|
||||
async (opts: ConfirmOptions) => {
|
||||
const picked = await ask({
|
||||
title: opts.title,
|
||||
message: opts.message,
|
||||
cancelLabel: opts.cancelLabel,
|
||||
choices: [{ id: "confirm", label: opts.confirmLabel ?? "", danger: opts.danger }],
|
||||
});
|
||||
return picked === "confirm";
|
||||
},
|
||||
[ask],
|
||||
);
|
||||
}
|
||||
|
||||
/** Three or more outcomes. Resolves to the chosen id, or **null** when the user cancels,
|
||||
* presses Escape or clicks the backdrop — dismissal is never one of the actions. */
|
||||
export function useChoice(): ChoiceFn {
|
||||
return useContext(ChoiceContext);
|
||||
}
|
||||
// The dialog behind useConfirm()/useChoice() (see confirmContext.ts for the hooks and their contract).
|
||||
|
||||
export function ConfirmProvider({ children }: { children: ReactNode }) {
|
||||
const { t } = useTranslation();
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { useLayoutEffect, useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { CalendarDays, Check, ChevronDown } from "lucide-react";
|
||||
import { PopoverItem, PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
|
||||
import { PopoverItem, PopoverSurface } from "./ui/AnchoredPopover";
|
||||
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
|
||||
import { type FeedFilters } from "../lib/api";
|
||||
import { deriveDateKey, selectDateChip, type DatePreset } from "../lib/dateFilter";
|
||||
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
// @vitest-environment jsdom
|
||||
//
|
||||
// E22 S5: Settings → Diagnosztika. Raising the level is a consent, so choosing Részletes or Teljes
|
||||
// must change NOTHING until the consent box's own button is pressed; going back to Alap needs no
|
||||
// consent. i18n is a key-passthrough (with the count, so the retention days are visible); the level
|
||||
// endpoint is stubbed, and `me` lives in a real query cache so a saved level re-renders like in the app.
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
|
||||
import type { DiagnosticsLevel, DiagnosticsState, Me } from "../lib/api";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
|
||||
vi.mock("react-i18next", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("react-i18next")>();
|
||||
return {
|
||||
...actual,
|
||||
useTranslation: () => ({
|
||||
t: (key: string, opts?: { count?: number }) =>
|
||||
opts?.count != null ? `${key}:${opts.count}` : key,
|
||||
i18n: { language: "en" },
|
||||
}),
|
||||
};
|
||||
});
|
||||
|
||||
const setLevel = vi.fn(
|
||||
(level: DiagnosticsLevel): Promise<{ ok: true; diagnostics: DiagnosticsState }> =>
|
||||
Promise.resolve({
|
||||
ok: true,
|
||||
diagnostics: {
|
||||
level,
|
||||
until: level === "INFO" ? null : new Date(Date.now() + 86_400_000).toISOString(),
|
||||
retention_days: 7,
|
||||
},
|
||||
}),
|
||||
);
|
||||
vi.mock("../lib/api", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../lib/api")>();
|
||||
return {
|
||||
...actual,
|
||||
api: { ...actual.api, setDiagnosticsLevel: (l: DiagnosticsLevel) => setLevel(l) },
|
||||
};
|
||||
});
|
||||
|
||||
import DiagnosticsSettings from "./DiagnosticsSettings";
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
function mount(me: Partial<Me>) {
|
||||
const qc = new QueryClient();
|
||||
qc.setQueryData(qk.me(), {
|
||||
id: 1,
|
||||
is_demo: false,
|
||||
preferences: {},
|
||||
diagnostics: { level: "INFO", until: null, retention_days: 7 },
|
||||
...me,
|
||||
});
|
||||
render(
|
||||
<QueryClientProvider client={qc}>
|
||||
<DiagnosticsSettings />
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
return qc;
|
||||
}
|
||||
|
||||
const levelButton = (name: string) =>
|
||||
within(screen.getByRole("group", { name: "settings.diagnostics.level" })).getByRole("button", {
|
||||
name,
|
||||
});
|
||||
|
||||
describe("DiagnosticsSettings", () => {
|
||||
it("choosing a detailed level asks first and changes nothing until confirmed", async () => {
|
||||
mount({});
|
||||
fireEvent.click(levelButton("settings.diagnostics.levels.DEBUG"));
|
||||
const consent = screen.getByTestId("diagnostics-consent");
|
||||
expect(consent.textContent).toContain("settings.diagnostics.consent.what.DEBUG");
|
||||
expect(consent.textContent).toContain("settings.diagnostics.consent.server:7");
|
||||
expect(setLevel).not.toHaveBeenCalled();
|
||||
|
||||
fireEvent.click(
|
||||
within(consent).getByRole("button", { name: "settings.diagnostics.consent.cancel" }),
|
||||
);
|
||||
expect(screen.queryByTestId("diagnostics-consent")).toBeNull();
|
||||
expect(levelButton("settings.diagnostics.levels.INFO").getAttribute("aria-pressed")).toBe(
|
||||
"true",
|
||||
);
|
||||
expect(setLevel).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("the consent button raises the level, and the status line offers the way back", async () => {
|
||||
mount({});
|
||||
fireEvent.click(levelButton("settings.diagnostics.levels.TRACE"));
|
||||
fireEvent.click(screen.getByTestId("diagnostics-consent-confirm"));
|
||||
await waitFor(() => expect(screen.getByTestId("diagnostics-status")).toBeTruthy());
|
||||
expect(setLevel).toHaveBeenCalledWith("TRACE");
|
||||
expect(screen.queryByTestId("diagnostics-consent")).toBeNull();
|
||||
expect(levelButton("settings.diagnostics.levels.TRACE").getAttribute("aria-pressed")).toBe(
|
||||
"true",
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.diagnostics.revert" }));
|
||||
await waitFor(() => expect(screen.queryByTestId("diagnostics-status")).toBeNull());
|
||||
expect(setLevel).toHaveBeenLastCalledWith("INFO");
|
||||
});
|
||||
|
||||
it("going back to the base level needs no consent", async () => {
|
||||
const until = new Date(Date.now() + 3_600_000).toISOString();
|
||||
mount({ diagnostics: { level: "DEBUG", until, retention_days: 7 } });
|
||||
fireEvent.click(levelButton("settings.diagnostics.levels.INFO"));
|
||||
expect(screen.queryByTestId("diagnostics-consent")).toBeNull();
|
||||
await waitFor(() => expect(setLevel).toHaveBeenCalledWith("INFO"));
|
||||
});
|
||||
|
||||
it("the focus goes back to the level picker when the box it was in goes away", async () => {
|
||||
mount({});
|
||||
fireEvent.click(levelButton("settings.diagnostics.levels.DEBUG"));
|
||||
const cancel = screen.getByRole("button", { name: "settings.diagnostics.consent.cancel" });
|
||||
cancel.focus();
|
||||
fireEvent.click(cancel);
|
||||
expect(document.activeElement).toBe(levelButton("settings.diagnostics.levels.INFO"));
|
||||
|
||||
fireEvent.click(levelButton("settings.diagnostics.levels.TRACE"));
|
||||
const confirm = screen.getByTestId("diagnostics-consent-confirm");
|
||||
confirm.focus();
|
||||
fireEvent.click(confirm);
|
||||
await waitFor(() => expect(screen.getByTestId("diagnostics-status")).toBeTruthy());
|
||||
expect(document.activeElement).toBe(levelButton("settings.diagnostics.levels.TRACE"));
|
||||
|
||||
const revert = screen.getByRole("button", { name: "settings.diagnostics.revert" });
|
||||
revert.focus();
|
||||
fireEvent.click(revert);
|
||||
await waitFor(() => expect(screen.queryByTestId("diagnostics-status")).toBeNull());
|
||||
expect(document.activeElement).toBe(levelButton("settings.diagnostics.levels.INFO"));
|
||||
});
|
||||
|
||||
it("the watch outlives a render in which the element it is watching is still there", async () => {
|
||||
// The distinguishing property of the restore, and the one the browser found: it keeps looking
|
||||
// until the element is GONE, instead of spending itself on the first render after the change.
|
||||
// Chrome commits a saved level as two renders and jsdom as one, so this drives the general
|
||||
// case directly — the user moved the focus somewhere else while the request was out, and that
|
||||
// somewhere else disappears later, several renders on.
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
// Raised, so the status line's countdown is mounted and its 30 s tick keeps re-rendering.
|
||||
const until = new Date(Date.now() + 3_600_000).toISOString();
|
||||
mount({ diagnostics: { level: "DEBUG", until, retention_days: 7 } });
|
||||
fireEvent.click(levelButton("settings.diagnostics.levels.TRACE"));
|
||||
const elsewhere = document.createElement("button");
|
||||
document.body.append(elsewhere);
|
||||
elsewhere.focus();
|
||||
|
||||
// Cancel closes the box; the watch is armed on `elsewhere`, which is still in the page.
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.diagnostics.consent.cancel" }));
|
||||
expect(document.activeElement).toBe(elsewhere); // their place is kept, nothing is stolen
|
||||
|
||||
// Renders keep coming (the countdown ticks) while it is still there, and must not spend it.
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(document.activeElement).toBe(elsewhere);
|
||||
|
||||
elsewhere.remove(); // now the focus really has fallen to <body>
|
||||
expect(document.activeElement).toBe(document.body);
|
||||
await vi.advanceTimersByTimeAsync(30_000);
|
||||
expect(document.activeElement).toBe(levelButton("settings.diagnostics.levels.DEBUG"));
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("the ticking status line is not a live region", async () => {
|
||||
const until = new Date(Date.now() + 3_600_000).toISOString();
|
||||
mount({ diagnostics: { level: "DEBUG", until, retention_days: 7 } });
|
||||
const status = screen.getByTestId("diagnostics-status");
|
||||
expect(status.closest('[role="status"], [aria-live]')).toBeNull();
|
||||
});
|
||||
|
||||
it("a `me` fetch that left before the change cannot put the old level back", async () => {
|
||||
const qc = mount({});
|
||||
let answerStale: (me: unknown) => void = () => {};
|
||||
const stale = qc
|
||||
.fetchQuery({
|
||||
queryKey: qk.me(),
|
||||
queryFn: () => new Promise((resolve) => (answerStale = resolve)),
|
||||
})
|
||||
.catch(() => null);
|
||||
fireEvent.click(levelButton("settings.diagnostics.levels.TRACE"));
|
||||
fireEvent.click(screen.getByTestId("diagnostics-consent-confirm"));
|
||||
await waitFor(() => expect(screen.getByTestId("diagnostics-status")).toBeTruthy());
|
||||
answerStale({
|
||||
id: 1,
|
||||
is_demo: false,
|
||||
preferences: {},
|
||||
diagnostics: { level: "INFO", until: null, retention_days: 7 },
|
||||
});
|
||||
await stale;
|
||||
expect(qc.getQueryData<Me>(qk.me())?.diagnostics.level).toBe("TRACE");
|
||||
expect(screen.getByTestId("diagnostics-status")).toBeTruthy();
|
||||
});
|
||||
|
||||
it("a lapsed level shows as the base level", () => {
|
||||
const until = new Date(Date.now() - 60_000).toISOString();
|
||||
mount({ diagnostics: { level: "TRACE", until, retention_days: 7 } });
|
||||
expect(levelButton("settings.diagnostics.levels.INFO").getAttribute("aria-pressed")).toBe(
|
||||
"true",
|
||||
);
|
||||
expect(screen.queryByTestId("diagnostics-status")).toBeNull();
|
||||
});
|
||||
|
||||
it("the shared demo account gets an explanation instead of the picker", () => {
|
||||
mount({ is_demo: true });
|
||||
expect(screen.queryByRole("group", { name: "settings.diagnostics.level" })).toBeNull();
|
||||
expect(screen.getByText("settings.diagnostics.demo")).toBeTruthy();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,186 @@
|
||||
import { useEffect, useRef, useState } from "react";
|
||||
import { useMutation, useQueryClient } from "@tanstack/react-query";
|
||||
import { useTranslation } from "react-i18next";
|
||||
|
||||
import { api, type DiagnosticsLevel } from "../lib/api";
|
||||
import { levelInForce } from "../lib/diagnostics";
|
||||
import { updateMeCache, useMe } from "../lib/useMe";
|
||||
import { Button } from "./ui/Button";
|
||||
import InlineNotice from "./ui/InlineNotice";
|
||||
import SegmentedToggle from "./ui/SegmentedToggle";
|
||||
import { Section, SettingRow } from "./ui/form";
|
||||
import MyLogViewer from "./MyLogViewer";
|
||||
|
||||
const LEVELS: readonly DiagnosticsLevel[] = ["INFO", "DEBUG", "TRACE"];
|
||||
|
||||
/** Hours and minutes until `until`, refreshed every 30 s — the status line counts down. */
|
||||
function useRemaining(until: string | null): { h: number; m: number } | null {
|
||||
const [now, setNow] = useState(() => Date.now());
|
||||
useEffect(() => {
|
||||
if (!until) return;
|
||||
const id = window.setInterval(() => setNow(Date.now()), 30_000);
|
||||
return () => window.clearInterval(id);
|
||||
}, [until]);
|
||||
if (!until) return null;
|
||||
const minutes = Math.max(0, Math.ceil((Date.parse(until) - now) / 60_000));
|
||||
return { h: Math.floor(minutes / 60), m: minutes % 60 };
|
||||
}
|
||||
|
||||
// Settings → Diagnosztika (E22 S5): the consented diagnostics level and the tab's own log.
|
||||
//
|
||||
// There is no separate on/off switch: Alap IS off. Choosing Részletes or Teljes does not change
|
||||
// anything yet — it opens the consent box under the picker, which says what will be recorded, that
|
||||
// the server keeps it and for how long, and that it lapses; only its button raises the level.
|
||||
// Returning to Alap needs no consent and happens at once.
|
||||
export default function DiagnosticsSettings() {
|
||||
const { t } = useTranslation();
|
||||
const me = useMe();
|
||||
const qc = useQueryClient();
|
||||
const state = me.diagnostics;
|
||||
const current = levelInForce(state);
|
||||
const [pending, setPending] = useState<DiagnosticsLevel | null>(null);
|
||||
const remaining = useRemaining(current === "INFO" ? null : (state?.until ?? null));
|
||||
|
||||
// The consent box and the status line each go away with the button that closed them — the one
|
||||
// holding the focus. Put it back on the level picker (its chosen option) rather than let it fall
|
||||
// to <body>, where the next Tab starts over from the top of the page. Only when it did fall: a
|
||||
// user who moved on while the request was out keeps their place.
|
||||
//
|
||||
// Watched until that button is actually GONE, not for one render: a saved level arrives as two
|
||||
// updates (the `me` cache, then the closed box), and the first render still has the button in the
|
||||
// page — measured in Chrome, where a one-shot check spent itself there and the focus then fell.
|
||||
const pickerRef = useRef<HTMLDivElement>(null);
|
||||
const closedFrom = useRef<Element | null>(null);
|
||||
useEffect(() => {
|
||||
const from = closedFrom.current;
|
||||
if (!from || from.isConnected) return;
|
||||
closedFrom.current = null;
|
||||
if (document.activeElement && document.activeElement !== document.body) return;
|
||||
pickerRef.current?.querySelector<HTMLElement>('[aria-pressed="true"]')?.focus();
|
||||
});
|
||||
const watchFocus = () => {
|
||||
closedFrom.current = document.activeElement;
|
||||
};
|
||||
|
||||
const change = useMutation({
|
||||
mutationFn: api.setDiagnosticsLevel,
|
||||
meta: { failTitle: "settings.diagnostics.failed" },
|
||||
onSuccess: (result) => {
|
||||
watchFocus();
|
||||
updateMeCache(qc, (prev) => ({ ...prev, diagnostics: result.diagnostics }));
|
||||
setPending(null);
|
||||
},
|
||||
});
|
||||
|
||||
const cancel = () => {
|
||||
watchFocus();
|
||||
setPending(null);
|
||||
};
|
||||
|
||||
const choose = (level: DiagnosticsLevel) => {
|
||||
if (level === "INFO") {
|
||||
setPending(null);
|
||||
if (current !== "INFO") change.mutate("INFO");
|
||||
} else {
|
||||
setPending(level === current ? null : level);
|
||||
}
|
||||
};
|
||||
|
||||
const levelName = (level: DiagnosticsLevel) => t(`settings.diagnostics.levels.${level}`);
|
||||
|
||||
return (
|
||||
<>
|
||||
<Section title={t("settings.diagnostics.title")}>
|
||||
{me.is_demo ? (
|
||||
<InlineNotice>{t("settings.diagnostics.demo")}</InlineNotice>
|
||||
) : (
|
||||
<SettingRow
|
||||
label={t("settings.diagnostics.level")}
|
||||
hint={t("settings.diagnostics.levelHint")}
|
||||
>
|
||||
<div ref={pickerRef}>
|
||||
<SegmentedToggle
|
||||
size="md"
|
||||
ariaLabel={t("settings.diagnostics.level")}
|
||||
value={pending ?? current}
|
||||
onChange={choose}
|
||||
options={LEVELS.map((level) => ({ value: level, label: levelName(level) }))}
|
||||
/>
|
||||
</div>
|
||||
</SettingRow>
|
||||
)}
|
||||
|
||||
{pending && (
|
||||
<div
|
||||
data-testid="diagnostics-consent"
|
||||
className="mt-2 rounded-xl border border-border bg-accent/5 p-3 text-sm"
|
||||
>
|
||||
<p className="font-semibold">{t(`settings.diagnostics.consent.title.${pending}`)}</p>
|
||||
<ul className="mt-1.5 mb-3 list-disc space-y-1 pl-5 text-muted">
|
||||
<li>{t(`settings.diagnostics.consent.what.${pending}`)}</li>
|
||||
<li>
|
||||
{t("settings.diagnostics.consent.server", {
|
||||
count: state?.retention_days ?? 7,
|
||||
})}
|
||||
</li>
|
||||
<li>{t("settings.diagnostics.consent.never")}</li>
|
||||
<li>
|
||||
{t("settings.diagnostics.consent.lapse")}{" "}
|
||||
<a
|
||||
href="/privacy#diagnostics"
|
||||
target="_blank"
|
||||
rel="noreferrer"
|
||||
className="text-accent hover:underline"
|
||||
>
|
||||
{t("settings.diagnostics.consent.policy")}
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
<div className="flex flex-wrap justify-end gap-2">
|
||||
<Button variant="ghost" size="sm" onClick={cancel}>
|
||||
{t("settings.diagnostics.consent.cancel")}
|
||||
</Button>
|
||||
<Button
|
||||
size="sm"
|
||||
data-testid="diagnostics-consent-confirm"
|
||||
loading={change.isPending}
|
||||
onClick={() => change.mutate(pending)}
|
||||
>
|
||||
{t("settings.diagnostics.consent.confirm")}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{current !== "INFO" && !pending && remaining && (
|
||||
// Not a live region: the countdown changes every minute, and a screen reader would read
|
||||
// the whole line out again each time for as long as the page is open.
|
||||
<div
|
||||
data-testid="diagnostics-status"
|
||||
className="mt-2 flex flex-wrap items-center justify-between gap-2 rounded-xl border border-warn/40 bg-warn/10 px-3 py-2 text-sm"
|
||||
>
|
||||
<span className="inline-flex items-center gap-2">
|
||||
<span className="h-2 w-2 shrink-0 rounded-full bg-warn" aria-hidden />
|
||||
<span>
|
||||
{t("settings.diagnostics.status", {
|
||||
level: levelName(current),
|
||||
time:
|
||||
remaining.h > 0
|
||||
? t("settings.diagnostics.hoursMinutes", remaining)
|
||||
: t("settings.diagnostics.minutes", remaining),
|
||||
})}
|
||||
</span>
|
||||
</span>
|
||||
<Button variant="link" loading={change.isPending} onClick={() => change.mutate("INFO")}>
|
||||
{t("settings.diagnostics.revert")}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</Section>
|
||||
|
||||
<Section title={t("settings.myLog.title")}>
|
||||
<MyLogViewer />
|
||||
</Section>
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -17,7 +17,7 @@ vi.mock("react-i18next", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("react-i18next")>();
|
||||
return { ...actual, useTranslation: () => ({ t: (key: string) => key }) };
|
||||
});
|
||||
vi.mock("./ConfirmProvider", () => ({ useConfirm: () => mocks.confirm }));
|
||||
vi.mock("./confirmContext", () => ({ useConfirm: () => mocks.confirm }));
|
||||
vi.mock("../lib/useLiveQuery", () => ({
|
||||
useLiveQuery: (key: readonly unknown[]) => ({
|
||||
data: key[0] === "downloads" ? mocks.jobs : undefined,
|
||||
|
||||
@@ -30,7 +30,7 @@ const ProfileEditor = lazy(() => import("./ProfileEditor"));
|
||||
const VideoEditor = lazy(() => import("./VideoEditor"));
|
||||
const ShareDialog = lazy(() => import("./ShareDialog"));
|
||||
const LibraryPlayer = lazy(() => import("./LibraryPlayer"));
|
||||
import { useConfirm } from "./ConfirmProvider";
|
||||
import { useConfirm } from "./confirmContext";
|
||||
import { useLiveQuery } from "../lib/useLiveQuery";
|
||||
import { useMe } from "../lib/useMe";
|
||||
import { api, type DownloadJob } from "../lib/api";
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { Component, type ReactNode } from "react";
|
||||
import { Component, type ErrorInfo, type ReactNode } from "react";
|
||||
import i18n from "../i18n";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { errorAttrs, log } from "../lib/log";
|
||||
|
||||
interface Props {
|
||||
children: ReactNode;
|
||||
@@ -18,7 +19,11 @@ export default class ErrorBoundary extends Component<Props, State> {
|
||||
return { crashed: true };
|
||||
}
|
||||
|
||||
componentDidCatch(error: Error) {
|
||||
componentDidCatch(error: Error, info: ErrorInfo) {
|
||||
log.fatal("ui.crash", "the app crashed while rendering", {
|
||||
...errorAttrs(error),
|
||||
...(info.componentStack ? { component_stack: info.componentStack } : {}),
|
||||
});
|
||||
notify({
|
||||
level: "fatal",
|
||||
title: i18n.t("errors.boundary.notifTitle"),
|
||||
|
||||
@@ -44,7 +44,7 @@ import SelectedViewChip from "./SelectedViewChip";
|
||||
import { Button } from "./ui/Button";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import { PageToolbar } from "./ui/PageShell";
|
||||
import { useWizard } from "./WizardProvider";
|
||||
import { useWizard } from "./wizardContext";
|
||||
import SortControl from "./ui/SortControl";
|
||||
import FilterSelect from "./ui/FilterSelect";
|
||||
import ItemCount from "./ui/ItemCount";
|
||||
@@ -230,15 +230,15 @@ export default function Feed({
|
||||
// rows are unchanged, so clearing overrides there would make a just-hidden card flash back.
|
||||
// Only a real refetch (page count doesn't grow) is authoritative.
|
||||
const pagesLenRef = useRef(0);
|
||||
const pagesLen = query.data?.pages?.length ?? 0;
|
||||
useEffect(() => {
|
||||
const len = query.data?.pages?.length ?? 0;
|
||||
const appended = len > pagesLenRef.current;
|
||||
pagesLenRef.current = len;
|
||||
const appended = pagesLen > pagesLenRef.current;
|
||||
pagesLenRef.current = pagesLen;
|
||||
if (!appended) {
|
||||
setOverrides({});
|
||||
setSavedOverrides({});
|
||||
}
|
||||
}, [query.dataUpdatedAt]);
|
||||
}, [query.dataUpdatedAt, pagesLen]);
|
||||
|
||||
// One shared query with the sidebar (same key + filters -> React Query dedupes to a single request):
|
||||
// the feed reads `.count`, the sidebar reads `.facets` (PERF-13, replaces feedCount + facets).
|
||||
|
||||
@@ -15,7 +15,7 @@ vi.mock("react-i18next", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("react-i18next")>()),
|
||||
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
|
||||
}));
|
||||
vi.mock("./WizardProvider", () => ({ useWizard: () => ({ openWizard: vi.fn() }) }));
|
||||
vi.mock("./wizardContext", () => ({ useWizard: () => ({ openWizard: vi.fn() }) }));
|
||||
vi.mock("./VirtualFeed", () => ({
|
||||
default: ({ items }: { items: { id: string; title: string }[] }) => (
|
||||
<ol aria-label="results">
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
// legacy global `tagMode` (older builds wrote it) into the per-category `tagModes` — the same coercion
|
||||
// the saved-view and URL paths do — so an upgrading user's topic any/all match isn't silently lost.
|
||||
import { describe, it, expect, beforeEach } from "vitest";
|
||||
import { loadAccountFilters } from "./FeedFiltersProvider";
|
||||
import { loadAccountFilters } from "./feedFiltersInit";
|
||||
import { accountKey, LS } from "../lib/storage";
|
||||
|
||||
const ID = 7;
|
||||
|
||||
@@ -11,7 +11,8 @@ import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { cleanup, render } from "@testing-library/react";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { NavigationProvider } from "./NavigationProvider";
|
||||
import { FeedFiltersProvider, useFeedFilters } from "./FeedFiltersProvider";
|
||||
import { FeedFiltersProvider } from "./FeedFiltersProvider";
|
||||
import { useFeedFilters } from "./feedFiltersContext";
|
||||
import { ACTIVE_ACCOUNT_KEY } from "../lib/activeAccount";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import type { Me } from "../lib/api";
|
||||
|
||||
@@ -1,89 +1,31 @@
|
||||
import {
|
||||
createContext,
|
||||
useCallback,
|
||||
useContext,
|
||||
useEffect,
|
||||
useMemo,
|
||||
useState,
|
||||
type ReactNode,
|
||||
} from "react";
|
||||
import { useCallback, useEffect, useMemo, useState, type ReactNode } from "react";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useQueryClient } from "@tanstack/react-query";
|
||||
import {
|
||||
getActiveAccount,
|
||||
coerceTagModes,
|
||||
coerceUntagged,
|
||||
DEFAULT_TAG_MODES,
|
||||
type FeedFilters,
|
||||
type Me,
|
||||
} from "../lib/api";
|
||||
import { getActiveAccount, type FeedFilters, type Me } from "../lib/api";
|
||||
import { filterParamUpdates, hasFilterParams, paramsToFilters } from "../lib/urlState";
|
||||
import { useUrlMirror } from "../lib/useUrlMirror";
|
||||
import { accountKey, LS, readJSON, writeJSON } from "../lib/storage";
|
||||
import { useNavigation } from "./NavigationProvider";
|
||||
import { accountKey, writeJSON } from "../lib/storage";
|
||||
import { logFilterChange } from "../lib/filterLog";
|
||||
import { useNavigation } from "./navigationContext";
|
||||
import {
|
||||
FeedFiltersActionsContext,
|
||||
FeedFiltersStateContext,
|
||||
type FeedFiltersActions,
|
||||
} from "./feedFiltersContext";
|
||||
import { DEFAULT_FILTERS, FILTERS_KEY, loadAccountFilters } from "./feedFiltersInit";
|
||||
|
||||
// Owns the GLOBAL feed-filter state (the feed's scope/tags/content/sort/show), lifted out of the App
|
||||
// god component so consumers read it via a hook instead of prop-drilling `filters`/`setFilters`
|
||||
// through the tree. Mirrors NavigationProvider (split contexts + one-line hooks); composed in
|
||||
// main.tsx inside NavigationProvider, around <App/>.
|
||||
//
|
||||
// TWO contexts on purpose (mirrors NavigationProvider): a STABLE actions context (setFilters never
|
||||
// changes reference) and a CHANGING state context (the filters object). Context consumers re-render
|
||||
// only when the context they read changes — so a component that dispatches filter changes without
|
||||
// reading the current filters can subscribe to actions ONLY and skip re-rendering on every tweak.
|
||||
//
|
||||
// SCOPE NOTE: this owns the FEED's filters only. The channel detail page keeps its OWN isolated
|
||||
// filter state (ChannelPage) — it renders a channel-scoped catalog, not the global feed — and the
|
||||
// reusable <Feed> is parameterized by a `filters` prop so it can render either. Because the two live
|
||||
// in structurally separate cells, opening a channel can never leak into the global feed (bug 2).
|
||||
|
||||
const DEFAULT_FILTERS: FeedFilters = {
|
||||
tags: [],
|
||||
tagModes: DEFAULT_TAG_MODES,
|
||||
untagged: false,
|
||||
channelIds: [],
|
||||
q: "",
|
||||
sort: "newest",
|
||||
scope: "my",
|
||||
includeNormal: true,
|
||||
includeShorts: false,
|
||||
includeLive: false,
|
||||
show: "unwatched",
|
||||
};
|
||||
|
||||
const FILTERS_KEY = LS.filters;
|
||||
|
||||
// This account's own persisted filters (per-account keys — never another account's). No account
|
||||
// (signed out) → defaults.
|
||||
export function loadAccountFilters(id: number | null): FeedFilters {
|
||||
// Your last-applied filters win, so a reload keeps whatever view you're on (a saved view you
|
||||
// picked, or manual tweaks) — setFilters persists them under LS.filters on every change. A starred
|
||||
// default view seeds a fresh account that has never stored filters, but that is applied ONCE by
|
||||
// SavedViewsProvider on its views query's first resolve (which then persists here like any edit) —
|
||||
// not read from a mirror on this synchronous path, which used to race that resolve. So here we only
|
||||
// ever read the account's own stored filters, else fall back to DEFAULT_FILTERS.
|
||||
const fKey = accountKey(FILTERS_KEY, id);
|
||||
let hasStored = false;
|
||||
try {
|
||||
hasStored = !!fKey && localStorage.getItem(fKey) != null;
|
||||
} catch {
|
||||
/* localStorage may be unavailable */
|
||||
}
|
||||
// Migrate a persisted legacy global `tagMode` into the per-category `tagModes`, matching what the
|
||||
// saved-view (asFilters) and URL (paramsToFilters) paths already do. coerceTagModes MUST run on the
|
||||
// RAW stored blob, not the object merged over DEFAULT_FILTERS: the merge injects DEFAULT_TAG_MODES
|
||||
// (all "or"), which coerceTagModes prefers over the legacy scalar and would mask it.
|
||||
if (hasStored && fKey) {
|
||||
const raw = readJSON<Partial<FeedFilters>>(fKey, {});
|
||||
return {
|
||||
...DEFAULT_FILTERS,
|
||||
...raw,
|
||||
tagModes: coerceTagModes(raw),
|
||||
untagged: coerceUntagged(raw),
|
||||
};
|
||||
}
|
||||
return { ...DEFAULT_FILTERS };
|
||||
}
|
||||
//
|
||||
// The contexts + hooks live in feedFiltersContext.ts; the defaults and the per-account seed in
|
||||
// feedFiltersInit.ts.
|
||||
|
||||
// URL wins over localStorage so a pasted link reproduces the exact view.
|
||||
function loadInitialFilters(me: Me | null | undefined): FeedFilters {
|
||||
@@ -94,20 +36,6 @@ function loadInitialFilters(me: Me | null | undefined): FeedFilters {
|
||||
return me?.is_demo ? { ...f, scope: "all" } : f;
|
||||
}
|
||||
|
||||
interface FeedFiltersActions {
|
||||
setFilters: (next: FeedFilters) => void;
|
||||
}
|
||||
|
||||
const StateContext = createContext<FeedFilters>(DEFAULT_FILTERS);
|
||||
const ActionsContext = createContext<FeedFiltersActions>({ setFilters: () => {} });
|
||||
|
||||
export function useFeedFilters(): FeedFilters {
|
||||
return useContext(StateContext);
|
||||
}
|
||||
export function useFeedFiltersActions(): FeedFiltersActions {
|
||||
return useContext(ActionsContext);
|
||||
}
|
||||
|
||||
export function FeedFiltersProvider({ children }: { children: ReactNode }) {
|
||||
const qc = useQueryClient();
|
||||
// Seeded for the account this tab is — BootGate mounts this provider only once the account is
|
||||
@@ -123,6 +51,7 @@ export function FeedFiltersProvider({ children }: { children: ReactNode }) {
|
||||
|
||||
const setFilters = useCallback((next: FeedFilters) => {
|
||||
setFiltersState(next);
|
||||
logFilterChange("feed", next, next.q);
|
||||
// Persist under THIS tab's active account so filters don't bleed between accounts.
|
||||
const key = accountKey(FILTERS_KEY, getActiveAccount());
|
||||
if (key) writeJSON(key, next);
|
||||
@@ -146,8 +75,10 @@ export function FeedFiltersProvider({ children }: { children: ReactNode }) {
|
||||
const actions = useMemo<FeedFiltersActions>(() => ({ setFilters }), [setFilters]);
|
||||
|
||||
return (
|
||||
<ActionsContext.Provider value={actions}>
|
||||
<StateContext.Provider value={filters}>{children}</StateContext.Provider>
|
||||
</ActionsContext.Provider>
|
||||
<FeedFiltersActionsContext.Provider value={actions}>
|
||||
<FeedFiltersStateContext.Provider value={filters}>
|
||||
{children}
|
||||
</FeedFiltersStateContext.Provider>
|
||||
</FeedFiltersActionsContext.Provider>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { Library, User } from "lucide-react";
|
||||
import SegmentedToggle from "./ui/SegmentedToggle";
|
||||
import { useFeedFilters, useFeedFiltersActions } from "./FeedFiltersProvider";
|
||||
import { useFeedFilters, useFeedFiltersActions } from "./feedFiltersContext";
|
||||
|
||||
// The feed's Mine/Library scope, relocated from the filter sidebar to sit just right of the header
|
||||
// search box — mirroring PlexScopeToggle's placement, so the primary "what am I browsing" control is
|
||||
|
||||
@@ -1,7 +1,12 @@
|
||||
import { createContext, useContext, useMemo, type ReactNode } from "react";
|
||||
import { useMemo, type ReactNode } from "react";
|
||||
import { parseFeedView, type FeedView } from "../lib/feedView";
|
||||
import { LS } from "../lib/storage";
|
||||
import { usePref } from "../lib/usePref";
|
||||
import {
|
||||
FeedViewActionsContext,
|
||||
FeedViewStateContext,
|
||||
type FeedViewActions,
|
||||
} from "./feedViewContext";
|
||||
|
||||
// Owns the feed's view mode (the card/row/tile density switcher), lifted out of App so both the feed
|
||||
// page and the channel page read it via hooks instead of prop-drilling `view`/`setView` through App.
|
||||
@@ -10,20 +15,8 @@ import { usePref } from "../lib/usePref";
|
||||
// The view mode is an account PREFERENCE, not a filter: it persists to the server on pick and follows
|
||||
// the account across devices. usePref carries that server-sync (seeded from the server pref, with the
|
||||
// localStorage mirror as the fallback) so this provider is just the two contexts over it.
|
||||
|
||||
interface FeedViewActions {
|
||||
setView: (v: FeedView) => void;
|
||||
}
|
||||
|
||||
const StateContext = createContext<FeedView>("cards");
|
||||
const ActionsContext = createContext<FeedViewActions>({ setView: () => {} });
|
||||
|
||||
export function useFeedView(): FeedView {
|
||||
return useContext(StateContext);
|
||||
}
|
||||
export function useFeedViewActions(): FeedViewActions {
|
||||
return useContext(ActionsContext);
|
||||
}
|
||||
//
|
||||
// The contexts + hooks live in feedViewContext.ts.
|
||||
|
||||
export function FeedViewProvider({ children }: { children: ReactNode }) {
|
||||
// parseFeedView also maps the pre-0.45 "grid"/"list" values forward (idempotent for current values).
|
||||
@@ -35,8 +28,8 @@ export function FeedViewProvider({ children }: { children: ReactNode }) {
|
||||
const actions = useMemo<FeedViewActions>(() => ({ setView }), [setView]);
|
||||
|
||||
return (
|
||||
<ActionsContext.Provider value={actions}>
|
||||
<StateContext.Provider value={view}>{children}</StateContext.Provider>
|
||||
</ActionsContext.Provider>
|
||||
<FeedViewActionsContext.Provider value={actions}>
|
||||
<FeedViewStateContext.Provider value={view}>{children}</FeedViewStateContext.Provider>
|
||||
</FeedViewActionsContext.Provider>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import Overlay from "./ui/Overlay";
|
||||
import { copyText } from "../lib/clipboard";
|
||||
import { usePrefs } from "./PrefsProvider";
|
||||
import { usePrefs } from "./prefsContext";
|
||||
|
||||
/**
|
||||
* A floating live appearance tuner. Mounted at the app root so it's reachable on every page; it
|
||||
|
||||
@@ -8,10 +8,10 @@ import PlexScopeToggle from "./PlexScopeToggle";
|
||||
import PlexViewToggle from "./PlexViewToggle";
|
||||
import FeedScopeToggle from "./FeedScopeToggle";
|
||||
import SyncStatus from "./SyncStatus";
|
||||
import { useNavigation } from "./NavigationProvider";
|
||||
import { useFeedFilters, useFeedFiltersActions } from "./FeedFiltersProvider";
|
||||
import { usePlaylists, usePlaylistsActions } from "./PlaylistsProvider";
|
||||
import { useChannels, useChannelsActions } from "./ChannelsProvider";
|
||||
import { useNavigation } from "./navigationContext";
|
||||
import { useFeedFilters, useFeedFiltersActions } from "./feedFiltersContext";
|
||||
import { usePlaylists, usePlaylistsActions } from "./playlistsContext";
|
||||
import { useChannels, useChannelsActions } from "./channelsContext";
|
||||
|
||||
// The floating top header: a row of glass pills over the content — the module's SearchBar where it
|
||||
// has one, and the per-user sync chip at the right end.
|
||||
|
||||
@@ -49,6 +49,7 @@ vi.mock("../lib/api", () => ({
|
||||
setState: () => Promise.resolve(),
|
||||
watchLaterAdd: () => Promise.resolve(),
|
||||
watchLaterRemove: () => Promise.resolve(),
|
||||
downloadIndex: () => Promise.resolve({}), // the cards' download buttons read it
|
||||
},
|
||||
DISCOVER_BUDGETS: ["short", "medium", "long", "any"],
|
||||
discoverCardKey: (c: { source: string; id: string }) => `${c.source}:${c.id}`,
|
||||
@@ -58,10 +59,10 @@ vi.mock("../lib/api", () => ({
|
||||
// WHAT is handed over is: the jump target decides both what plays and where Back goes.
|
||||
const setPage = vi.fn();
|
||||
const setJumpTo = vi.fn();
|
||||
vi.mock("./NavigationProvider", () => ({
|
||||
vi.mock("./navigationContext", () => ({
|
||||
useNavigationActions: () => ({ setPage: (p: string) => setPage(p) }),
|
||||
}));
|
||||
vi.mock("./PlexProvider", () => ({
|
||||
vi.mock("./plexContext", () => ({
|
||||
usePlexActions: () => ({ setJumpTo: (t: unknown) => setJumpTo(t) }),
|
||||
}));
|
||||
|
||||
|
||||
@@ -53,6 +53,7 @@ vi.mock("../lib/api", () => ({
|
||||
setState: () => Promise.resolve(),
|
||||
watchLaterAdd: () => Promise.resolve(),
|
||||
watchLaterRemove: () => Promise.resolve(),
|
||||
downloadIndex: () => Promise.resolve({}), // the cards' download buttons read it
|
||||
comfortPins: (...a: unknown[]) => comfortPins(...a),
|
||||
addComfortPin: (...a: unknown[]) => addComfortPin(...a),
|
||||
removeComfortPin: (...a: unknown[]) => removeComfortPin(...a),
|
||||
@@ -63,8 +64,8 @@ vi.mock("../lib/api", () => ({
|
||||
discoverCardKey: (c: { source: string; id: string }) => `${c.source}:${c.id}`,
|
||||
}));
|
||||
|
||||
vi.mock("./NavigationProvider", () => ({ useNavigationActions: () => ({ setPage: vi.fn() }) }));
|
||||
vi.mock("./PlexProvider", () => ({ usePlexActions: () => ({ setJumpTo: vi.fn() }) }));
|
||||
vi.mock("./navigationContext", () => ({ useNavigationActions: () => ({ setPage: vi.fn() }) }));
|
||||
vi.mock("./plexContext", () => ({ usePlexActions: () => ({ setJumpTo: vi.fn() }) }));
|
||||
|
||||
import Home from "./Home";
|
||||
|
||||
|
||||
@@ -38,8 +38,8 @@ import SegmentedToggle from "./ui/SegmentedToggle";
|
||||
import { LoadingState, StateMessage } from "./ui/QueryState";
|
||||
import VideoCard from "./VideoCard";
|
||||
import { PlexPosterCard } from "./plexCards";
|
||||
import { useNavigationActions } from "./NavigationProvider";
|
||||
import { usePlexActions } from "./PlexProvider";
|
||||
import { useNavigationActions } from "./navigationContext";
|
||||
import { usePlexActions } from "./plexContext";
|
||||
import { navItemAttrs } from "../lib/listNav";
|
||||
|
||||
// The Home surface (E24) — "Mit nézzek most?".
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { useState } from "react";
|
||||
import { Check, Globe } from "lucide-react";
|
||||
import { LANGUAGES, type LangCode } from "../i18n";
|
||||
import { PopoverItem, PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
|
||||
import { PopoverItem, PopoverSurface } from "./ui/AnchoredPopover";
|
||||
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
|
||||
|
||||
// Compact language picker (globe + current code). Presentational: the parent decides what
|
||||
// changing the language does (set it; persist server-side when signed in).
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
import { useMemo } from "react";
|
||||
import { useCallback, useMemo } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import Modal from "./ui/Modal";
|
||||
import { api, type DownloadJob } from "../lib/api";
|
||||
import { jobTitle } from "./downloadRow";
|
||||
import { jobTitle } from "./downloadTitle";
|
||||
import { exposeVideoToBackgroundPlay } from "../lib/browserBackgroundPlay";
|
||||
import { useMediaLogRef } from "../lib/mediaLog";
|
||||
|
||||
// E7 S4b — in-app player for a completed library item. We do NOT reuse PlayerModal (a cross-origin
|
||||
// YouTube IFrame shell) or PlexPlayer (a Plex HLS/session shell): both are bound to their own domain
|
||||
@@ -30,6 +31,14 @@ export default function LibraryPlayer({ job, onClose }: { job: DownloadJob; onCl
|
||||
// Our self-hosted poster (always same-origin, never expires) is the primary; the source thumbnail is
|
||||
// the fallback. Same precedence as the library-row thumbnail.
|
||||
const poster = job.poster_url ? api.posterImageUrl(job.id) : (asset?.thumbnail_url ?? undefined);
|
||||
const logMedia = useMediaLogRef("library");
|
||||
const videoRef = useCallback(
|
||||
(el: HTMLVideoElement | null) => {
|
||||
exposeVideoToBackgroundPlay(el);
|
||||
logMedia(el);
|
||||
},
|
||||
[logMedia],
|
||||
);
|
||||
|
||||
return (
|
||||
<Modal title={jobTitle(job)} onClose={onClose} maxWidth="max-w-4xl">
|
||||
@@ -42,13 +51,13 @@ export default function LibraryPlayer({ job, onClose }: { job: DownloadJob; onCl
|
||||
className="w-40 h-40 rounded-xl object-cover border border-border"
|
||||
/>
|
||||
) : null}
|
||||
<audio src={fileUrl} controls autoPlay className="w-full">
|
||||
<audio ref={logMedia} src={fileUrl} controls autoPlay className="w-full">
|
||||
{t("downloads.player.unsupported")}
|
||||
</audio>
|
||||
</div>
|
||||
) : (
|
||||
<video
|
||||
ref={exposeVideoToBackgroundPlay}
|
||||
ref={videoRef}
|
||||
src={fileUrl}
|
||||
poster={poster}
|
||||
controls
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
// @vitest-environment jsdom
|
||||
//
|
||||
// E22 S5: "Saját napló" shows this tab's own log live, newest first, filters it by level group and
|
||||
// text, and copies exactly what is filtered (oldest first, one JSON event per line).
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { act, cleanup, fireEvent, render, screen, within } from "@testing-library/react";
|
||||
|
||||
import { clear, log, setLevel } from "../lib/log";
|
||||
|
||||
vi.mock("react-i18next", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("react-i18next")>();
|
||||
return {
|
||||
...actual,
|
||||
useTranslation: () => ({
|
||||
t: (key: string, opts?: { count?: number }) =>
|
||||
opts?.count != null ? `${key}:${opts.count}` : key,
|
||||
}),
|
||||
};
|
||||
});
|
||||
const copied: string[] = [];
|
||||
vi.mock("../lib/clipboard", () => ({
|
||||
copyText: (text: string) => {
|
||||
copied.push(text);
|
||||
return Promise.resolve(true);
|
||||
},
|
||||
}));
|
||||
|
||||
import MyLogViewer from "./MyLogViewer";
|
||||
|
||||
beforeEach(() => {
|
||||
setLevel("DEBUG");
|
||||
log.info("nav.view", "push feed", { page: "feed" });
|
||||
log.warn("http.request", "GET /api/feed 503", { status: 503 });
|
||||
log.debug("ui.click", "click feed-refresh", { control: "feed-refresh" });
|
||||
});
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
clear();
|
||||
setLevel("INFO");
|
||||
copied.length = 0;
|
||||
});
|
||||
|
||||
const rows = () =>
|
||||
within(screen.getByTestId("my-log-lines"))
|
||||
.queryAllByRole("button")
|
||||
.map((b) => b.textContent);
|
||||
|
||||
describe("MyLogViewer", () => {
|
||||
it("lists the tab's events newest first and follows new ones", async () => {
|
||||
render(<MyLogViewer />);
|
||||
expect(rows().map((r) => r!.includes("ui.click"))).toEqual([true, false, false]);
|
||||
await act(async () => {
|
||||
log.error("app.uncaught", "TypeError: x is undefined");
|
||||
await new Promise((r) => requestAnimationFrame(() => r(null)));
|
||||
});
|
||||
expect(rows()[0]).toContain("app.uncaught");
|
||||
});
|
||||
|
||||
it("a row keeps its place and the focus as new events arrive above it", async () => {
|
||||
render(<MyLogViewer />);
|
||||
const row = within(screen.getByTestId("my-log-lines")).getAllByRole("button")[0]!;
|
||||
row.focus();
|
||||
fireEvent.click(row); // at Részletes this click is itself a new event in real use
|
||||
await act(async () => {
|
||||
log.debug("ui.click", "a newer event");
|
||||
await new Promise((r) => requestAnimationFrame(() => r(null)));
|
||||
});
|
||||
expect(row.isConnected).toBe(true);
|
||||
expect(document.activeElement).toBe(row);
|
||||
expect(row.getAttribute("aria-expanded")).toBe("true");
|
||||
});
|
||||
|
||||
it("filters by level group and by text, and copies only what is shown", async () => {
|
||||
render(<MyLogViewer />);
|
||||
fireEvent.click(screen.getByRole("button", { name: /settings\.myLog\.filters\.warning/ }));
|
||||
expect(rows()).toHaveLength(1);
|
||||
expect(rows()[0]).toContain("GET /api/feed 503");
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: /settings\.myLog\.filters\.all/ }));
|
||||
fireEvent.change(screen.getByRole("searchbox"), { target: { value: "FEED-REFRESH" } });
|
||||
expect(rows()).toHaveLength(1);
|
||||
|
||||
const copy = async () => {
|
||||
await act(async () => {
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.myLog.copy" }));
|
||||
});
|
||||
return copied[copied.length - 1]!.split("\n").map((l) => JSON.parse(l).event);
|
||||
};
|
||||
expect(await copy()).toEqual(["ui.click"]); // only what the search left
|
||||
|
||||
fireEvent.change(screen.getByRole("searchbox"), { target: { value: "" } });
|
||||
expect(await copy()).toEqual(["nav.view", "http.request", "ui.click"]); // oldest first
|
||||
});
|
||||
|
||||
it("opens a row to its details and clears the log", async () => {
|
||||
render(<MyLogViewer />);
|
||||
const first = within(screen.getByTestId("my-log-lines")).getAllByRole("button")[0]!;
|
||||
fireEvent.click(first);
|
||||
expect(first.getAttribute("aria-expanded")).toBe("true");
|
||||
expect(screen.getByTestId("my-log-lines").textContent).toContain('"control": "feed-refresh"');
|
||||
|
||||
await act(async () => {
|
||||
fireEvent.click(screen.getByRole("button", { name: "settings.myLog.clear" }));
|
||||
await new Promise((r) => requestAnimationFrame(() => r(null)));
|
||||
});
|
||||
expect(rows()).toEqual([]);
|
||||
expect(screen.getByText("settings.myLog.emptyAll")).toBeTruthy();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,244 @@
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { Copy, Search } from "lucide-react";
|
||||
|
||||
import { copyText } from "../lib/clipboard";
|
||||
import { clear, events, subscribe, type Level, type LogEvent } from "../lib/log";
|
||||
import { notify } from "../lib/notifications";
|
||||
import { Button } from "./ui/Button";
|
||||
import FilterChipGroup from "./ui/FilterChipGroup";
|
||||
|
||||
type Filter = "all" | "error" | "warning" | "info" | "detail";
|
||||
|
||||
const FILTER_LEVELS: Record<Exclude<Filter, "all">, readonly Level[]> = {
|
||||
error: ["ERROR", "CRITICAL"],
|
||||
warning: ["WARNING"],
|
||||
info: ["INFO"],
|
||||
detail: ["DEBUG", "TRACE"],
|
||||
};
|
||||
|
||||
// Rows rendered at once. The ring holds up to 2000 events; drawing all of them on every new event
|
||||
// would make the settings page the slowest thing in the app while it is watching the app.
|
||||
const ROWS_SHOWN = 500;
|
||||
|
||||
const LEVEL_TAG: Record<Level, { text: string; className: string }> = {
|
||||
TRACE: { text: "TRACE", className: "text-muted opacity-80" },
|
||||
DEBUG: { text: "DEBUG", className: "text-muted" },
|
||||
INFO: { text: "INFO", className: "text-info" },
|
||||
WARNING: { text: "WARN", className: "text-warn" },
|
||||
ERROR: { text: "ERROR", className: "text-danger" },
|
||||
CRITICAL: { text: "FATAL", className: "text-danger" },
|
||||
};
|
||||
|
||||
/** The ring, live: re-read at most once per animation frame however fast events arrive. */
|
||||
function useLogEvents(): LogEvent[] {
|
||||
const [list, setList] = useState(events);
|
||||
useEffect(() => {
|
||||
let frame: number | null = null;
|
||||
const unsubscribe = subscribe(() => {
|
||||
if (frame !== null) return;
|
||||
frame = window.requestAnimationFrame(() => {
|
||||
frame = null;
|
||||
setList(events());
|
||||
});
|
||||
});
|
||||
setList(events()); // whatever arrived between the first render and the subscription
|
||||
return () => {
|
||||
unsubscribe();
|
||||
if (frame !== null) window.cancelAnimationFrame(frame);
|
||||
};
|
||||
}, []);
|
||||
return list;
|
||||
}
|
||||
|
||||
// A row's React key: the event OBJECT's own number, handed out the first time it is drawn. The ring
|
||||
// keeps the same objects, so the key stays put as the list grows at the top — an index key shifted
|
||||
// every row on every new event, remounting all of them and dropping the focus off the row button
|
||||
// (at Részletes a click on a row logs a line itself, so it happened on every click).
|
||||
const rowIds = new WeakMap<LogEvent, number>();
|
||||
let nextRowId = 0;
|
||||
const rowKey = (event: LogEvent) => {
|
||||
let id = rowIds.get(event);
|
||||
if (id === undefined) {
|
||||
id = nextRowId++;
|
||||
rowIds.set(event, id);
|
||||
}
|
||||
return id;
|
||||
};
|
||||
|
||||
const matches = (event: LogEvent, needle: string) =>
|
||||
!needle ||
|
||||
[event.event, event.msg, event.request_id ?? "", event.attrs ? JSON.stringify(event.attrs) : ""]
|
||||
.join(" ")
|
||||
.toLowerCase()
|
||||
.includes(needle);
|
||||
|
||||
const clock = (ts: string) => {
|
||||
const d = new Date(ts);
|
||||
const pad = (n: number, w = 2) => String(n).padStart(w, "0");
|
||||
return `${pad(d.getHours())}:${pad(d.getMinutes())}:${pad(d.getSeconds())}.${pad(d.getMilliseconds(), 3)}`;
|
||||
};
|
||||
|
||||
// "Saját napló" — this tab's own client log (lib/log.ts), exactly what it would send while a raised
|
||||
// diagnostics level is on: newest first, filtered by level group and free text, a row opens to its
|
||||
// correlation id and attributes, and "Copy" puts the FILTERED events on the clipboard as JSON lines
|
||||
// (one event per line) for a bug report.
|
||||
export default function MyLogViewer() {
|
||||
const { t } = useTranslation();
|
||||
const all = useLogEvents();
|
||||
const [filter, setFilter] = useState<Filter>("all");
|
||||
const [query, setQuery] = useState("");
|
||||
const [open, setOpen] = useState<LogEvent | null>(null);
|
||||
|
||||
const counts = useMemo(() => {
|
||||
const out: Record<Filter, number> = {
|
||||
all: all.length,
|
||||
error: 0,
|
||||
warning: 0,
|
||||
info: 0,
|
||||
detail: 0,
|
||||
};
|
||||
for (const e of all) {
|
||||
for (const [group, levels] of Object.entries(FILTER_LEVELS)) {
|
||||
if (levels.includes(e.level)) out[group as Filter] += 1;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}, [all]);
|
||||
|
||||
const shown = useMemo(() => {
|
||||
const needle = query.trim().toLowerCase();
|
||||
const levels = filter === "all" ? null : FILTER_LEVELS[filter];
|
||||
const out: LogEvent[] = [];
|
||||
for (let i = all.length - 1; i >= 0; i--) {
|
||||
const e = all[i]!;
|
||||
if ((!levels || levels.includes(e.level)) && matches(e, needle)) out.push(e);
|
||||
}
|
||||
return out;
|
||||
}, [all, filter, query]);
|
||||
|
||||
const copy = async () => {
|
||||
// Oldest first, as a log reads — the list shows newest first only for watching it live.
|
||||
const text = [...shown]
|
||||
.reverse()
|
||||
.map((e) => JSON.stringify(e))
|
||||
.join("\n");
|
||||
const ok = await copyText(text);
|
||||
notify(
|
||||
ok
|
||||
? { level: "success", message: t("settings.myLog.copied", { count: shown.length }) }
|
||||
: { level: "error", message: t("settings.myLog.copyFailed") },
|
||||
);
|
||||
};
|
||||
|
||||
const filters: readonly Filter[] = ["all", "error", "warning", "info", "detail"];
|
||||
|
||||
return (
|
||||
<div data-testid="my-log" className="overflow-hidden rounded-xl border border-border bg-bg">
|
||||
<p className="px-3 pt-2 text-xs text-muted">{t("settings.myLog.hint")}</p>
|
||||
<div className="flex flex-wrap items-center gap-2 border-b border-border p-2">
|
||||
<label className="flex min-w-[10rem] flex-1 items-center gap-1.5 rounded-lg border border-border bg-card px-2 py-1 text-sm focus-within:border-accent">
|
||||
<Search className="h-3.5 w-3.5 shrink-0 text-muted" aria-hidden />
|
||||
<input
|
||||
type="search"
|
||||
value={query}
|
||||
onChange={(e) => setQuery(e.target.value)}
|
||||
placeholder={t("settings.myLog.search")}
|
||||
aria-label={t("settings.myLog.search")}
|
||||
className="min-w-0 flex-1 bg-transparent outline-none"
|
||||
/>
|
||||
</label>
|
||||
<FilterChipGroup
|
||||
ariaLabel={t("settings.myLog.filterLabel")}
|
||||
value={filter}
|
||||
onChange={setFilter}
|
||||
options={filters.map((f) => ({
|
||||
value: f,
|
||||
label: t(`settings.myLog.filters.${f}`),
|
||||
count: counts[f],
|
||||
}))}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="max-h-96 overflow-auto font-mono text-2xs" data-testid="my-log-lines">
|
||||
{shown.length === 0 ? (
|
||||
<p className="px-3 py-6 text-center font-sans text-sm text-muted">
|
||||
{all.length === 0 ? t("settings.myLog.emptyAll") : t("settings.myLog.empty")}
|
||||
</p>
|
||||
) : (
|
||||
shown.slice(0, ROWS_SHOWN).map((e) => {
|
||||
const tag = LEVEL_TAG[e.level];
|
||||
const isOpen = open === e;
|
||||
return (
|
||||
<div key={rowKey(e)} className="border-b border-border/50 last:border-b-0">
|
||||
<button
|
||||
type="button"
|
||||
aria-expanded={isOpen}
|
||||
onClick={() => setOpen(isOpen ? null : e)}
|
||||
className={`grid w-full grid-cols-[6.5em_3.5em_minmax(0,1fr)] items-baseline sm:grid-cols-[6.5em_3.5em_minmax(0,10em)_minmax(0,1fr)] gap-2 px-3 py-1 text-left hover:bg-card ${
|
||||
isOpen ? "bg-accent/10" : ""
|
||||
}`}
|
||||
>
|
||||
<span className="tabular-nums text-muted">{clock(e.ts)}</span>
|
||||
<span className={`font-semibold ${tag.className}`}>{tag.text}</span>
|
||||
{/* A phone has no room for a fourth column: the code leads the message there. */}
|
||||
<span className="hidden truncate sm:block">{e.event}</span>
|
||||
<span className="truncate">
|
||||
<span className="sm:hidden">{e.event} · </span>
|
||||
{e.msg}
|
||||
</span>
|
||||
</button>
|
||||
{isOpen && (
|
||||
<pre className="overflow-x-auto whitespace-pre bg-accent/5 px-3 py-2 text-muted">
|
||||
{JSON.stringify(
|
||||
{
|
||||
event: e.event,
|
||||
msg: e.msg,
|
||||
request_id: e.request_id,
|
||||
ts: e.ts,
|
||||
tab_id: e.tab_id,
|
||||
attrs: e.attrs,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}
|
||||
</pre>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
})
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap items-center justify-between gap-2 border-t border-border px-3 py-1.5 text-xs text-muted">
|
||||
<span>
|
||||
{t("settings.myLog.count", { count: all.length })}
|
||||
{shown.length > ROWS_SHOWN &&
|
||||
` · ${t("settings.myLog.capped", { shown: ROWS_SHOWN, count: shown.length })}`}
|
||||
</span>
|
||||
<span className="flex gap-1.5">
|
||||
<Button
|
||||
variant="outline"
|
||||
size="xs"
|
||||
icon={<Copy className="h-3.5 w-3.5" />}
|
||||
disabled={shown.length === 0}
|
||||
onClick={() => void copy()}
|
||||
>
|
||||
{t("settings.myLog.copy")}
|
||||
</Button>
|
||||
<Button
|
||||
variant="outline"
|
||||
size="xs"
|
||||
disabled={all.length === 0}
|
||||
onClick={() => {
|
||||
clear();
|
||||
setOpen(null);
|
||||
}}
|
||||
>
|
||||
{t("settings.myLog.clear")}
|
||||
</Button>
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -2,7 +2,7 @@ import { useState, useSyncExternalStore } from "react";
|
||||
import { qk } from "../lib/queryKeys";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { useNavigation, useNavigationActions } from "./NavigationProvider";
|
||||
import { useNavigation, useNavigationActions } from "./navigationContext";
|
||||
import { Info, LogOut, Settings, Shield, UserPlus, type LucideIcon } from "lucide-react";
|
||||
import { api, type Me } from "../lib/api";
|
||||
import { pinOnNextPage } from "../lib/activeAccount";
|
||||
@@ -17,7 +17,8 @@ import { Button } from "./ui/Button";
|
||||
import AvatarImg from "./ui/Avatar";
|
||||
import Tooltip from "./ui/Tooltip";
|
||||
import LanguageSwitcher from "./LanguageSwitcher";
|
||||
import { PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
|
||||
import { PopoverSurface } from "./ui/AnchoredPopover";
|
||||
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
|
||||
|
||||
// Primary app navigation: a thin icon-only left rail. The modules used to hide under the avatar
|
||||
// dropdown; they now live here, each revealing its name in a hover/focus flyout label. Account
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user