Compare commits

...
73 Commits
Author SHA1 Message Date
peter 8c10e57208 release: 0.88.0 (feature/e22-activity-trail)
gate / frontend (push) Failing after 3m31s
gate / backend (push) Successful in 4m44s
2026-09-18 00:05:50 +02:00
peter dd040adc96 chore: 0.88.0 release notes
E22 (activity trail): a consented, self-expiring diagnostics level with the user's own log visible
before anything is sent, last sign-in, bilingual privacy policy and terms, an admin diagnostics
section, and pickers for every closed-set setting. Plus what the epic's measurements turned up on
the way: doubled chat delivery, Settings on a phone, locale-ignoring dates and numbers, and the
feed poll narrowed to followed channels.
2026-09-18 00:05:46 +02:00
peter 4ab128ccab Merge branch 'main' into feature/e22-activity-trail 2026-09-18 00:02:12 +02:00
peter a28193a00c test: pin the focus restore, the choice labels and the phone layout to their fix
Three round-1 fixes whose tests passed against the broken version too.

The focus restore keeps watching until the element that closed the box is GONE, rather than spending
itself on the first render after the change — the distinction Chrome exposed and jsdom did not,
because jsdom commits the save in one render where Chrome takes two. Driven now through the general
case instead: the user moved the focus elsewhere while the request was out, renders keep coming
while that element is still there, and the restore happens only once it disappears.

The choice test's i18n mock returned an option's bare value, which is exactly what dropping the
`config.choices.*` lookup renders — a Hungarian admin would have seen `on`/`off` instead of
`GPU`/`Szoftver` with the gate green. Marked labels now, still short enough not to move the
segmented-bar budget the same file measures.

The admin config row's stacking had no test at all, only its sibling fix did — and stacking is the
only fix for a text or number key, whose fixed-width input is the wide control. Class contract, as
SearchBar.dom.test.tsx already does for the narrow-header rules, since jsdom does no layout.

Also: the time-zone cleanup deleted TZ, and Node re-reads it on assignment only, so the process kept
the last zone the file set. Harmless until someone appends a test below it.
2026-09-17 23:45:41 +02:00
peter 986bc8d01e fix: a batch the user themselves cancelled is not called a lapse
Picking "Alap" revokes the consent, and a batch still in flight is then answered by the server with
the same 409 a real expiry gets. The 409 arm only checked that the consent session had not changed,
not that this tab was still streaming — so the user's own log gained a "diagnostics level lapsed"
line for a level they had just turned off themselves, plus a pointless `me` refetch. The same guard
also stops a lapse this tab has already announced being announced a second time.

Measured in Chrome by holding the POST at the network layer, switching to Alap and only then
answering 409: the line appears without the guard and not with it.

Three tests that named the session counter never reached it — they settled the refusal while the
stream was already stopped, where `unsubscribe` does the work, so the counter could be deleted from
both arms with the suite green. The refusal now settles after the re-raise, which is the ordering it
exists for. Same for the queue trim on the 429 path: the old test unshifted a batch into an empty
queue, so the trim was a no-op and only the subscribe path was ever measured.
2026-09-17 23:45:29 +02:00
peter 07e3ea3737 docs: the deep-backfill card counts the backlog, not the next pass
Two reviewers in a row read the admin card's selection against `run_deep_backfill`'s, saw it was
missing the job's `recent_synced_at` filter, and called it a card that counts channels the job will
never touch. It is the opposite: the card is the same backlog figure /api/sync/my-status shows the
user, and a channel still missing its recent pull is one the same job tick fetches first. Narrowing
it would hide genuinely queued work. Written down where the next reader will look.
2026-09-17 23:45:29 +02:00
peter b2eaa9e00b fix: a credential, an erased account and a stale level stay out of the log
Round 2 of the E22 review, on what round 1 changed.

`redact_query` decodes a parameter before the text rules see it, which is what lets `%40` be
recognised as an address — but `_SECRET_PARAM_RE` was written for the raw, encoded form, where a
value holds no space, `&` or quote. On decoded text its value class stops at the first one, so
`?state=a%20b` kept its tail and `?code=%20abc` was not matched at all. The parameter-NAME rule now
decides, on the key, before any text rule runs.

`forget_user` dropped the account's level elevation without recomputing the root level, so a process
whose only elevated account had just deleted itself kept building TRACE records to throw away, until
some later request happened to recompute it. Every other path that drops an elevation recomputes.

The sink's writer took the erasure lock and THEN waited for a pooled connection, while every request
path already holds a connection when it reaches that lock — so under pool pressure the writer held a
process-global lock against the very requests that could hand a connection back. Same order on both
sides now; the lock still spans the check, the insert and the commit.

The storage-budget notice said the rest of the hour was not kept, and then smaller batches kept
landing underneath it: the window slides and a refused batch is not charged to it. Says what it does.

Tests: the route's half of the erasure rule had no test — only the sink's did — and a source-level
contract test now covers the third insert nobody has written yet. `logs._erased` is process-global
with an hour's memory while the DB fixture hands serial ids back out from 1, so it is swept before
each test like the rate limiters are.
2026-09-17 23:45:16 +02:00
peter 791ba5f507 fix: a calendar date is the same day everywhere; legal pages use the shared language picker
formatDate read a date-only value as UTC midnight and showed it in the viewer's zone, so the legal pages' last update read a day early west of UTC. Date-only values are shown in UTC. The legal pages' hand-rolled language buttons are the landing page's LanguageSwitcher. formatDateTime and formatDate get the two-language test formatNumber already had.
2026-09-17 03:53:40 +02:00
peter 7789db8240 fix: admin config pickers fit a phone
A segmented bar cannot shrink or wrap, and the five log levels (~300 px) are wider than a phone leaves beside the navigation rail. A choice is a bar only when its labels are short; the rows stack below sm.
2026-09-17 03:53:40 +02:00
peter f92cd9017f fix: the diagnostics tab keeps its focus, its level and its consent
Deep-review round 1 of E22.

- After a 429 the stream sent one POST per new event while a full batch
  waited; it now stays quiet for 15 s. Past the queue cap the oldest
  events go, as the comment always said, a refused batch included.
- A batch refused after the level was lowered came back into the queue
  of a later consent in the same tab; each consent is its own session.
- A `me` fetch that left before a level or preference change could put the
  old value back for up to a minute. `updateMeCache` writes the cache and
  cancels that fetch; usePref goes through it too.
- The consent box's buttons and "revert now" took the focus with them to
  <body>; it returns to the level picker. Watched until the button is
  gone — in Chrome the saved level arrives in two renders, and a one-shot
  check spent itself on the first (jsdom never showed it).
- The countdown line is no longer a live region re-read every minute.
- "Saját napló" rows were keyed by index on a newest-first list, so every
  new event remounted every row and dropped the focus off a row button.
2026-09-17 03:53:40 +02:00
peter 4bfba4aa86 refactor: one definition of a followed channel and of a choice spelling
The explore purge rebuilt followed_channel's two EXISTS clauses inline, so what is polled and what may be deleted could drift; it now uses the predicate, which moved to sync/explore.py. The env validator and sysconfig each folded a choice value on their own; both call config.canonical_choice. The deep-backfill comment no longer calls recent backfill unconditional.
2026-09-17 03:53:29 +02:00
peter 86bc008a81 fix: a crashed job's crash line carries its job id
The bind sat inside the try, so it was already reset when the crash line and the cleanup were logged — the one line most worth finding by job id had none.
2026-09-17 03:53:29 +02:00
peter fac5c4465a fix: stored logs keep bad rows, erased accounts and unconsented queries out
Deep-review round 1 of E22.

- A NUL character (`GET /assets/%00` logs the decoded path) made Postgres
  refuse the row and with it the sink's whole batch, every user's rows
  included. `to_row` replaces NUL in every text column and in attrs.
- An erased account's lines kept arriving after purge_user's DELETE: the
  erasing request's own access line, a queued row, a background task.
  `logs.forget_user` runs first; later rows are stored without the account,
  and the check and the insert share a lock with the erasure.
- An admin setting the whole process to TRACE logged every user's query
  string. The query is now logged only under the user's own Teljes consent,
  and decoded before redaction, so `%40` no longer hides an address.
- The browser log ingest has an hourly storage budget per account (5 MiB,
  one notice when spent) and reads only a JSON Content-Type, so a
  same-site text/plain form post cannot write into a user's log.
  RateLimiter takes a per-attempt cost for the byte budget.
- The root level is applied under the elevation lock: two concurrent
  changes could leave it at the base while an elevation was active.
- Tests for the running sink (level gate, drain on stop) and for the
  admin log level applied at runtime and at boot.
2026-09-17 03:53:29 +02:00
peter 9c9b1cf6d5 test: the gate's output carries no noise to hide a real warning in
- Home tests mocked the api without downloadIndex, so every card's download button logged a
  missing-queryFn error (64 stderr blocks per run).
- QueryState rendered without an i18n instance; Modal.fullscreen awaited a synchronous act,
  which React reports as an act environment it does not support.
- pytest: Starlette 1.x's TestClient is built on httpx2 and warned on every run without it
  (pinned in requirements-dev); alembic.ini declares its path separator.
2026-09-17 03:03:37 +02:00
peter c3d88d9b20 refactor: component files export only components; lint fails on a warning
The 43 react-refresh/only-export-components warnings the gate let through meant an edit to a
provider (or AnchoredPopover, Toaster, plexDetailUi, …) could not be hot-swapped: Fast Refresh
reloads a module that also exports hooks or helpers.

- Each provider's contexts, the types its hooks return and the hooks themselves live in a sibling
  context module (navigationContext.ts, channelsContext.ts, …); the provider file keeps the
  component. loadAccountFilters moved next to the feed filters' defaults (feedFiltersInit.ts),
  following navigationInit/playlistsInit.
- Helpers that sat in component files have their own modules: welcomeMode, downloadTitle,
  plexPlayable, channelSync, ui/anchoredPopoverHooks, ui/pageToolbarSlot, ui/pagerContent,
  ui/toastLevels, ui/overlayOrder, ui/tableAlign, lib/plexDetail; the cells of the channel columns
  and the registry's page wrappers are components files of their own (channelCells, modulePages).
- usePersistedTab, an alias of useAccountPersistedState, is gone; its callers use the original.

npm run lint is now eslint --max-warnings 0, so a warning fails the gate like an error does.
2026-09-17 02:59:09 +02:00
peter 569190480d refactor: hook dependencies name everything the hooks read
The 12 react-hooks/exhaustive-deps warnings the gate let through:
- ShareDialog and YearFilter built a fresh [] per render while their query loaded, so their
  memos recomputed on every render; they share one empty list now.
- VideoEditor's drag effect called a seek/moveBoundary recreated each render; both are stable
  callbacks in its deps.
- WizardProvider re-checks onboarding on exactly the fields it reads (not the me object, whose
  60 s refetch would reopen a closed wizard).
- Channels, Feed and PlexPlayer list the stable values they already used.
- The player's keydown effect is bound once on purpose (its helpers read refs): said so, with a
  disable, as the codebase's other deliberate sites do.
- dialogFocus and the YouTube player's cleanup read refs React may already have let go of; they
  use what they captured when the effect ran.
2026-09-17 02:45:19 +02:00
peter 5883ea509d feat: Settings → Diagnostics, a consented log level and the tab's own log
A user can raise their diagnostics level to Részletes (DEBUG) or Teljes (TRACE) for 24 hours from a
new Settings tab, after a consent box that says what is recorded, that the server keeps it and for
how long (the instance's own retention), and that it lapses. Alap is the base and needs no consent.

- users.diagnostics_level/_until (migration 0087), written only by PUT /api/me/diagnostics, which
  computes the expiry itself; the demo account cannot raise it. /api/me reports the level in force.
- The server logs an elevated user's requests at that level and stores every line of them
  (logs.LevelGate on the stdout handler and the sink; the root is lowered only while an elevation
  is active). Teljes adds the redacted query string to the request line.
- POST /api/diagnostics/events stores the browser's log as source=web rows: account from the
  session, body capped while streaming, events/attrs capped, re-redacted, events under the
  consented level dropped, timestamps clamped, 30 batches a minute, 409 once the level lapsed.
- lib/diagnostics.ts applies the level to the tab and streams only what happens after consent, in
  batches, never its own request lines, with a keepalive flush on pagehide; stops on lapse (its
  own timer), on withdrawal and on 409.
- Detail hooks at DEBUG: which control was clicked (test id, icon, app copy in English — never user
  text, input values or a link's query), visibility/bfcache/resize, filter changes
  (lib/filterLog.ts; typed search only at TRACE), player events (lib/mediaLog.ts; media errors are
  warnings at every level). Offline/online at the base level.
- "Saját napló" viewer: live, newest first, level groups with counts, search, row details, copy
  the filtered events as JSON lines, clear.
- Privacy policy and Terms exist in Hungarian and English (a contract test holds both languages to
  the same sections); the policy now describes the server logs and diagnostic logging, self-service
  deletion and password sign-in.

Fixed on the way:
- A sync auth dependency ran logs.set_user on a copied context, so the account never reached a
  handler's own log lines (measured); a per-request mutable scope carries it now.
- The Settings tab rail left a phone no room (clipped labels, buttons over text): below `sm` it is
  a scrolling row above the content.
- PUT /api/me/preferences took a bare dict body; it is a Pydantic model now.
2026-09-17 02:42:12 +02:00
peter a210992bfc feat: client log with request ids, joined to the server's
The browser now keeps its half of the one event shape logs.py defines:
lib/log.ts is a levelled, redacted, bounded (2000 events) in-memory ring
with the server's level names and severity numbers. Nothing leaves the
browser yet; sending is the consented diagnostics level (S5).

- lib/http.ts `send` is the one fetch: every request carries an
  X-Request-Id and becomes one http.request event. The server echoes the
  matched route template in X-Route (only for a matched route, so a raw
  path with a value in it is never sent back); without one the client
  templates the path itself. req, beacon, the upload XHR, the public
  watch page and the legal pages all go through it.
- Hooks: app start snapshot, uncaught errors and rejections, History
  push/replace/popstate (a scroll-only replace is not an event; parameter
  values only at DEBUG, typed search only at TRACE), notifications, the
  error dialog, the crash boundary, failed queries and mutations (key
  head only), and the messages socket (lifecycle, frame types at DEBUG).
- Redaction is one contract: backend/tests/fixtures/log_redaction_cases.json
  runs against both redact() implementations.
- Lint: fetch, XMLHttpRequest and WebSocket each belong to one logging
  module, console to none; a contract test proves the rules fire.

Fixes a socket takeover: the close of a socket released by the last
subscriber cleared the newer socket a quick re-subscribe had opened and
scheduled a third, leaving two live sockets that delivered every message
twice (measured in Chromium: 3 created / 2 live before, 2 / 1 after).
2026-09-17 01:44:29 +02:00
peter 0517de1f59 fix: poll and backfill only channels somebody follows
The scheduled RSS poll and recent backfill covered every channel row,
but rows also come from search results, playlist sync and dropped
subscriptions, and nothing deletes them. On prod 582 of 979 polled
channels had no follower, three of them dead feeds answering 404 on
every poll. Both jobs now take subscribed or explored channels only,
and the scheduler card counts the same set. As a deliberate trade-off,
the shared Library no longer receives new uploads from channels nobody
follows.
2026-09-17 00:38:46 +02:00
peter b59fd38a9c feat: store warnings and request lines in log_events
The log now has a stored, queryable copy for the diagnostics console:
every WARNING and above plus the request line of every request, kept
for log_retention_days (default 7, no "forever") and pruned hourly by
the new log_gc job.

Rows are built on the logging thread from the same redacted fields as
the stdout line, queued without blocking and written in batches by one
background thread on its own connection. A full queue or a failed write
drops and reports instead of retrying. Flood control keeps 30 of any
one message per minute plus a count of the rest; anonymous request
lines are capped at 60 a minute per client, and the client address is
never stored. Deleting an account deletes its stored events.

Measured on prod first: a YouTube RSS outage wrote 11k warnings in four
hours, one per channel per poll. The poll now writes one warning with
per-reason counts and a few examples, and a failure on our side gets
one traceback per poll instead of one per channel.
2026-09-17 00:20:14 +02:00
peter 88a166e4aa fix: call the audit log "Auditnapló" in Hungarian
"Naplózás" (logging) named the audit log in the nav, its page and its
Configuration tab, right beside the new "Naplózási szint" (log level)
under Diagnosztika — two different things under one word. The scheduler
hint also pointed at a "Beállítások → Naplózás" path that does not
exist; it is Konfiguráció → Auditnapló, as the English already says.
2026-09-17 00:04:01 +02:00
peter 07766c43e8 feat: closed-set admin knobs are pickers, not free text
log_level, search_source, download_layout and the transcode hw/preset
were typed by hand, and every consumer carried its own guess at what a
typo meant. The value set is now a Literal in config.py: pydantic
rejects a bad env value at boot (any letter case accepted), sysconfig
refuses a bad override and derives the admin picker's options from the
same Literal, and the consumers read the value as-is.

The Configuration page renders a choice key as a segmented bar, or a
dropdown past six options, with translated option names. Its tabs now
follow the registry's order instead of a hand-kept copy that had
already dropped the explore group.

A single-choice FilterSelect now opens on the option in use: in the
browser it could land on the first row instead.
2026-09-17 00:03:56 +02:00
peter bb3c6dec45 release: 0.87.1 (bug/youtube-cookie-host)
gate / frontend (push) Failing after 3m13s
gate / backend (push) Successful in 4m22s
2026-09-16 04:41:40 +02:00
peter 3cf8f04d73 chore: 0.87.1 release notes 2026-09-16 04:41:10 +02:00
peter 2b56c5c2eb fix: name the error yt-dlp actually raises for a cookie-emptied client set
A review round read yt-dlp's source and found the comment behind the previous commit's safety-net
marker did not match it. Measured in the worker with a real account cookie and
`player_client=android_vr`: yt-dlp logs `Skipping client "android_vr" since it does not support
cookies`, drops it, and the request left with no client fails with "Requested format is not
available" — not "No player clients have been requested". That raise sits EARLIER in yt-dlp, before
the cookie-capability removal, so the cookie path can never reach it; and an unknown or renamed
client is skipped with a warning and replaced by yt-dlp's defaults, so the other scenario the
comment named cannot reach it either.

So the marker bought nothing: the symptom that does occur was already retriable here before this
branch. It is dropped, the comment now states what was measured, and the test asserts the real
symptom instead of a message the code cannot produce. The cookie-capability filter itself is
vindicated by the same measurement — without it every signed-in download would burn its first
attempt on a client yt-dlp discards.

Also from the round: an enqueue test for a `source_kind="youtube"` job, the one join nothing else
made (a YouTube job stores a bare 11-char ref and the URL the cookie gate keys on is rebuilt by the
adapter — both ends were pinned apart, the join was not, and a break there would stop every YouTube
cookie silently while signed-out downloads kept working). And copy corrections: the group's only
unhedged claim is hedged, "described the video" loses the pipeline vocabulary, and four Hungarian
phrasings that read as translated English are rewritten.
2026-09-16 04:29:13 +02:00
peter c06a456a9a fix: a YouTube stream refused after extraction is the gate, not a network fault
Measured on the isolated test stack the same night, on the very same video ~90 minutes after it was
answering with the bot message: yt-dlp got the player response and picked its formats, and the CDN
then refused the stream itself — "unable to download video data: HTTP Error 403: Forbidden".
Signed out it did that on every attempt; with the owner's cookie attached the video downloaded in
full. So the wall moves between the player response and the stream, and the text it wears there was
costing the download twice.

retriable: the format URLs belong to the client that minted them, so the next client set is a real
second chance — but this text classifies as `network` and nothing in the YouTube marker list matched
it, so the ladder ended the job on attempt 1 and never tried the fallback.

catalog: `youtube.media_forbidden` for the cookie-less case, so the UI stops calling it a network
error when nothing was wrong with the network, and names what actually gets past it. Scoped exactly
as narrowly as the measurement: another host keeps the generic `network` category, and so does a
download that already carried a cookie — telling a signed-in user to sign in is the failure the
catalog exists to avoid. EN + HU copy.
2026-09-16 03:43:29 +02:00
peter 864f673ef5 fix: a YouTube video behind the bot wall is downloadable with cookies
A YouTube link can fail with "Sign in to confirm you're not a bot" while every
other YouTube download works — YouTube gates INDIVIDUAL videos behind a
signed-in session. Measured on prod job 273: all twelve of yt-dlp's player
clients fail on that one video while its neighbours download fine, the bgutil
PO-token sidecar is live and answering, and the newest yt-dlp fails identically.
So the download is not broken by our stack or a stale yt-dlp, and no retry,
client switch or upgrade gets past it. A cookie is the only way in.

The failure copy already sends the user to Settings -> Account -> Download
authentication, but YouTube was not in the cookie host registry, so that
dropdown could not help them: the app asked for something it did not offer.

credentials: YouTube joins SUPPORTED_HOSTS (youtube.com, youtu.be,
youtube-nocookie.com). The Settings dropdown, the SEC-12 host scope and the
private-asset cache salt all derive from the row, so that is the whole wiring.

adapters: a signed-in YouTube download runs only cookie-capable player clients.
Measured against yt-dlp's own INNERTUBE_CLIENTS table: with account cookies
attached yt-dlp REMOVES every client whose SUPPORTS_COOKIES is False, and a set
it empties that way raises "No player clients have been requested" — which is
not a per-client flake it retries past. Our primary set is android_vr, one of
those, so without this the cookie would have failed the download on attempt 1
and never reached the POT-backed web fallback it exists for. A test pins the
client list against the installed yt-dlp so a version bump turns the gate red
instead of silently un-authing every YouTube download, and the empty-set message
joins the retriable markers as the second line of defence.

errors: catalog entries for the wall with and without a cookie, so a user who
has already added cookies is told they were rejected (YouTube rotates exported
ones) rather than to add some. The marker is quote-agnostic: YouTube writes the
apostrophe as U+2019, so "you're not a bot" with a straight quote never matches
the real text. EN + HU copy.
2026-09-16 02:03:00 +02:00
peter 6c63df6704 feat: one structured log for every process, with a request id
A deployment printed four different log formats and none of them could be
joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in
main.py for anything not started by uvicorn, logging.basicConfig() in the worker
and in the download child (no timestamp at all), and alembic's fileConfig.
app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log
data model, no SDK): ts, level, severity, source, logger, event code, message,
attributes, plus the correlation the context carries — request id, account,
download job.

Redaction is applied at the HANDLER, not at the call sites: every message, every
attribute and every traceback is scrubbed on the way out, so a secret cannot
leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies,
secret config values and email addresses never reach a line.

The access log is ours now (`http.request`): the route template instead of the
concrete path, the status, the duration and the account — and never the query
string, which uvicorn's access log printed on every single request (`?account=`,
`?source=`, the `?token=` of a share link). A well-formed client request id is
accepted and echoed in X-Request-Id, so a browser line and a server line join up.
The account comes from `request.state`, filled by the auth dependency: the
middleware is the outermost one, so it runs outside SessionMiddleware and in a
different task context.

`log_level` becomes an admin knob (sysconfig), applied without a restart and
re-applied at startup — best-effort on purpose, since a logging knob must never
be what stops the app from booting. Every compose service gets bounded json-file
logs (3×50 MB): Docker keeps every line forever by default, and the API now
writes one line per request.

Verified on the localdev stack: JSON everywhere including alembic and the
worker, the query strings gone, a supplied X-Request-Id echoed and logged, and
three /healthz hits of which only the one made while the level was DEBUG
produced a line.
2026-09-16 01:37:02 +02:00
peter a5bcd9fca2 fix: numbers follow the UI language, not the browser's
`n.toLocaleString()` groups in the BROWSER's locale, so an English UI in a
Hungarian browser printed "1 234 567" while a Hungarian UI in an English
browser printed "1,234,567" — the number half of the date defect fixed in
618ef812. One `formatNumber` in lib/format.ts (a per-language cached
Intl.NumberFormat, like the relative-time formatters beside it) now serves
every grouped figure: Stats, Scheduler, the channel columns and the stats
popover, VideoCard's view tooltip, the ItemCount pill, and formatTotalHours.

ReleaseNotes' `toLocaleDateString(undefined, …)` was the one DATE site the
previous sweep missed; it moves onto formatDate.

The new cases assert both languages and that the two differ, instead of
comparing against the runtime's own default: that default is Hungarian on the
dev box and English on a CI image, so a one-language assertion would pass
under the old bug on half the machines. Verified in Chromium on :5173 with the
browser forced to hu-HU and the UI in English — Stats, Scheduler and the
channel table all keep comma grouping.
2026-09-16 01:05:53 +02:00
peter 618ef812e6 fix: dates follow the UI language, not the browser's
Seven sites formatted with a bare toLocaleString()/toLocaleDateString(), so an
English UI on a Hungarian machine printed Hungarian-shaped dates (and the reverse).
They now go through formatDateTime/formatDate, which take the i18n language.
2026-09-16 00:49:23 +02:00
peter e299c66121 feat: show each account's last sign-in in Users
A right-aligned column at the end of every Users & roles row says how long ago the
account signed in, with the exact moment as the hint and as a line in the account
panel; a dash where nothing is recorded, and the demo account says it is not recorded.
formatDateTime formats that moment in the UI language rather than the browser's.
2026-09-16 00:45:03 +02:00
peter 11aef4ab7a feat: record when an account last signed in
users.last_login_at (migration 0085, no backfill) is stamped by _establish_session,
which the Google callback now shares with password login instead of repeating its
session setup inline. The demo entry does not stamp its shared account. The admin
user list carries the field.
2026-09-16 00:34:50 +02:00
peter 4e52bdd4ca fix: a signed-out tab no longer asks for saved views
The providers mount for the public landing too, and the saved-views query had no
enabled gate, so every signed-out load logged a 401 for /api/saved-views. The query
now waits for a signed-in account in BootGate's me cache, like WizardProvider.
2026-09-15 23:45:13 +02:00
peter 595490cb98 release: 0.87.0 (improvement/watch-background-audio)
gate / frontend (push) Failing after 3m19s
gate / backend (push) Successful in 4m25s
2026-09-15 03:02:19 +02:00
peter 7537882235 chore(release): 0.87.0 2026-09-15 03:01:59 +02:00
peter 47d968eeea fix: review round 4 — an account switch unloads as the old account, a share link's lockout costs five networks
Deep-review round 4, targeted at round 3's changes (f569b19c..e7f38396).

Accounts
- Switching, adding or signing out an account moved the tab's pin before
  the page unloaded, so the players' resume beacons fired on the way out
  under the NEW account: one account's playback position saved into the
  other. The pin is now handed to the next page (pinOnNextPage) and taken
  over at import; clearActiveAccount is gone.
- Switching to a suspended account was refused but left it in the
  switcher, refused on every click; a deactivated one was not refused at
  all, and made the default it signed every account in the browser out on
  the next request. Every account _session_user would refuse — gone,
  deactivated, suspended, revoked — is refused and dropped from the list.

Share page (/watch)
- An IPv6 client is limited by its /64 (client_ip becomes client_key, for
  every limiter): keyed on the full address, one subscriber could rotate
  through endless identities. The comments now state what the per-link
  ceiling costs: guessing from five networks can keep a link shut for its
  window, and guesses already running when it fills are still checked.
- The link is looked up before the limiter records the attempt, so made-up
  tokens no longer grow it by an entry each.

Forms
- Failing the same way twice is announced again: the sign-in, passphrase
  and setup errors are keyed on the attempt (a clear and a set in one
  handler left the alert node untouched).
- The install wizard's own validation showed the generic error — it threw
  plain objects its catch did not read. It sets the step's message.

Tests: a re-pin leaving the previous account's notices behind; the pin
hand-over; switching to an ended account; the /64 viewer; made-up tokens;
the repeated alerts. Each mutation-checked RED for its own reason.
2026-09-15 02:53:22 +02:00
peter e7f38396a2 fix: review round 3 — a new tab keeps its account's notices, a revoked tab account lands on the default
Deep-review round 3 over main..improvement/watch-background-audio.

Account boot
- The notification history and settings, and the hints toggle, are read
  when their modules load, which in a new tab (bookmark, typed URL) is
  before BootGate pins the account: they read as nobody's, and the tab's
  first notice replaced the account's stored history (up to 100 entries)
  with a one-item list. They now read again when the tab is pinned
  (onAccountPinned, run by setActiveAccount); a notice raised before the
  pin stays, after the history.
- A tab pinned to an account still in this browser's wallet but whose
  session ended (suspended, deleted, or revoked by "log out everywhere")
  got "signed out" from /api/me while the default account was still
  signed in, and signing in again as anyone else left it there. A refused
  per-tab account now leaves the wallet, and the probe answers as the
  default, so the tab reloads onto it.

Share page (/watch)
- The password unlock is limited per viewer (link + client IP), under a
  per-link ceiling that counts only wrong passwords. Keyed on the link
  alone, ten wrong guesses every five minutes locked out every viewer,
  the ones with the password included. client_ip moves to app.ratelimit
  so the public routes can key on it.
- Its failure messages and the load-failure screen are announced
  (role=alert, the password input described by its error). The sign-in,
  passphrase, password-change and setup form errors get role=alert too.

Comments that still described removed machinery are corrected: App's
collapse flags, SettingsPanel's Save bar, ConfigPanel, authed_format_sig.

Tests: the account's history and hints across the pin; a revoked and a
suspended pinned account; the per-viewer limit and the ceiling; the
limiter's exhausted(); the share page's announced failures; a share
link's feed filters persisted. Each mutation-checked RED for its own
reason.
2026-09-15 02:25:00 +02:00
peter f569b19c49 fix: review round 2 — a signed-out tab acts as nobody, honest share-page failures
Deep-review round 2 over main..improvement/watch-background-audio.

Account resolution (server)
- A request naming an account that is no longer signed in to this browser
  (X-Siftlode-Account or ?account=) now acts as nobody instead of the
  session's default account. A tab still pinned to an account signed out in
  another tab went on acting as the default: its player checkpoints and
  preference saves landed in that account, the messages socket and Plex
  playback ran as it, and its Sign out signed THAT account out. A hidden tab
  never polls /api/me, so a tab playing in the background kept doing it.
  The /api/me bootstrap probe alone keeps the fallback, which is how such a
  tab learns who it now is: the app's 401 handler reloads, and BootGate
  re-pins the tab.

BootGate
- On an identity change the tab reloads without moving its pin first. The
  tree being torn down still runs its unmount saves before the page goes,
  and with the pin already moved they wrote into the new account's local
  keys (a player volume still pending). The reloaded page corrects the pin.

Share page (/watch)
- Only a 404 says the link is gone. A 5xx, a dropped connection, and the
  unlock's own rate limit all rendered "no longer available ... expired or
  revoked" to a viewer with a valid link, with no way to try again. A load
  failure now offers Retry; too many password tries says so and keeps the
  form; an unlock that gets no answer says so instead of doing nothing.
  Strings in en/hu/de.

Downloads
- A cookie-authed asset is recognised under the current signature version
  and the one before it, so a job queued before a deploy that bumps the
  version still downloads with its cookie instead of failing on the login
  wall with an error that says no cookie was used.

The app chunk is preloaded through lazyWithPreload, so a failed fetch no
longer leaves an unhandled rejection beside the ErrorBoundary. The panel
layout seed is one helper (seedLayout).

Tests: the stale-tab refusal, its harmless Sign out and the probe fallback;
the pre-bump authed asset; BootGate keeping the pin on reload and staying
mounted through a same-account change; the reveal clock's later-commit
hand-off; the prefs, panel layout and demo-scope seeds from the me cache;
the share page's failure states. Each mutation-checked RED for its own
reason.
2026-09-15 01:35:25 +02:00
peter e795d191c5 fix(boot): review round 1 — reconcile the tab pin, one loading clock, honest probe failures
Deep-review round 1 over main..improvement/watch-background-audio.

BootGate
- The tab's account pin is corrected to the account the server answered as.
  The server honours the pin only for an account still signed in to this
  browser and otherwise answers as the default, so a tab pinned to an account
  signed out in another tab ran as the default on the server while reading
  and writing the gone account's local state.
- A mounted tab whose identity changes (its session ended, or the case above)
  reloads instead of rebuilding the tree in place, like every account switch;
  that also drops the previous account's query cache. The account key on the
  tree goes: the readyFor gate already unmounted it on every change.
- An unanswered setup probe shows the error with Retry, like an unanswered
  account probe, instead of the signed-out landing to a signed-in user.
- Comments: /api/me never 401s; the ended-session path is identity → nobody.

Loading states
- LoadingState's reveal delay is one clock across loading states that hand
  over to each other (boot screen → page-chunk fallback → the page's own
  fetching state), instead of a CSS delay restarted by each, which blanked a
  visible spinner for another 300 ms at every hand-off. BootScreen is now the
  shared loading visual, full screen, counting from navigation start.

Plex shared link
- Persists only the dimensions the link carries again. Writing the default
  filters for an omitted `pf` created the stored-filters key, which is what
  tells SavedViewsProvider the account picked filters, so the account's
  default Plex saved view was never applied again.

Watch page
- The Repeat toggle uses the shared pressed-pill paint (a solid fill when on)
  instead of Button variants that differ only in colour, and its flag goes
  through storage.ts readJSON/writeJSON.
- The actions wrap under the title when the row is too narrow, instead of
  squeezing the title into a column a word wide on a phone; the title breaks
  long words.

Comments that still justified code by the removed adopt effects (usePref,
PrefsProvider, FeedViewProvider) are rewritten against the seed-once model.

Tests: BootGate error/retry for both probes, setup mode, a failed background
refetch keeping the app, the pin correction and the reload on an identity
change; the shared reveal clock; the Plex link persist; the audio signature
actually moved off version 4; a video download (av and v) still remuxed to
an mp4. Each mutation-checked RED for its own reason.
2026-09-15 00:53:15 +02:00
peter 4ad7183102 feat(players): keep a video playing in the background on Brave iOS; repeat on the watch page
On iPad Brave, a shared video stopped the moment the browser left the
foreground, even with Brave's "Enable Background Audio" on, while an mp3
kept playing. Brave iOS keeps a web <video> going by re-playing one the OS
paused (brave-core ios/browser/web/media/resources/media_backgrounding.ts),
but it only hooks a <video> present at document start or added as a
MutationObserver addedNode that IS a video. React inserts a player's
<video> inside its freshly mounted wrapper in one go, so Brave never saw it.

lib/browserBackgroundPlay re-inserts a mounted <video> at its own position,
once per element: it becomes an added node, the DOM React manages does not
change, and playback is not interrupted. The watch page, the downloads
library player and the Plex player hand their <video> to it. Measured on the
owner's iPad (Brave) with a prototype of the same idea: the video kept
playing in another app and with the screen locked. The YouTube modal is not
ours to change: its <video> lives in YouTube's cross-origin iframe, and
nothing of ours pauses it on backgrounding.

The prototype's two audio-element variants (a muted video following an
<audio>, and an <audio> taking over on hide) went silent in the background on
the iPad and are removed.

The watch page gets a Repeat toggle, on by default and remembered per
browser (LS.watchRepeat; the page has no account): it sets `loop` on the
<video> or the audio-only <audio>.
2026-09-14 23:58:30 +02:00
peter 2c3aaec62d feat(watch): prototype background sound for a shared video on iPad (dev only)
On iPad Brave a shared mp3 keeps playing when the browser leaves the
foreground, but a shared h264 video stops at once, even with Brave's own
"Enable Background Audio" on. Three variants behind a `?bg=` param on the
/watch page, for a real-iPad comparison; without the param the page is
unchanged.

  a: the sound comes from a hidden <audio> on the same file, the <video> runs
     muted and follows it (drift correction, user seeks and mute moved onto
     the audio; our own catch-up seeks are not copied back, which re-seeked
     the sound on every correction and sent both back seconds in WebKit).
  b: the <video> plays with sound; an <audio> unlocked by the start tap takes
     over while the page is hidden and hands back on return.
  c: no audio trick: the <video> is inserted as a node of its own into an
     already-mounted wrapper. Brave iOS's background play only hooks a video
     present at document start or added as a MutationObserver addedNode that
     IS a video (brave-core ios/browser/web/media/resources/
     media_backgrounding.ts); the page inserted the video inside a new wrapper
     in one go, so Brave never saw it. A re-implementation of that detection
     rule reports the default page's video unhooked and mode c's hooked.

Modes a and b were driven in Chromium and WebKit with a simulated backgrounding
(document.hidden forced plus the OS's pause): sound continues, the picture
catches up on return, a user pause stops both. iPadOS's own media policy can
only be judged on the iPad.
2026-09-14 23:40:51 +02:00
peter 70598d47a3 fix(dev): proxy the messages WebSocket through Vite, which stops a reset killing the dev server
The Vite dev proxy had no `ws: true` on /api, so the in-app messages socket
(/api/messages/ws) was an upgrade nobody handled: it never connected on :5173,
and the first reset on it (a closed tab, a dropped connection) threw an
unhandled ECONNRESET that took the whole dev server down. Reproduced with a raw
upgrade + RST (dev server dead); with `ws: true` the backend answers the
upgrade, the dev server survives the reset and a closed tab, and the signed-in
socket is accepted through the proxy.
2026-09-14 23:08:41 +02:00
peter 5858e7b513 test(e2e): run the net on WebKit and an iPad profile; opt the account out of Home
The Playwright net now runs every flow in three projects: chromium (Desktop
Chrome), webkit (Desktop Safari) and ipad (iPad gen 7 landscape: touch, mobile
viewport). WebKit catches the engine-level differences every iOS browser
inherits; the config and the README say plainly what it does NOT catch: iPad
media refusals. Playwright's WebKit decodes through the host OS, and on Windows
it played an mp3-in-mp4 and an AV1 mp4 that a real iPad refuses, so media
formats are checked on a real iPad against the dev stack over the LAN.

global.setup opts the e2e account out of starting on Home (the Startup setting),
which the feed, channel and player specs need since E24. It only takes effect
with the new-tab boot fix: before it, all twelve of those specs landed on Home
in every project. 49/49 green on :8080 across the three projects.
2026-09-14 22:29:33 +02:00
peter 5346efcd10 fix(boot): a new tab knows its account before anything renders
With Settings -> Appearance -> Startup turned off, every NEW tab or window still
opened on Home; only a same-tab reload honoured it. The provider tree mounted
before /api/me answered, so NavigationProvider's initializer could not read the
per-account opt-out, landed on Home and mirrored ?page=home into the address bar.
When App pinned the account a moment later, the re-hydrate re-read that mirror
and stayed on Home. The same mount-then-patch shape made a new tab start on
default filters, the default theme and the browser language, and correct each
after the first paint.

BootGate (main.tsx, above the providers) now owns the setup-status and /api/me
queries and mounts the tree only once the identity is settled: it pins the tab's
account and adopts the account's language in a layout effect, so the first frame
of the app is already right. The tree is keyed on the account id, which makes
"per-account state is read once, for this account" a structural guarantee.

With that, the whole re-hydrate machinery is gone: useAccountRehydrate, the
reactive active-account store, the markDirty guards, the pin-deferred saved-view
seed and the present-dimension bookkeeping of the shared Channels/Plex links.
PrefsProvider, FeedFiltersProvider, usePref and App's panel layouts seed from the
me cache instead of adopting it in an effect; the theme is applied in layout
effects. The app chunk is fetched in parallel with the probe, and the boot screen
only fades its label in after 400 ms, on one clock across the gate and the
Suspense fallback.

Module switches no longer flash a placeholder either. Measured frame by frame
in Chromium and WebKit, every first visit to a module showed "Loading..." for
17-70 ms: the lazy chunk's Suspense fallback, then often the first fetch's
LoadingState. Pages are now lazyWithPreload components, and App fetches the
chunks of every module the account can reach (plus the channel page) once the
first page is idle; a preloaded page renders synchronously and never suspends.
LoadingState, the one shared loading visual, fades in only after 300 ms, so a
fast load shows nothing in between and a slow one still gets its spinner.
2026-09-14 22:29:33 +02:00
peter 6169e32ca5 fix(downloads): a signed-in Facebook reel keeps its title, and an mp3 stays an mp3
Since a Facebook cookie was added on prod, every reel came down titled
"Video": measured on dev, a signed-in read gives yt-dlp the placeholder
"Video" (an 18+ reel read as Chrome gives "Facebook") while the post's text
is still in the description. A new SourceAdapter.title hook takes the whole
first line of the description in that case, and both title writes (the early
fill and the finished download) go through it, so the row, the file name and
the share preview carry the post's text again. The line is not cut with an
ellipsis: normalize_title's NFKC turns "…" into "..." and collapses it to
".", which read as a finished sentence (seen in dev UAT).

The "Audio (MP3)" profile's extracted .mp3 was remuxed into an .mp4 by the
browser-safe step (the audio spec is clamped to container mp4 too) and served
as video/mp4, which iPad's WebKit will not play (prod job 268). Audio mode now
skips that step, so an .mp3 is served as audio/mpeg and an .m4a as audio/mp4.
Audio downloads version their cache identity on their own (v5), so a cached
remuxed audio re-derives without re-downloading every cached video.
2026-09-14 17:46:46 +02:00
peter 90523284ab release: 0.86.0 (feature/e11.5-download-failures)
gate / frontend (push) Failing after 3m17s
gate / backend (push) Successful in 4m17s
2026-09-14 01:53:06 +02:00
peter 17bfea8754 release: 0.86.0 notes 2026-09-14 01:52:34 +02:00
peter 2a9ddb78ac fix(downloads): E11.5 epic-end review round 2
Pause now re-reads the job under the same asset-then-job row lock as resume,
cancel, delete and redownload. Its unlocked write waited for a concurrent
cancel's lock and then set the released job back to paused, a holding status,
so a later delete released the same hold again and could free a file another
user's job still held (measured: the overlapping-request test ends 'paused'
instead of 'canceled' with the old read). The lock test now calls the route
handlers, so it also pins which helper each route uses.

Comments follow the code: resolve_url's contract names both verdicts and that a
failed page read is raised as itself; the inherited-failure diagnosis asks the
cookie gate as it stands when the job is claimed, so its advice fits the cookie
that job's Retry would use; the undecryptable-cookie warning no longer claims a
download goes ahead, since the worker parent reaches it only to diagnose.
2026-09-14 01:42:57 +02:00
peter e887233210 fix(downloads): E11.5 epic-end review round 1
A refused Facebook cookie on a group post is no longer "no video in this post":
measured on dev, Facebook answers with its login page (no post id at all), so the
adapter now raises its own POST_NOT_SHOWN verdict and the catalog explains it as
the refused cookie. The adapter owns both post verdicts and the "cannot parse
data" wall wording, so the Chrome retry and the catalog read one constant. The
group-post "no formats" entry is cookie-scoped (with a cookie yt-dlp only gets the
post's video link) and uses the adapter's own group-post test instead of a
"/groups/" substring.

Job transitions that move an asset's ref count (resume, cancel, delete,
redownload) re-read the job under a row lock, asset first, so two overlapping
requests can no longer release the same hold. get_or_create_asset loses an insert
race inside a SAVEPOINT instead of rolling back the caller's pending writes (Retry
lost its cleared error). Every worker-side job failure goes through
finalize_if_running and writes error_detail, so a cancel that landed first is kept
and a failure never shows an earlier one's copy. The worker asks the download
child's own cookie gate (credentials.usable_cookie) when a job inherits a failed
asset.

UI: the Facebook copy no longer points at a Retry button the failure window does
not have, says "paste" like Settings, and the HU wording is fixed
("Újraletöltés" on the re-download confirm). A category this client has no copy
for falls back to the generic sentence instead of a raw key.

Tests: route serialization of error_detail, the probed duration through a real
_download, the page-read cap and a failed read, a Retry off a pending asset, the
confirm-button labels, the lock and savepoint races.
2026-09-14 00:50:05 +02:00
peter b17d92d7c2 fix(downloads): confirm dialogs name the action on their confirm button
The download center's confirm button repeated the dialog's question
("Re-download this file?"). It now reads the action: Stop download,
Re-download, Remove, Remove from my list.
2026-09-13 23:38:11 +02:00
peter 90ee34a1b4 feat(downloads): a Facebook group post downloads its own video, or says it has none
Given a group post's link, yt-dlp either found no formats or took another
video from the page's feed data, so a "successful" retry of such a post was a
wrong download. With the user's cookie the Facebook adapter now reads the post
page, takes the single video node in the script blocks that carry the post id,
and downloads that video's watch link (the Chrome retry does the rest). No
single video in the post fails with its own message and the group-post copy,
reworded to say the post has no video; the post link is never handed to yt-dlp
as a fallback. Without a cookie nothing changes.
2026-09-13 23:38:11 +02:00
peter 118fc70e08 fix(downloads): take the duration from the file when yt-dlp gives none
A Facebook or Instagram reel extracts with no duration, so the asset kept a
NULL duration_s although the file plays for minutes, and the Library row had
no trim action. The finalize probe now returns the duration with the codecs
(the file wins, info fills a gap), and runs before the sidecars so the .nfo
runtime gets it too.
2026-09-13 23:12:16 +02:00
peter 636372511b chore: remove the unreferenced backend/Dockerfile
Nothing builds from it: both compose files build the root Dockerfile, and the
test image has its own Dockerfile.test. It installed the unpinned
requirements.txt, contradicting the lock header that every image installs
requirements.lock.
2026-09-13 23:12:16 +02:00
peter 1354f5b503 feat(downloads): a Facebook cookie download stuck at "Cannot parse data" is retried as Chrome
Measured on dev: with the user's cookie some reels still answer "Cannot parse
data" and only extract when the request impersonates Chrome. Impersonation is
a second attempt behind that one error, only when a cookie is attached — links
that already download with the cookie keep their plain path, and without a
cookie the same error is the login wall.

curl_cffi is pinned inside yt-dlp's supported range (0.10-0.15; outside it
every target silently reports "unavailable"); the lock only gains the new
packages. A test fails the gate if the locked pair can no longer impersonate.
2026-09-13 22:48:21 +02:00
peter 98ec1ffcab fix(downloads): Retry and Re-download take a cookie added after the failure
The login-wall copy tells the user to add their cookies and press Retry, but
Retry kept the job on the shared asset the worker never attaches a cookie to,
so it failed on the same wall. Resume now moves a pending/errored job onto the
asset enqueue would pick with the cookie (never off a file or a download in
flight).

Re-download salted its fresh asset with the job's previous asset id, which the
cookie gate cannot recompute, so even a cookie download rebuilt without the
cookie. The rebuild salt is now the new asset's own id (plus the user's cookie
salt when one applies), and is_authed_asset recognises it.
2026-09-13 22:31:49 +02:00
peter 18c11ca437 feat(downloads): a per-site failure catalog — Facebook's walls say why and what to do
A failed Facebook download read "Download failed": yt-dlp words the login wall
(18+ reel, closed group) as "Cannot parse data", which no category matched.
Measured on dev with the owner's cookie before writing a rule — the same text is
the login wall without a cookie and a refused cookie with one.

- errors.diagnose(msg, url, cookie_attached) tries a host-scoped catalog first
  (all-of markers, cookie state, link fragment) and falls back to the category.
  First entries: facebook.login_wall, facebook.cookie_rejected,
  facebook.group_post_video. The module doc is the recipe for the next entry.
- download_jobs.error_detail (0084) + backfill of the stored Facebook parse
  failures that were unknown.
- The pipeline's failure branch passes the link and whether the cookie went
  with it; resume and redownload clear code and detail (redownload never
  cleared error_code).
- A job meeting an already-failed asset got the text but no code, so it read
  "Download failed" where the first job said why. It is diagnosed again, with
  the cookie state from credentials.is_authed_asset (now shared with the
  cookie gate); the redacted network text classifies back to network.
- FE: failCopyKeys gives the row and the failure window one lookup — catalog
  copy, else the category. Copy in EN and HU; a contract test pins every
  backend entry to both locales.
2026-09-13 21:53:25 +02:00
peter ca5e446fbd release: 0.85.0 (feature/e11-power-navigation)
gate / frontend (push) Failing after 3m13s
gate / backend (push) Successful in 4m36s
2026-09-13 21:02:22 +02:00
peter 4c7ab12a3e release: 0.85.0 — E11 Power Navigation
Back, dialogs and menus keep your place, and the keyboard walks every list: five sprints plus three epic-end review rounds (19 → 14, 18 → 12, 14 → 14). Minor: j/k list navigation, s and g-chords, and Back-to-item are features.
2026-09-13 21:02:22 +02:00
peter c6b552e75f fix(keys): epic-end review round 3 — the part-way clip count shows, busy buttons keep focus, cards say their names
Targeted round over 3cd0c525's fix surfaces: 14 raised, 14 fixed.

- VideoEditor: the "only X of Y clips" toast went silent exactly on the job-cap/quota refusal it was
  written for (req had already reported the error); it now always shows. The cut list holds still while
  Create queues, and a retry continues the "(i/N)" series.
- ChannelPage: Subscribe/Unsubscribe use Button `loading`, "Load more" and ChannelDiscovery's Subscribe
  block the click with aria-disabled instead of native disabled; busy Block/Hide no longer light up on
  hover.
- listNav: an item can name itself (`navItemAttrs(i, name)`) — Plex posters, seasons, episodes, message
  rows and download rows pass their title; focus on a toast's Undo keeps the removed card's slot.
- AnchoredPopover: the hand-back only for focus-managed surfaces (a hover card no longer pulls focus
  back); the dead panel-ref check is gone.
- AccountPanel: no confirmation mail offered to the demo account; send_failed copy no longer blames
  the message for an unreachable server. Pager comment points at scrollParent.
2026-09-13 20:47:13 +02:00
peter 3cd0c525e2 fix(keys): epic-end review round 2 — focus comes back to the card, the right rail and the reason
Whole-epic deep-review of E11 (18 raised, 12 kept; B and G refuted, four latent test-hardening notes dropped),
plus the admin refusal copy gap round 1 surfaced.

- A context menu opened from a card the keyboard is on gives the focus back to the card. floating-ui returns
  focus through the opener's first tabbable descendant even when handed a ref, so for a non-tabbable opener the
  surface restores it itself (measured in Chrome: j, Shift+F10, Escape left it on the thumbnail link).
- Back finds a named list (a rail) by its name: a rail that appeared above it after the cache expired sent the
  focus into the wrong rail. An unnamed list is matched only when the page has the same number of lists.
- The slot kept for a removed item is forgotten once focus moves on to page content (search, a chip), so j after
  new results starts at the first visible item; a menu or dialog opened from the item keeps it.
- The list keys reach the cheat sheet only while a list has an item.
- A j-focused card root is a named group while it holds the focus.
- The channel page's Block and Hide toggles keep the focus while their request runs.
- A split export that fails part-way says how many clips were added and deselects them, so a retry adds only
  the rest.
- Config save, the one-click share link and "log out other sessions" report failures with their reason.
- VirtualGrid uses the one scroll-parent walk listNav measures against.
- Admin refusals (activate, resend, invite, set-password link) say why in HU and EN instead of "try again";
  the email-off special cases collapse into that one reason. The contract reads the codes from localizeDetail.
  HU copy points to Konfiguráció → E-mail / SMTP; HU "unreachable" matches the EN change of round 1.
2026-09-13 19:18:04 +02:00
peter f130a1c05a fix(keys): epic-end review round 1 — Back, menus and dialogs keep the focus where it belongs
Deep-review of E11 (19 raised, 14 confirmed; 2 refuted, 3 latent test-hardening notes dropped).

- Back from a sub-view (a Plex show page, a Messages thread) restores the item it was opened from
  even when focus was still inside the view: the first attempt now waits one task, past the
  sub-view's own popstate listener (measured in Chrome: it left focus on <body>).
- The restore listener is installed at load, so the first Back after a reload finds it; a push
  cancels a restore still waiting for the last landing's lists.
- Navigating over an overlay asks it again, a render apart, while its entry stays: the Plex
  player peels one panel per Back step, so Shift+W over info + stats closed one panel and dropped
  the navigation.
- j/k: a native <button> item keeps its own Tab stop; landing twice on the same item no longer
  leaks its tabindex; after the focused item is removed (a hidden card) the next j goes on from
  its slot when that slot is on screen.
- An abandoned `g` no longer swallows a modifier chord (g, then Mod+K opened the browser search).
- A hover card above a dialog no longer switches its Tab trap off (player, Modal, wizard,
  Lightbox): new layerStack `anyBlockingLayerAbove`.
- "Manage views…" closes its menu before opening the dialog, through a shared `closeThenAct`
  that ActionMenu now uses too.
- Copy: the EN welcome says "Feed"; the EN unreachable reason no longer claims a change;
  stale comments about PlayerModal's scroll lock and dialog contract.
2026-09-13 17:17:49 +02:00
peter bbd31c7143 feat(keys): Back returns focus to the item a view was opened from; "Feed" everywhere; every backend id named
Owner answers to the E11 S5 raises (2026-09-13): 1 → A, 2 → "Feed" in every language, 3 → translate the
proxy label and fix every config key missing in BOTH languages ("maintenance", "tmdb" — old bugs).

Focus across a view change (1-A). Opening a Plex title from Home's rail, a channel from the Channels
table or a title from a Plex grid switches the page or the sub-view, which unmounts the list; Back used
to hand focus to <body>, so the next `j` started over at the top. The YouTube player (an overlay)
already restored onto the card.
- `lib/listNav`: `navPosition()` (list's place in reading order, the page's list count, item index)
  and `restoreNavPosition()`: focuses the item once the page has as many lists on screen as when it was
  left, polls on a timer (a hidden tab runs no animation frames), gives up after 2 s or at once when
  something already holds focus, focuses without scrolling (the scroll restore owns the position).
- `lib/history`: `rememberNavPosition()` stamps `sfNav` onto the entry being left and returns the state
  without it (a new entry must not inherit it); one popstate listener restores and spends the stamp,
  so a later landing on the same entry (an overlay closing over it) cannot pull focus to an old item.
- Called before every view push: `setPage`, `openChannel`, `enterYtSearch`, `useHistorySubview.open`,
  the Channels tab push. `navPosition.wiring.contract.test` fails on any `history.pushState(` without
  it (the overlay stack's own push is the one exemption).

"Feed" (2). The module is "Feed" in both languages: every "hírfolyam" in the Hungarian UI (≈45 strings,
suffixed: Feedből, Feeded, Feedben…) and the prose "feed" in English that means the module ("your
Feed", "Hide from Feed"); "RSS feed" keeps its meaning. The welcome email says it too.

Backend ids without a name (3). The Configuration page showed `maintenance`, `tmdb`,
`maintenance_revalidate_batch`, `download_upload_max_bytes`, `audit_retention_days`, `plex_overlay_map`,
`tmdb_api_key` and `tmdb_language` as raw identifiers — the lookup falls back to the id, and the i18n
parity test stayed green because both languages missed the same keys. Added labels + hints (EN/HU),
the two groups (and their tab order), the missing `scheduler.jobDesc.demo_reset`, removed the stale
`plex_sync_interval_min` field, and "YouTube API kimenő proxy". `i18n/backendIds.contract.test` reads
`sysconfig.SPECS`, `scheduler.JOB_INTERVALS` and the audit/quota action enums from the backend sources
and fails on an id without its name in either language, or a name left for an id the backend dropped.
2026-09-13 07:36:41 +02:00
peter 8731508645 feat(keys): the conversation list walks with j/k; the cheat sheet names keys in the reader's language
E11 S5 — the HU/EN sweep and the keyboard-only pass over every module.

- The keyboard cheat sheet printed "Space" and "Menu" on a Hungarian sheet: a keycap that carries a
  WORD (not a legend every layout prints alike, like Esc, Ctrl or Enter) is now named by the caller in
  the reader's language (`formatChord(chord, words)`, `keys.cap.*`: Szóköz / Menü).
- The Plex player's ⌫/Esc line said "Stop & back to feed", but the player goes back to wherever it
  was opened from (Home, a Plex page): "Stop & go back" / "Leállítás és vissza".
- Messages: the conversation list is a keyboard list (NavGrid + navItemAttrs, the name button is the
  open element; the pop-out button keeps its own Enter). D1 covers every list whose rows open one
  thing; this one was missed in S4. Notification cards have no single open action and stay out, like
  the download queue.

Sweep, measured: every static `t("…")` key used in the source exists in both languages (the only
misses are template prefixes); the HU values equal to EN are proper nouns and loanwords (Admin, Plex,
Trakt, Shorts). Palette in Hungarian: "téma" makes "Téma: Midnight" the active row ahead of the channel
"Topik - mindig témában", live region "7 találat".

Keyboard-only pass in Chrome, Hungarian UI — 1024×768: every `g` chord lands on its module, `j` rings
the first item (solid 2px accent) on Home, Feed, Channels, Playlists, Plex and Downloads, `s` focuses
the search on Feed, Channels, Playlists and Plex, no page scrolls sideways. 3440×1440: `j` → → ↓
walks Home, Feed (8 columns: 0 → 1 → 9), Channels, Playlists, Plex and Downloads, no sideways scroll.
The help sheet lists every line translated; inside the Plex player "Szóköz / K" and "⌫ / Esc
Leállítás és vissza". Messages: a real `j` rings the row, Enter opens the thread.
2026-09-13 07:00:51 +02:00
peter b5d37ca142 feat(keys): j/k and the arrows walk the channel, playlist and download lists
Third and last slice of the E11 keyboard layer (S4c). S4a/S4b covered every card grid and rail; the
table-shaped pages — the channel manager, a playlist, the download center — were still Tab-stop
country: a channel row carries a priority stepper, four action buttons and a tag picker before the next
row's name.

- `ui/DataTable keyboardNav`: the rows are the items, Enter clicks the element a column marks
  `data-nav-open`. DataTable renders its rows twice (a table, and a card list below `md`); both register
  and `lib/listNav` now walks only lists on screen (`checkVisibility`), so `j` past the last row never
  carries on into the hidden copy.
- `lib/listNav` leaves a key something before it already handled (`defaultPrevented`): during a keyboard
  drag a playlist row's grip moves the row with the arrows, and dnd-kit takes the key on the document
  before listNav's window listener. Without the guard ↓ moved the focus off the grip and dropped the drag.
- `ui/NavGrid as="tbody"` for a table whose body is the list (the playlist, under dnd-kit's
  SortableContext).
- Wired: Subscriptions and Discovery tables, the card layout (`ChannelLayoutGrid` → VirtualGrid
  `keyboardNav`), Hidden/Blocked cards (the ChannelLink name is a channel's open element); a playlist's
  rows (the title plays; the grip keeps Space/Enter); the download queue, library and shared lists (the
  play button opens, `IconBtn open`; a row without one opens nothing).
- `listNav.wiring.contract.test`: `VirtualGrid` and `DataTable` must say `keyboardNav` like `Carousel`,
  `DownloadRow` is a card; the audit log's table is the one exemption (its rows open nothing), and an
  exemption that no longer matches a tag fails the test.

Measured in Chrome: Channels table j → row, ring round the whole row; a row walked down keeps 12px above
the viewport edge, walked up clears the sticky header by 35px; Enter opens the channel. Cards 3 columns:
→ 0→1, ↓ 1→4. Hidden and Discovery: j + Enter open the channel. Playlist: j + Enter plays, Escape returns
focus to the row; a keyboard drag (Enter on the grip, ↓) moves the row with focus kept on the grip, and
Escape cancels with the order unchanged. Downloads library: Enter plays a video row, does nothing on a
plain file row.
2026-09-13 06:08:05 +02:00
peter eb0507f942 fix(keys): a card reached by Tab rings the whole card, not just its poster
Owner report: on the Plex browse grid, Tab onto the first card ringed only the poster, and the first
arrow key then moved the ring round the whole card. The poster is the card's own role=button Tab stop,
so Tab gave it the app-wide button ring, while listNav's arrows focus the card root (D2: the ring goes
round the whole item).

The card's "open" element (`data-nav-open` — the Plex poster, a video card's title link) now stands
for the card: when it holds keyboard focus the card root wears the ring and the element's own ring is
transparent. Controls inside the card (watched toggle, actions) keep their own ring.

Measured in Chrome (jsdom evaluates neither :focus-visible nor :has): Plex browse, Tab from the sort
control → poster focused, card outline 2px accent, poster outline transparent; → moves to card 1 with
the same ring. Feed, Tab onto a card's title link → the card is ringed, the link is not.
2026-09-13 05:47:43 +02:00
peter 6d09b94bbb feat(keys): j/k and the arrows walk Home's rails and every Plex grid and rail
Second slice of the E11 keyboard layer (S4b). S4a gave the feed, the library and the channel page
`j`/`k`, the arrows and Enter; Home's shelves and the whole Plex module were still Tab-stop country —
a Plex poster is its own Tab stop, and a rail's cards sat behind every control of the cards before.

- `lib/listNav`: a list that mounts every item no longer declares a count or columns — both are
  measured off the DOM when a key is pressed, so an auto-fill grid's columns follow its width and a
  rail is one row. ↑ on a list's top row and ↓ on its bottom row go on to the neighbouring list, onto
  the item horizontally nearest the one left (owner decision D4: ←→ within a rail, ↑↓ to the rail
  above or below). The first `j` skips cards a rail has scrolled out to the left.
- `Carousel keyboardNav` registers the scroller; the scroller gets a 4px inset (against a -4px margin,
  layout unchanged) so the focus ring is not clipped, and `[data-nav-index]` a 2.5rem inline scroll
  margin so a card stepped into view clears the rail's edge fade and paging arrow.
- `ui/NavGrid`: the plain, fully mounted grid that registers itself.
- `navIndex` on PlexPosterCard / SeasonCard / EpisodeCard (the card root is the item, the poster the
  open element — the ring goes round poster and title, D2), PosterTile and Home's ShelfTile (the button
  is the item); cast, related and collection-strip buttons carry the attrs directly.
- Wired: Home's comfort and resume rails; Plex Browse titles + episodes, Discover rails + genres,
  collection view, playlists/collections browse, person known-for + filmography, show seasons, season
  episodes, cast/related strips, info-page cast + collection strips.
- `listNav.wiring.contract.test`: every `Carousel` says `keyboardNav`, every card says `navIndex`, and a
  card with `navIndex` sits inside a NavGrid / Carousel / VirtualGrid — both halves are opt-in, and
  either one missing leaves a list the keyboard silently skips.
2026-09-13 05:38:36 +02:00
peter 5274216981 feat(ui): PageUp/PageDown page through every dropdown list, not just the palette
The palette got PageUp/PageDown, and the same crawl stayed everywhere else a list opens under a
control: the sidebar's channel filter, the Plex genre/studio/collection pickers, the filter and sort
menus, submenus of every playlist you own. floating-ui's list navigation has only the arrows and
Home/End.

- `lib/listPaging`: the one definition of a page (the rows wholly in view minus one, a fixed 10 with
  no layout) and of where it lands (stops at both ends, never on a row it may not take, looking back
  toward the start rather than past it). The palette now uses it instead of its own copy, and
  `listNav` takes its scroll-parent helper from here.
- `AnchoredPopover`: one keydown handler on the reference AND the surface, so a combobox (keys on its
  input, `aria-activedescendant`) and a roving menu (keys on the focused row) both page. It runs after
  floating-ui's own, which has by then switched to keyboard modality and scrolls the new row in. A
  disabled row is never the target; a submenu's key does not also page the menu around it; with no
  mounted rows (closed, a hover card, a dialog) the key keeps its native meaning.

Measured in the browser: the channel filter shows 8 rows and pages by 7 with the active row in view
after the scroll; a menu pages to its last row and back, the page behind it does not scroll; the
palette still pages by 11 of its 12 visible rows.
2026-09-13 05:20:53 +02:00
peter 75d89613d0 feat(palette): PageUp/PageDown move a page of results at a time
The jump-to palette lists every module, channel, playlist, view and action — 461 rows on the
owner's account, ~11 of them in view — and the arrows were the only way through it: forty presses to
cross the channel group.

- PageDown/PageUp step by the rows that fit in the list right now, minus one, so the row you paged
  from stays in sight at the new page's edge. Measured per press (the list is 50vh and group headings
  eat into it); a fixed 10 when there is no layout to measure.
- Unlike the arrows they stop at the ends instead of wrapping: a page-sized jump that lands back at
  the top reads as having lost your place.
- Home/End stay the text field's own.

Measured in the browser: 0 → 10 → 20 rows, stops on row 461, PageUp from the end steps 9 (a group
heading is in view), the selected row stays visible throughout.
2026-09-13 05:03:35 +02:00
peter 8fb276aa90 feat(keys): j/k walk the feed, s finds the search box, g jumps to a module
A feed card cost ~10 Tab stops, so the tenth card sat ~135 Tabs from the top of the page, and the
only keyboard focus ring (a 60% accent outline on the thumbnail) all but vanished on the dark glass.
First slice of the E11 keyboard layer (S4a); Home, Plex and the tables follow in S4b/S4c.

- `lib/listNav`: one window listener over every registered list, walked in document order. `j`/`k`
  step item to item (on into the next list — owner decision D4), the arrows move around a grid, Enter
  clicks the item's own `data-nav-open` element, so it opens exactly like a click. An item root is
  never a Tab stop: it gets `tabindex=-1` only while the keyboard is on it, and keeps it until the next
  item takes it, so the player's focus restore lands back on the card. The first `j` takes the first
  card you can see, not the top of the list.
- `VirtualGrid` takes part with `keyboardNav` (column count + scrolling an unmounted row in);
  `VirtualFeed` turns it on, so the feed, the library and the channel page all have it.
- Keymap: two-step sequences (`g h`) whose pending step belongs to the scope that started it; the
  page-level scopes (`page`, `search`, `list`) are quiet while a layer that owns the keyboard is open,
  enforced in `dispatch`. A test pins the owner's rule that none of them needs a modifier on the
  Hungarian layout.
- `g h/f/c/p/x/n/m/d/s/,` jump to a module this account can reach; `s` focuses the header search box
  and selects the old query; Escape in it hands the keyboard back to the page (Plex's typeahead spends
  its first Escape on its panel).
- Hover popovers are passive layers: an open tooltip still takes Escape first but no longer silences
  the page keys.
- Focus ring (owner decision D2 = B): solid 2px accent around the whole item; the app-wide button/
  link/row ring goes from 60% to solid too. The item keeps a transparent resting outline because the
  video card transitions `all` — without it every `j` flashed a 3px white ring (measured in Chrome).
- Help sheet: the new keys, in a fixed order after the surface's own; a `g` chord only for a reachable
  module; opened over a player or dialog it leaves out the page-level keys and the context-menu key,
  which nothing can reach there (the latter pre-existing since E10).
- CLAUDE.md: a canonical-patterns row for list/grid keys (`useListNav` + `navItemAttrs` +
  `data-nav-open`, not a private arrow-key handler).
2026-09-13 04:16:20 +02:00
peter 78a9324381 fix(ui): the palette ranks what you typed, and lib stops leaning on components
Typing "theme" put the channel "The Meltzone Podcast" (matched only through its handle
`@themeltzone`) above every "Theme: …" row, and Enter opened the channel — the palette listed
matches in pure group order and always highlighted the first one.

- CommandPalette ranks each word: label starts with it > elsewhere in the label > only the hint,
  keywords or group name; an item ranks by its worst word, then the sum. The group order stays
  fixed, rows are ranked inside each group, and the best row overall starts highlighted. The
  highlight is held by id and follows the best match until the arrows or the pointer pick a row,
  so a channel list landing mid-typing no longer shifts it.
- A11y: result groups are `role="group"` named by their heading; `aria-expanded` is false with
  nothing to choose; the listbox stays mounted (hidden) so `aria-controls` never names a missing
  element, and "Nothing matches" sits outside it; a polite status announces the count once the
  typing pauses.
- `Column`/`FilterMap`/`filterRows`/`isActive` move from `ui/tableHeader` to `lib/tableColumns`,
  and the player's auto/loop modes from `playerModalControls` to `lib/playerModes` — three lib
  helpers imported the column contract from a ui component, and `playerModalHooks` its types from a
  feature component.
- eslint: `lib/` may not import from `components/` at any depth, type-only and re-exports included;
  the boundary contract test proves it fires. CLAUDE.md and the ui README say so.
- keymap: `dispatch`'s docblock sat stacked on `matchesShortcut`'s; moved onto `dispatch`.
2026-09-13 02:14:07 +02:00
peter 8a50538275 fix(ui): a write that fails says what failed and why, wherever it fails
`req` opens its blocking dialog for 400/409/422/5xx only. A 403/404/429 was left to the caller and a
502-504 or lost connection got just the self-clearing connection notice, so 34 mutations without a
reporting onError and 15 direct api writes with a bare or missing catch failed in silence: a hidden
channel that snapped back, a share link that never revoked, a confirmed account delete that ended
with nothing said.

- lib/queryClient: one MutationCache reporter. react-query awaits it BEFORE the mutation's own
  onError, so it waits a task and skips a failure the site already reported; the toast title comes
  from `meta.failTitle`, `meta.errorShownInline` opts out.
- notifyActionError: title names WHAT failed, the line under it says WHY (`failureReason`: a
  structured localized detail, else the status). It marks the error reported (`markReported`), so
  nothing reports one failure twice; `notifyKnownFailure` for a site's own specific sentence.
- Every silent site named (45 new HU/EN titles, 4 reasons). Stale "the global dialog shows it"
  comments corrected on the job actions, channel block, Plex sync, cookies, download enqueue,
  account delete and usePref.
- Channels passed the tag hook's channel id to the reporter as the error; the tag and priority hooks
  now report themselves, with the real error.
- A contract test keeps every useMutation naming its failure; CLAUDE.md gains the pattern row.
- history.dom.test: the "navigates only once the entries are back" assertion raced an async act's
  macrotask flush (1 in 5 runs); it now asserts from a later popstate listener.
2026-09-13 01:45:28 +02:00
peter b4aea7fa5b fix(ui): dialogs, menus and busy buttons keep the keyboard focus where it belongs
Measured in Chrome before each change:

- A dialog opened from a menu lost its focus: the menu row ran its action
  and closed in one batch, and floating-ui's focus return (a microtask)
  moved the focus to the row BEHIND the confirm dialog's scrim 7 ms after
  it opened, where Enter activated the row. MenuItemList now closes first
  and runs the action a task later; the dialog also captures the row, not
  a dying menu item, as its restore target.
- A side panel opened as a flyout collapsed the moment a dialog opened
  from inside it, unmounting the row the dialog hands the focus back to.
  useHoverFocusWithin holds while the focus is inside an aria-modal dialog.
- ui/Button's loading state set native disabled; Chrome moved the focus of
  the pressed button to <body> and never gave it back. Loading is now
  aria-disabled with a click guard that also cancels implicit submission.
- Escape/Enter in the playlist rail's rename field dropped the focus on
  <body>. The row refocuses from its ref callback when it re-attaches (a
  post-commit focus sticks; the earlier attempt ran before the button
  existed).
- PlayerModal was the one aria-modal shell without the dialog contract:
  it is now named by its title, traps Tab, restores the focus to its
  opener and shares one counted body scroll lock with every dialog.
2026-09-13 01:04:11 +02:00
peter 92c305bab9 fix(nav): a navigation closes every overlay before it pushes
The overlay history stack let a navigation land ABOVE open overlays and
"buried" their entries with a counter. Measured in Chrome: Shift+W over
About navigated under the still-open dialog, the next Back both undid the
navigation and closed About, and the Back after that did nothing; Shift+W
on Home with the player open did nothing at all. The counter could also be
spent by a different overlay closing first, and callers that could not
report whether they pushed were assumed to have.

setPage and openChannel now close every open overlay and push only once
the history is back in sync (one entry per live overlay, no programmatic
back() in flight, no reconcile queued), so nothing is ever buried and the
whole mechanism goes: navigateOverOverlays, closeThenNavigate, buryUnder,
setPage's boolean, and the three test files that pinned them. The palette
closes only itself before running a row, so a theme picked over the player
keeps the player. Shift+W on Home with the player open now closes it
(owner decision 2026-09-09).

The wait is on state, not on "a popstate": the old 250 ms fallback could
run with back()s still in flight. An overlay that refuses to close within
1.5 s drops the command instead of being navigated out from under.
2026-09-13 00:17:09 +02:00
455 changed files with 20589 additions and 4009 deletions
+8
View File
@@ -28,6 +28,14 @@ ALLOWED_EMAILS=
# Admin emails (subset of the above) get the admin role.
ADMIN_EMAILS=
# Logging (E22). LOG_LEVEL is the BOOTSTRAP level — the level a process starts at and the only one
# the worker/download child read; the API's running level is the admin Configuration page's
# `log_level` knob, which takes effect without a restart. TRACE | DEBUG | INFO | WARNING | ERROR, in
# any letter case; any other value stops the boot with an error. LOG_FORMAT is "json" (one object per
# line — what a log collector ingests) or "text" (a readable line for a local terminal).
LOG_LEVEL=INFO
LOG_FORMAT=json
# Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
FRONTEND_ORIGIN=
+118
View File
@@ -7,6 +7,124 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.88.0 — 2026-09-18
When something goes wrong, you can now turn on detailed logging for yourself for 24 hours — and read your own log on the spot, before any of it is sent anywhere. Siftlode also has a privacy policy and terms at last, in both languages, saying exactly what is recorded and for how long.
### Features
- Settings → Diagnostics: choosing Detailed or Full changes nothing until you confirm a box that tells you what will be recorded, that the server keeps it, for how long, and that it switches itself off after 24 hours. Going back to Basic needs no confirmation and takes effect at once.
- “My log”, on the same tab, shows what your browser has recorded — newest first, any line openable. Nothing leaves the browser before you have consented.
- What you search for and filter by is recorded only at Full, and only under your own consent — never an administrator's. Passwords, tokens, cookies and email addresses are recorded at no level, by anyone.
- Your last sign-in is shown in your account panel, and in the administrator's user list.
- A privacy policy and terms of service, in Hungarian and English, covering what the server logs and what each diagnostics level adds.
- Administrators get a Diagnostics section in Configuration: the log level takes effect without a restart, and stored logs are cleaned up on a schedule (7 days by default).
- Every setting with a fixed set of values is now picked from those values instead of typed in, so a wrong one can no longer be saved.
### Fixes
- Every message arrived twice if you left a conversation and came back quickly: two chat connections stayed open where there should have been one.
- Settings was unusable on a phone — the tab rail took the width and left almost nothing for the panel beside it.
- Dates and numbers follow the language you picked in Siftlode, not the one your computer happens to be set to. An English Siftlode on a Hungarian machine no longer shows Hungarian dates.
- An administrator setting shown as a dropdown opened on its first option instead of the one actually in force.
- In Hungarian, the audit log and the new log settings had names you could not tell apart.
- The scheduled feed poll now covers only channels somebody actually follows — on this instance 582 of 979 polled channels had no follower at all. The trade-off is deliberate: new uploads from a channel nobody follows no longer appear in the shared Library.
### Internal
- One logging mechanism replaced four. The API, the worker, the downloader and the migration tool now write the same structured line; secrets are scrubbed where the line is written rather than at each of ~125 places that write one; and every request carries an id that joins a line from your browser to the server line of the same request. The browser's log uses the same shape and the same redaction rules, checked from both sides against one shared set of cases.
- Two adversarial review rounds over the epic: 56 findings raised, 32 survived three independent attempts to refute each, all fixed. Every regression test was proven to fail with its own fix removed — which is how a test that passed only on leftover state from an earlier test was caught, and how a focus bug that jsdom could not reproduce at all was settled in a real browser.
## 0.87.1 — 2026-09-16
A YouTube video that YouTube will only show to a signed-in viewer can now be downloaded: paste your YouTube cookies in Settings the way the failure message already told you to — until now that list had no YouTube in it.
### Features
- YouTube joins the sites you can store download cookies for (Settings → Account → Download authentication). A signed-in YouTube download automatically uses the players that accept your login.
### Fixes
- A YouTube download that stops at “Sign in to confirm you’re not a bot” says so in your language and points at a setting that now exists. If you had already added cookies, it tells you they were refused instead of asking for them again.
- A video YouTube describes but then refuses to hand over is tried again with another player instead of failing at once, and it is no longer called a network error — nothing was wrong with the network.
### Internal
- The gated video was measured end to end on an isolated stack before and after the fix: every player client, two yt-dlp versions, and finally the real download (519 MB) with cookies attached. One deep-review round — 25 findings raised, none survived adversarial verification, and the one real defect came out of the verification itself: a comment of mine named an error yt-dlp does not raise there. Running it settled what reading it could not.
## 0.87.0 — 2026-09-15
A shared or library video keeps playing when you switch apps on an iPad, and a new tab opens as the account it belongs to without flashing another page first. Facebook reels keep their titles and an mp3 stays an mp3.
### Features
- Background play on iPad (Brave): the shared-video page, the library player and the Plex player keep playing in another app and on the locked screen.
- The shared-video page has a Repeat toggle, on by default.
- Switching modules no longer flashes a loading screen: the pages load quietly in the background, and a slow answer shows its spinner only after a moment.
### Fixes
- A new tab honours your Startup choice and opens as its own account — no Home page flash, no other account's language or settings on the first frame.
- A signed-in Facebook reel keeps its real title instead of “Video” or “Facebook”, and an mp3 download stays an mp3 (it used to be wrapped into an mp4 an iPad would not play).
- A tab whose account was signed out, suspended or signed out everywhere acts as nobody and comes back as the account still signed in, instead of writing into another account or sticking on the sign-in page.
- Switching or adding an account no longer saves the video you were watching into the other account's resume position.
- A new tab keeps the account's notification history instead of replacing it with its own first notice.
- The shared-video page tells a failed load, a wrong password, too many attempts and an unreachable server apart, with Retry where it helps. Its password limit counts per viewer, so one person guessing cannot lock everyone else out.
- Form errors on sign-in, the message passphrase, password change and setup are read out by screen readers — again on a second identical failure — and the setup wizard names what is wrong instead of a generic error.
### Internal
- WebKit and iPad projects in the Playwright suite; the Vite dev proxy forwards websockets.
- Four deep-review rounds (12 → 8 → 8 → 8, the last one targeted), every regression test mutation-checked; half of the later rounds' findings were older defects, fixed here.
## 0.86.0 — 2026-09-14
A failed download now tells you why it failed and what you can do about it, instead of repeating yt-dlp's raw error. Facebook comes first: downloads that used to die on “Cannot parse data” go through, and a Facebook group post downloads its own video instead of a different one from the page.
### Features
- A failure says why, in your language: the row names the cause (“Facebook needs you to be signed in”, “Facebook did not accept your cookies”, “No video found in this post”) and its details window says what to do next. The raw error text is still there below it.
- A Facebook download with your cookies that stops at “Cannot parse data” is tried again as Chrome, which gets past it.
- A Facebook group post link downloads the post's own video. Before, yt-dlp either found nothing or quietly downloaded another video from the page; a post without a video now says so.
- Retry and Re-download use a cookie you added after the failure, so the Retry the message asks for actually goes out signed in.
### Fixes
- A Re-download of a download that needed your cookies used to rebuild without them.
- A job queued behind a download that already failed shows that failure's explanation instead of a bare “Download failed”.
- A download yt-dlp reports no length for takes its length from the file, so its Edit / trim action appears.
- Pressing Pause, Stop or Remove twice, or in two tabs, can no longer free a file another download still uses.
- The confirm buttons in Downloads name the action (Stop download, Re-download, Remove) instead of repeating the question.
### Internal
- Four sprints, each Facebook behaviour measured on the development server with a real cookie before it was built. Two epic-end review rounds (20 → 17 fixed, then a targeted 4 → 4), every regression test mutation-checked.
## 0.85.0 — 2026-09-13
Siftlode can now be driven from the keyboard, and Back, dialogs and menus stop losing your place. j/k and the arrow keys walk every list and grid, Enter opens what you are on, and Back puts you on the item you opened a view from instead of at the top of the page.
### Features
- j/k and the arrow keys walk the feed, the library, a channel page, the Home rails, every Plex grid and rail, the Channels, Playlists and Downloads tables and the conversation list; Enter opens the item you are on. After you hide or delete an item, the next j goes on from where it was.
- Back returns you to the item a view was opened from — the Plex title you opened from a Home rail, the channel you opened from its row — with the keyboard still on it.
- Shortcuts to get around: s jumps to the search field, g-chords jump to a module, and the cheat sheet (h) names every key in the language you are reading. PageUp/PageDown move a page at a time in the command palette and in every dropdown list.
- The command palette ranks what you typed first: “theme” now puts the Theme rows above a channel whose handle happens to contain it.
- A card reached with Tab or j shows one ring around the whole card, and a screen reader calls it by its title.
### Fixes
- Back and overlays keep their books straight: Back after navigating over About also closes About, a late clean-up no longer undoes the navigation you just made, and Shift+W on Home closes the player.
- Dialogs and menus hand the keyboard back where it came from. Escape from a card’s context menu returns to the card, not its thumbnail link; closing the player under another dialog no longer leaves the page unscrollable; a button that is busy with a request keeps the focus instead of dropping it to the page.
- A write that fails says so, with the reason — several saves, share links and admin actions used to fail silently or in English under the Hungarian interface.
- A split video export that fails part-way (the job limit) tells you how many clips were added and leaves only the rest selected, so trying again does not queue the first ones twice.
- “Feed” is called Feed everywhere, in both languages.
### Internal
- Five sprints: history integrity, one dialog/focus contract, palette ranking and accessibility, the keyboard layer, and a Hungarian/English keyboard-only pass. Focus and event-order questions were measured in a real browser before each fix, not argued.
- Three epic-end review rounds — 19 → 14, 18 → 12 and a targeted 14 → 14 — each regression test mutation-checked. The last round's findings all sat inside the previous round's own fixes, which is where the loop was closed.
## 0.84.0 — 2026-09-12
A bug-fix release, from a list you wrote while using Siftlode. Two of these were destroying your own data: a YouTube video rewound itself when you reopened it, and clearing a resume point on a half-watched Plex item told Plex nothing. Subtitles now appear in seconds instead of minutes, the Plex player's bitrate is a real measurement rather than a guess, and a channel you turn full history on for fills in front of you instead of looking frozen.
+9 -3
View File
@@ -58,13 +58,14 @@ how it gets built. Concretely:
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`,
`security`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
`security`, `logs`, `logsink`, `diagnostics`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → module
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
boundary is one-way and **lint-enforced**, not just described: a platform component may not import a
feature component or a domain API slice (`no-restricted-imports` in `eslint.config.js`, with its own
test). A new generic component goes in `ui/`; see [`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
feature component or a domain API slice, and one layer down `lib/` may not import from `components/` at
all, not even a type (`no-restricted-imports` in `eslint.config.js`, with its own test). A new generic
component goes in `ui/`; see [`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
Data via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through
`lib/api/*` slices over `core.req<T>`.
@@ -89,8 +90,13 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE action button | `ui/Button` — pick a `SIZES` rung | a hand-rolled `<button className="px-3 py-2 …">` |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
## House rules that bite (learned from real bugs)
+1 -1
View File
@@ -1 +1 @@
0.84.0
0.88.0
-19
View File
@@ -1,19 +0,0 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
RUN adduser --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
EXPOSE 8000
CMD ["sh", "entrypoint.sh"]
+2
View File
@@ -1,6 +1,8 @@
[alembic]
script_location = alembic
prepend_sys_path = .
# Split the path list the way the OS does (alembic's legacy space/comma/colon split is deprecated).
path_separator = os
[loggers]
keys = root,sqlalchemy,alembic
+4 -4
View File
@@ -1,17 +1,17 @@
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from app.config import settings
from app.db import Base
from app.logs import configure_logging
import app.models # noqa: F401 (import so models register on Base.metadata)
config = context.config
config.set_main_option("sqlalchemy.url", settings.database_url)
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# Migrations run at container start (entrypoint.sh), so their output is part of the API's log
# stream — one format with everything else, not alembic.ini's own fileConfig shape.
configure_logging("alembic")
target_metadata = Base.metadata
@@ -0,0 +1,41 @@
"""add download_jobs.error_detail — the domain-specific entry of the failure catalog (E11.5 S1)
Revision ID: 0084_dl_job_error_detail
Revises: 0083_comfort_pins
Create Date: 2026-09-13
`error_code` is the generic category (it decides whether a Retry is offered). `error_detail` names
the host-specific catalog entry `downloads.errors.classify` matched on top of it — `facebook.login_wall`
and its siblings — so the UI can say what went wrong ON THAT SITE and what to do about it, instead of
the category's generic sentence. Nullable: most failures have no host-specific entry.
Backfill: a job's code is written once, at failure time, so a new rule never reaches the rows that
failed before it. The one shape measured on prod (2026-09-13, jobs 9/66/75/145/156/157/182/256/257) is
yt-dlp's `[facebook] <id>: Cannot parse data`, stored as `unknown`. Without a cookie that is Facebook's
login wall. Whether a cookie was attached at the time is not recorded anywhere, so every old row gets
the no-cookie entry — on prod no account had a Facebook cookie, so that is what they all were.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0084_dl_job_error_detail"
down_revision: str | None = "0083_comfort_pins"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("error_detail", sa.String(length=48), nullable=True))
op.execute(
sa.text(
"UPDATE download_jobs SET error_code = 'login_required', "
"error_detail = 'facebook.login_wall' "
"WHERE (error_code IS NULL OR error_code = 'unknown') "
"AND error ILIKE '%[facebook]%cannot parse data%'"
)
)
def downgrade() -> None:
op.drop_column("download_jobs", "error_detail")
@@ -0,0 +1,26 @@
"""add users.last_login_at — when the account last signed in (E22 S1)
Revision ID: 0085_users_last_login_at
Revises: 0084_dl_job_error_detail
Create Date: 2026-09-16
Stamped by the one sign-in path (`routes/auth._establish_session`), which the Google callback and
password login share. No backfill is possible: the signal exists only at sign-in, so existing rows
read NULL until each account signs in again, and the admin UI says so rather than inventing a date.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0085_users_last_login_at"
down_revision: str | None = "0084_dl_job_error_detail"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("users", sa.Column("last_login_at", sa.DateTime(timezone=True), nullable=True))
def downgrade() -> None:
op.drop_column("users", "last_login_at")
@@ -0,0 +1,48 @@
"""add log_events — stored log events for diagnostics (E22 S3)
Revision ID: 0086_log_events
Revises: 0085_users_last_login_at
Create Date: 2026-09-17
Written by app/logsink.py's background writer (WARNING and above, request lines), pruned by the log_gc
job after `log_retention_days`. Sized against prod on 2026-09-16: ~2k rows on a normal day, ~14k on a
YouTube-RSS-outage day before flood control — a few MB over the 7-day window. `user_id` carries no
foreign key on purpose (see models.LogEvent).
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0086_log_events"
down_revision: str | None = "0085_users_last_login_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"log_events",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("ts", sa.DateTime(timezone=True), nullable=False),
sa.Column("level", sa.String(10), nullable=False),
sa.Column("sev", sa.SmallInteger(), nullable=False),
sa.Column("source", sa.String(16), nullable=False),
sa.Column("logger", sa.String(128), nullable=False),
sa.Column("event", sa.String(64), nullable=True),
sa.Column("msg", sa.Text(), nullable=False),
sa.Column("attrs", postgresql.JSONB(), nullable=True),
sa.Column("exc", sa.Text(), nullable=True),
sa.Column("request_id", sa.String(64), nullable=True),
sa.Column("user_id", sa.Integer(), nullable=True),
sa.Column("debug_session_id", sa.String(64), nullable=True),
sa.Column("job_id", sa.Integer(), nullable=True),
sa.Column("version", sa.String(64), nullable=True),
)
op.create_index("ix_log_events_ts", "log_events", ["ts"])
op.create_index("ix_log_events_request_id", "log_events", ["request_id"])
op.create_index("ix_log_events_user_ts", "log_events", ["user_id", "ts"])
def downgrade() -> None:
op.drop_table("log_events")
@@ -0,0 +1,31 @@
"""add users.diagnostics_level + diagnostics_until — a user's consented diagnostics level (E22 S5)
Revision ID: 0087_users_diagnostics_level
Revises: 0086_log_events
Create Date: 2026-09-17
Both NULL = the base level (Alap). A raised level is ALWAYS paired with its expiry, computed by the
server at consent time (`app/diagnostics.py`), so no client can ask for a level that never ends.
Columns rather than `users.preferences`: that JSON is written wholesale by the client, and a consent
with a lifetime must not be something a client can write.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0087_users_diagnostics_level"
down_revision: str | None = "0086_log_events"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("users", sa.Column("diagnostics_level", sa.String(8), nullable=True))
op.add_column(
"users", sa.Column("diagnostics_until", sa.DateTime(timezone=True), nullable=True)
)
def downgrade() -> None:
op.drop_column("users", "diagnostics_until")
op.drop_column("users", "diagnostics_level")
+4 -3
View File
@@ -20,9 +20,10 @@ Three layers, by *who* runs the code — not by feature:
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, … are single-file primitives many
layers import. They depend on nothing above them (a core module must never import a `routes/…`
module).
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table) and `diagnostics` (a user's consented log
level), … are single-file primitives many layers import. They depend on nothing above them (a core
module must never import a `routes/…` module).
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
*second* caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down
+67 -17
View File
@@ -6,12 +6,14 @@ from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from starlette.requests import HTTPConnection
from app import diagnostics
from app import email as email_mod
from app import logs
from app import mailtext
from app import sysconfig
from app.config import settings
from app.db import get_db
from app.models import Invite, User
from app.models import Invite, LogEvent, User
from app.security import decrypt_optional
from app.utils import canonical_email
@@ -245,6 +247,12 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
)
# Raw delete so Postgres applies ON DELETE CASCADE / SET NULL on every dependent table.
db.execute(delete(User).where(User.id == user_id))
# Stored log events name the account by id with no foreign key to cascade from (models.LogEvent),
# so they go explicitly — they are request history, i.e. personal data. Forgotten FIRST: this very
# request's access line, a queued row, a later background task would otherwise put rows back
# under the id after this DELETE (logs.forget_user).
logs.forget_user(user_id)
db.execute(delete(LogEvent).where(LogEvent.user_id == user_id))
# Every canonically-equal invite goes, not just the literal spelling: is_allowed matches on the
# canonical form, so leaving a sibling row behind would keep granting access to a mailbox whose
# account was just erased — and prod really does carry approved pairs for one mailbox.
@@ -269,21 +277,29 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
ACTIVE_ACCOUNT_HEADER = "x-siftlode-account"
def resolved_user_id(conn: HTTPConnection) -> tuple[int | None, bool]:
def resolved_user_id(conn: HTTPConnection, *, strict: bool = True) -> tuple[int | None, bool]:
"""Which account this connection acts as, and whether that's the session's default account.
Takes any HTTPConnection — an HTTP Request OR a WebSocket — so the WS push channel shares this
exact per-tab resolution instead of re-implementing it.
The signed cookie holds the *wallet* (`account_ids`, every account signed into this browser)
plus a default `user_id`. A caller may override the default with the X-Siftlode-Account header,
but ONLY for an account already in the wallet — so a tab can't impersonate an account that never
authenticated here. Everything else (WebSocket, plain navigations) keeps using the cookie default.
plus a default `user_id`. A caller may name its account with the X-Siftlode-Account header (or
`?account=`), but ONLY an account in the wallet — so a tab can't impersonate an account that
never authenticated here. A connection that names no account acts as the cookie default.
A connection naming an account NOT in the wallet acts as nobody (`None`) — typically a tab still
pinned to an account that was signed out in another tab. It must not fall back to the default:
that tab would go on acting as an account it never showed, its player checkpoints and pref saves
landing there and its Sign out signing THAT account out. Only the bootstrap /api/me probe passes
`strict=False` and gets the default, which is how such a tab learns who it now is (the frontend
re-pins it and reloads). An account whose session ended while it stayed in the wallet (deleted,
suspended, revoked) leaves the wallet when it is refused (`_session_user`), so a tab pinned to it
takes the same way back.
"""
default_id = conn.session.get("user_id")
wallet = conn.session.get("account_ids") or []
# Header for normal XHR; ?account= for contexts that can't set headers (WebSocket, and a
# plain <a> file download). Both are wallet-gated below, so neither can impersonate an
# account that never signed into this browser.
# plain <a> file download). Both are wallet-gated below.
hdr = conn.headers.get(ACTIVE_ACCOUNT_HEADER) or conn.query_params.get("account")
if hdr:
try:
@@ -292,28 +308,46 @@ def resolved_user_id(conn: HTTPConnection) -> tuple[int | None, bool]:
hid = None
if hid is not None and hid in wallet:
return hid, hid == default_id
if strict:
return None, False
return default_id, True
def drop_from_wallet(session: dict, user_id: int) -> None:
"""Take one account out of this browser's wallet (`account_ids`); the session default is untouched."""
session["account_ids"] = [a for a in (session.get("account_ids") or []) if a != user_id]
def _end_account_session(request: Request, user_id: int, is_default: bool) -> None:
"""An account whose session here is no longer valid. Only nuke the whole session when it is the
SESSION DEFAULT; a per-tab account just leaves the wallet (other tabs' default may still be valid).
Leaving is what lets a tab pinned to it recover: its next /api/me names an account no longer signed
in here and gets the default. Left in, it answered "signed out" — the landing page, with the default
still signed in, and no way off it but closing the tab."""
if is_default:
request.session.clear()
else:
drop_from_wallet(request.session, user_id)
def current_user(request: Request, db: Session = Depends(get_db)) -> User:
user_id, is_default = resolved_user_id(request)
return _session_user(request, db, *resolved_user_id(request))
def _session_user(request: Request, db: Session, user_id: int | None, is_default: bool) -> User:
if not user_id:
raise HTTPException(status_code=401, detail="Not authenticated")
user = db.get(User, user_id)
if user is None or not user.is_active or user.is_suspended:
# Suspended/deactivated mid-session → drop the session so the block takes effect at once.
# But only nuke the whole session when the SESSION DEFAULT identity is gone; a stale
# per-tab header account just 401s that one tab (other tabs' default may still be valid).
if is_default:
request.session.clear()
# Suspended/deactivated mid-session → end it so the block takes effect at once.
_end_account_session(request, user_id, is_default)
raise HTTPException(status_code=401, detail="Not authenticated")
# SA4: reject a cookie whose recorded epoch is behind the account's current one (revoked by a
# password reset/change or "log out everywhere"). A cookie with no recorded epoch (pre-SA4, or an
# account signed in before this shipped) is treated as epoch 0 — so the FIRST bump revokes it too.
epochs = request.session.get("epochs") or {}
if int(epochs.get(str(user_id), 0)) != (user.session_epoch or 0):
if is_default:
request.session.clear()
_end_account_session(request, user_id, is_default)
raise HTTPException(status_code=401, detail="Not authenticated")
# Always keep the session default in the switchable list — covers sessions created before
# multi-session existed (their account_ids was never seeded), so the switcher isn't empty.
@@ -322,15 +356,31 @@ def current_user(request: Request, db: Session = Depends(get_db)) -> User:
if default_id and default_id not in accounts:
accounts.append(default_id)
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
# Attribute this request's log lines to the account (E22). Here, not in the request middleware:
# this is the one place a session becomes an account, and the middleware runs outside both the
# session middleware and this task's context. `request.state` is shared across middleware layers
# (it lives on the ASGI scope), so the access-log line can read it back. The user's consented
# diagnostics level rides along: their requests are logged — and stored — in that detail (S5).
logs.set_user(user.id, diagnostics.effective_level(user), user.diagnostics_until)
request.state.user_id = user.id
return user
def optional_current_user(request: Request, db: Session = Depends(get_db)) -> User | None:
"""Like `current_user` but returns None instead of raising 401 when there's no valid session.
Used by the bootstrap /api/me probe so the logged-out landing doesn't log a failed request to
the browser console (a non-2xx fetch is a console error no matter how the app handles it)."""
the browser console (a non-2xx fetch is a console error no matter how the app handles it).
The one lenient resolution (`strict=False`): a tab naming an account no longer signed in here gets
the default account, which is how it learns who it now is — see `resolved_user_id`. That includes an
account refused right here, which `_session_user` has just taken out of the wallet."""
user_id, is_default = resolved_user_id(request, strict=False)
try:
return current_user(request, db)
return _session_user(request, db, user_id, is_default)
except HTTPException:
if is_default:
return None
try:
return _session_user(request, db, *resolved_user_id(request, strict=False))
except HTTPException:
return None
+65 -6
View File
@@ -1,6 +1,7 @@
from pathlib import Path
from typing import Literal, get_args, get_origin
from pydantic import model_validator
from pydantic import ValidationInfo, field_validator, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
# Shipped placeholder; production must override it. Used by the startup guard below.
@@ -20,6 +21,33 @@ def _version_from_file() -> str:
return "dev"
# Closed value sets. A setting typed with one of these is a CHOICE: pydantic rejects anything else at
# boot, and sysconfig derives the admin picker's options from the same Literal — so the list lives in
# exactly one place and a consumer never has to guess what a typo should fall back to.
SearchSource = Literal["scrape", "api"]
DownloadLayout = Literal["plex", "flat"]
LogLevel = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"]
LogFormat = Literal["json", "text"]
TranscodeHw = Literal["auto", "on", "off"]
# libx264's presets, fastest first (`placebo` left out on purpose: no measurable gain for its cost).
TranscodePreset = Literal[
"ultrafast", "superfast", "veryfast", "faster", "fast", "medium", "slow", "slower", "veryslow"
]
def choices_of(annotation: object) -> tuple[str, ...]:
"""The allowed values of a choice-typed setting, or () for any other type."""
return tuple(get_args(annotation)) if get_origin(annotation) is Literal else ()
def canonical_choice(options: tuple[str, ...], raw: object) -> str | None:
"""The option `raw` names, in its canonical spelling (any letter case, surrounding space
ignored), or None. The ONE matching rule for a choice value from the env and from the admin
API, so a spelling one accepts the other cannot refuse."""
folded = str(raw).strip().casefold()
return next((option for option in options if option.casefold() == folded), None)
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
@@ -57,7 +85,7 @@ class Settings(BaseSettings):
# VPS Caddy front door. A request from any other peer (someone hitting the app port directly)
# is NOT trusted and its raw socket IP is used, so it can't spoof its rate-limit identity via
# XFF. Empty = trust no proxy (use the direct peer) — correct for a directly-exposed / local-dev
# deploy. Comma-separated. See routes.auth._client_ip + the deploy docs.
# deploy. Comma-separated. See ratelimit.client_key + the deploy docs.
trusted_proxy_ips: str = ""
# --- Outbound email (onboarding: access-request + approval notices) ---
@@ -85,7 +113,7 @@ class Settings(BaseSettings):
search_daily_limit_per_user: int = 70
# Source for live YouTube search: "scrape" (InnerTube, no API quota) or "api" (search.list,
# 100 units/page). Scrape is the default; flip to "api" if scraping is blocked/breaks.
search_source: str = "scrape"
search_source: SearchSource = "scrape"
# Recent-first backfill: how far back to fetch on the first pass per channel.
backfill_recent_max_videos: int = 100
backfill_recent_max_days: int = 365
@@ -155,6 +183,9 @@ class Settings(BaseSettings):
# per-client responses are flaky, and android_vr can get bot-flagged under heavy load) the
# worker retries with the FALLBACK set — the POT-capable web clients (web needs the token +
# Deno) plus android for audio. Both admin-tunable when YouTube shifts what works.
# A SIGNED-IN download (the user added YouTube cookies) keeps only the cookie-capable clients
# of these two sets — yt-dlp drops the rest, so android_vr / android / ios never run there.
# See adapters.YouTubeAdapter._COOKIE_CAPABLE.
download_player_clients: str = "android_vr"
download_player_clients_fallback: str = "web_safari,web,android"
@@ -221,10 +252,27 @@ class Settings(BaseSettings):
audit_retention_days: int = 180
audit_gc_minutes: int = 1440
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: str = "plex"
download_layout: DownloadLayout = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
sponsorblock_default: bool = False
# --- Diagnostics / logging (E22) ---
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
# startup and on change without a restart. This is what a process logs at before it can reach
# the database — and the only knob the worker/runner children read, since they are short-lived.
# TRACE (below DEBUG) is the "Teljes" diagnostics level; see app/logs.py.
log_level: LogLevel = "INFO"
# "json" (one object per line — what Loki and the log_events sink ingest) or "text" (a readable
# line for a local terminal). Redaction applies to both.
log_format: LogFormat = "json"
# Also write events to the `log_events` table (app/logsink.py): WARNING and above, the request
# lines, and — once they exist — client batches. Off only where no table is wanted (the tests).
log_db_sink: bool = True
# Days a `log_events` row is kept (the privacy policy promises this), and how often the log_gc job
# enforces it — hourly, so a row never outlives the promise by more than an hour.
log_retention_days: int = 7
log_gc_minutes: int = 60
# --- Plex integration (optional) ---
# The Plex module plays the LOCAL physical media file directly (our own ffmpeg/direct-serve);
# Plex is used only for metadata + optional watch-state sync. Requires the Plex server to be
@@ -266,7 +314,7 @@ class Settings(BaseSettings):
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
plex_transcode_hw: str = "auto"
plex_transcode_hw: TranscodeHw = "auto"
# Software-fallback libx264 tuning (IGNORED on the VAAPI path). `preset` trades CPU for size; `crf`
# trades quality for size (lower = better + bigger). `max_height` caps the SOFTWARE transcode's output
# height (0 = keep source) so a CPU re-encode of a tall source stays affordable — the GPU path always
@@ -276,7 +324,7 @@ class Settings(BaseSettings):
# call, not a fix. Only affects files that go through remux/transcode — a direct-played file is
# served untouched, but those are browser-native stereo anyway, not the 5.1 mixes this is for.
plex_audio_loudnorm: bool = False
plex_transcode_preset: str = "veryfast"
plex_transcode_preset: TranscodePreset = "veryfast"
plex_transcode_crf: int = 20
plex_transcode_max_height: int = 1080
# Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
@@ -325,6 +373,17 @@ class Settings(BaseSettings):
https OAUTH_REDIRECT_URL as the signal for 'real deployment' vs local dev."""
return self.oauth_redirect_url.lower().startswith("https://")
@field_validator("*", mode="before")
@classmethod
def _canonical_choice(cls, value: object, info: ValidationInfo) -> object:
"""A choice setting accepts its value in any letter case (`LOG_LEVEL=debug`) and stores the
canonical spelling; an unknown value still fails the boot with pydantic's own message."""
field = cls.model_fields.get(info.field_name or "")
options = choices_of(field.annotation) if field else ()
if options and isinstance(value, str):
return canonical_choice(options, value) or value
return value
@model_validator(mode="after")
def _enforce_production_secrets(self) -> "Settings":
"""Fail fast rather than boot a public instance with the shipped placeholder
+61
View File
@@ -0,0 +1,61 @@
"""A user's diagnostics level (E22 S5): what they consented to, and for how long.
Three levels, named in the UI Alap / Részletes / Teljes:
* INFO — the base, for everyone, no consent. The browser keeps its log in memory; nothing extra
is sent or stored.
* DEBUG — consented. The server logs this user's requests in detail and stores every line of
them, and the browser streams its own log to `POST /api/diagnostics/events`.
* TRACE — consented. As DEBUG, plus what the user typed into a search.
A raised level always lapses after `ELEVATION_HOURS`: the expiry is computed HERE at consent time,
never taken from the client, and an expired level reads as INFO without anything having to clear it.
The shared demo account cannot raise its level — its log would mix every visitor's session.
Core layer: imports `models` and `sysconfig`.
"""
from datetime import datetime, timedelta, timezone
from typing import Literal, get_args
from sqlalchemy.orm import Session
from app import sysconfig
from app.models import User
DiagnosticsLevel = Literal["INFO", "DEBUG", "TRACE"]
LEVELS: tuple[str, ...] = get_args(DiagnosticsLevel)
BASE: DiagnosticsLevel = "INFO"
ELEVATION_HOURS = 24
def effective_level(user: User, now: datetime | None = None) -> DiagnosticsLevel:
"""The level in force for `user` right now: the consented one until it expires, else BASE."""
level, until = user.diagnostics_level, user.diagnostics_until
if level not in LEVELS or level == BASE or until is None:
return BASE
if until <= (now or datetime.now(timezone.utc)):
return BASE
return level # type: ignore[return-value] — narrowed by the LEVELS check above
def state(db: Session, user: User) -> dict:
"""The `/api/me` view of it: the level in force, when a raised one lapses, and how long stored
log lines are kept — the consent text states that number, so it must be the instance's own."""
level = effective_level(user)
return {
"level": level,
"until": user.diagnostics_until.isoformat() if level != BASE else None,
"retention_days": sysconfig.get_int(db, "log_retention_days"),
}
def set_level(user: User, level: DiagnosticsLevel) -> None:
"""Record a consent (or its withdrawal). The caller commits. Raising the level again restarts
the clock — each consent is for the next ELEVATION_HOURS, not an extension of an older one."""
if level == BASE:
user.diagnostics_level = None
user.diagnostics_until = None
return
user.diagnostics_level = level
user.diagnostics_until = datetime.now(timezone.utc) + timedelta(hours=ELEVATION_HOURS)
+159 -7
View File
@@ -11,10 +11,22 @@ Resolution works both before extraction (from source_kind / URL host) and after
`extractor_key` in the info dict), so a non-YouTube "url" job still lands on the right adapter.
"""
import re
from urllib.parse import urlparse
import yt_dlp
from yt_dlp.networking.impersonate import ImpersonateTarget
from app.config import settings
# Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
# below and the failure catalog (errors.py) decide on it, so the wording exists once.
FACEBOOK_PARSE_WALL = "cannot parse data"
# A source's own verdicts on a link, raised from `resolve_url`. errors.py classifies them; it imports
# them from here because this module must not import errors (errors → credentials → adapters).
POST_WITHOUT_VIDEO_MESSAGE = "No single video was found in this post."
POST_NOT_SHOWN_MESSAGE = "Facebook did not show this post to the signed-in session."
class SourceAdapter:
"""Generic source: yt-dlp handles everything and the ref is already a fetchable URL."""
@@ -22,9 +34,21 @@ class SourceAdapter:
key = "generic"
# --- extraction (worker._extract_with_fallback / backfill) ---------------------------------
def client_sets(self) -> list:
"""Player-client sets to attempt, in order. A generic source needs no client override, so a
single default attempt (`[None]`) — the per-client fallback is a YouTube-only concept."""
def resolve_url(self, url: str, ydl_opts: dict) -> str:
"""The link to hand yt-dlp for `url`, resolved BEFORE the first attempt. `ydl_opts` are the
download's own options (cookie, timeouts, address family) for any page this needs to read.
Raises a RuntimeError carrying one of this module's verdicts when the page read shows the
download cannot go ahead: the content is there but holds no single video
(`POST_WITHOUT_VIDEO_MESSAGE`), or the page is not the content at all — the login page a
refused cookie gets (`POST_NOT_SHOWN_MESSAGE`), never the no-video verdict. A failed page read
is raised as itself, so it is classified like any other download error. Generic: `url`."""
return url
def client_sets(self, authed: bool = False) -> list:
"""Attempts to make, in order — each entry is handed back to `ydl_opts_extra`, and the next one
is tried when `retriable` says so. A generic source makes a single default attempt (`[None]`).
`authed` is whether the acting user's cookie file goes with the download."""
return [None]
def ydl_opts_extra(self, clients) -> dict:
@@ -37,6 +61,10 @@ class SourceAdapter:
return False
# --- metadata -----------------------------------------------------------------------------
def title(self, info: dict) -> str | None:
"""The download's raw title, before `titles.normalize_title` cleans it. Generic: yt-dlp's."""
return info.get("title")
def uploader_url(self, info: dict) -> str | None:
"""The channel/uploader page URL. Generic: whatever yt-dlp exposes directly (YouTube gives
`channel_url`), else None (reddit / a raw media URL have no derivable channel page)."""
@@ -65,12 +93,46 @@ class YouTubeAdapter(SourceAdapter):
"Requested format is not available",
"Only images are available",
"Sign in to confirm",
# Measured 2026-09-16: a gated video whose EXTRACTION succeeds and whose media fetch is then
# refused by the CDN ("unable to download video data: HTTP Error 403"). The wall moves
# between the player response and the stream, so this is the same per-client flakiness one
# step later — and it used to end the job on attempt 1, because the text classifies as
# `network` and nothing here matched it. The format URLs belong to the client that minted
# them, so the next client set is a real second chance.
"unable to download video data",
)
# Measured in the worker against a REAL account cookie (2026-09-16, yt-dlp 2026.07.04): with
# `player_client=android_vr` yt-dlp logs `Skipping client "android_vr" since it does not support
# cookies`, drops it, and the request that is left with no client fails with "Requested format is
# not available" — which is already a retriable marker above, so the cost is a wasted attempt
# rather than a hard failure. Our primary set is android_vr ALONE, so without this filter every
# signed-in download would burn its first attempt that way before reaching the POT-backed web
# fallback the cookie exists for. The list is yt-dlp's own INNERTUBE_CLIENTS[...]
# ["SUPPORTS_COOKIES"], pinned by test_adapters against the installed yt-dlp, so a version bump
# that moves the table turns the gate red instead of silently un-authing every YouTube download.
_COOKIE_CAPABLE = frozenset(
{
"mweb",
"tv",
"tv_downgraded",
"web",
"web_creator",
"web_embedded",
"web_music",
"web_safari",
}
)
# Where a signed-in download lands when the admin's knobs hold no cookie-capable client at all.
# Both clients in ONE set: yt-dlp queries each and merges their formats, which is the cover for
# the per-client flakiness that the two-tier ladder gives a signed-out download.
_AUTHED_DEFAULT = ("web_safari", "web")
def client_sets(self) -> list:
return [
c for c in (settings.player_client_list, settings.player_client_fallback_list) if c
] or [None]
def client_sets(self, authed: bool = False) -> list:
sets = [c for c in (settings.player_client_list, settings.player_client_fallback_list) if c]
if authed:
kept = [[c for c in s if c in self._COOKIE_CAPABLE] for s in sets]
return [s for s in kept if s] or [list(self._AUTHED_DEFAULT)]
return sets or [None]
def ydl_opts_extra(self, clients) -> dict:
# == --extractor-args "youtube:player_client=…;youtubepot-bgutilhttp:base_url=…" (list values).
@@ -93,6 +155,63 @@ class YouTubeAdapter(SourceAdapter):
class FacebookAdapter(SourceAdapter):
key = "facebook"
# Measured (E11.5, yt-dlp 2026.07.04): with the user's cookie, some reels answer a plain request
# with "Cannot parse data" and parse only when the request looks like Chrome. Impersonation is a
# SECOND attempt behind that one error, not the default: the links that already download with the
# cookie keep their proven path, and only the ones stuck at the wall go through curl_cffi. Without
# a cookie the same error is the login wall, which no disguise gets past.
_IMPERSONATE = "impersonate"
# A post page is ~4–5 MB of inline JSON; anything far past that is not a post page.
_POST_PAGE_MAX_BYTES = 16 * 1024 * 1024
def resolve_url(self, url: str, ydl_opts: dict) -> str:
# Measured (E11.5 S4): given a group post's link, yt-dlp either finds no formats or — worse —
# downloads ANOTHER video from the page's feed data. The post's own video is the one Video node
# in the page scripts that carry the post id; its watch link downloads (plain first, then the
# Chrome retry above). Needs the cookie: signed out, the page is the login wall.
post_id = facebook_group_post_id(url)
if post_id is None or not ydl_opts.get("cookiefile"):
return url
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
page = raw.decode("utf-8", "replace")
# Measured (E11.5 review): a refused or expired cookie gets the login page (~450 KB, a login
# form, no post id at all), which is not a post without a video — the user's cookie is the cause.
if _quoted(post_id) not in page:
raise RuntimeError(POST_NOT_SHOWN_MESSAGE)
video_id = facebook_post_video_id(page, post_id)
if video_id is None:
# Never fall back to the post link: that is exactly where the wrong video comes from.
raise RuntimeError(POST_WITHOUT_VIDEO_MESSAGE)
return f"https://www.facebook.com/watch/?v={video_id}"
def client_sets(self, authed: bool = False) -> list:
return [None, self._IMPERSONATE] if authed else [None]
def ydl_opts_extra(self, clients) -> dict:
if clients == self._IMPERSONATE:
return {"impersonate": ImpersonateTarget.from_str("chrome")}
return {}
def retriable(self, error_text: str) -> bool:
return FACEBOOK_PARSE_WALL in error_text.lower()
# Measured (2026-09-14): a SIGNED-IN read gives the placeholder "Video" (plain and as Chrome), and an
# 18+ reel only readable as Chrome gives "Facebook"; the post's text is still in `description`. A
# signed-out read titles the page with that same text behind the engagement counts, cut short.
_PLACEHOLDER_TITLES = frozenset({"video", "facebook"})
def title(self, info: dict) -> str | None:
raw = info.get("title")
if (raw or "").strip().lower() not in self._PLACEHOLDER_TITLES:
return raw
line = next(
(ln.strip() for ln in (info.get("description") or "").splitlines() if ln.strip()), ""
)
# The whole first line, never cut here with an ellipsis: normalize_title's NFKC turns "…" into
# "..." and then collapses it to ".", which read as a finished sentence ("… live at."). The
# stored title is unbounded; display_name and the file name clip it themselves.
return line or raw # no description either (the 18+ reel): nothing better on the page
def uploader_url(self, info: dict) -> str | None:
# Facebook gives neither channel_url nor uploader_url, but exposes a numeric `uploader_id`
@@ -107,6 +226,39 @@ class FacebookAdapter(SourceAdapter):
return True
_FB_GROUP_POST_PATH = re.compile(r"^/groups/[^/]+/(?:permalink|posts)/(\d+)/?$")
_SCRIPT_BLOCK = re.compile(r"<script[^>]*>(.*?)</script>", re.S)
_VIDEO_NODE = re.compile(r'"__typename":"Video","id":"(\d+)"')
def _quoted(post_id: str) -> str:
# How the page JSON carries the id: quoted, so a longer id that merely starts with it never counts.
return f'"{post_id}"'
def facebook_group_post_id(url: str) -> str | None:
"""The post id of a Facebook group post link (`/groups/<group>/permalink|posts/<id>/`), else None."""
parsed = urlparse(url)
if not _host_is(parsed.netloc.lower(), "facebook.com"):
return None
match = _FB_GROUP_POST_PATH.match(parsed.path)
return match.group(1) if match else None
def facebook_post_video_id(page: str, post_id: str) -> str | None:
"""The id of the one video attached to `post_id`, read from its page: the Video nodes of the script
blocks that carry the post id (the page's other ~10 videos — feed, reels — sit in other blocks).
None for no video AND for more than one: guessing between two would be the wrong-video bug again."""
quoted = _quoted(post_id)
ids = {
vid
for block in _SCRIPT_BLOCK.findall(page)
if quoted in block
for vid in _VIDEO_NODE.findall(block)
}
return ids.pop() if len(ids) == 1 else None
class RedditAdapter(SourceAdapter):
key = "reddit"
+65 -19
View File
@@ -1,15 +1,17 @@
"""Per-user, per-host download cookies — the single source of truth for "does the acting user's
cookie apply to this download, and how does that keep the result private".
Two call sites depend on ONE consistent answer (house rule: an invariant lives in shared code, not
Two sides depend on ONE consistent answer (house rule: an invariant lives in shared code, not
in prose):
* `enqueue` salts the MediaAsset cache signature with the user id WHEN a cookie applies, so the
authed download becomes a PRIVATE asset that is never cache-served to another user.
authed download becomes a PRIVATE asset that is never cache-served to another user. Retry
(`service.resume`) and Re-download (`service.redownload`) ask the same question again, so a
cookie the user added after a failure goes with the next attempt.
* the worker attaches the cookie file to yt-dlp ONLY when a cookie still applies AND the asset it
is downloading into is that private, salted identity — so a cookie can never poison a shared
asset (the load-bearing cross-user privacy fix).
Both go through `cookie_salt` / `authed_format_sig` / `host_key_for_url` here, so they can't drift.
Both key the private identity on `cookie_salt` / `host_key_for_url` here, so they can't drift.
Security controls enforced here:
* **host scope (SEC-12)** — `host_key_for_url` matches the download URL's netloc against a curated
@@ -30,7 +32,7 @@ from sqlalchemy.orm import Session
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential
from app.models import DownloadCredential, MediaAsset
from app.security import decrypt_optional, encrypt
log = logging.getLogger("siftlode.downloads.credentials")
@@ -53,6 +55,11 @@ class HostSpec:
# all derive from this, so there is nothing else to wire up.
SUPPORTED_HOSTS: tuple[HostSpec, ...] = (
HostSpec("facebook", "Facebook", ("facebook.com", "fb.watch")),
# YouTube gates INDIVIDUAL videos behind "Sign in to confirm you're not a bot" (measured on prod
# job 273: every player client fails on that one video while its neighbours download fine, and
# the newest yt-dlp fails the same way — a signed-in session is the only way through). Without a
# row here the `login_required` copy sent the user to a dropdown that could not help them.
HostSpec("youtube", "YouTube", ("youtube.com", "youtu.be", "youtube-nocookie.com")),
HostSpec("instagram", "Instagram", ("instagram.com",)),
HostSpec("twitter", "X (Twitter)", ("twitter.com", "x.com")),
HostSpec("tiktok", "TikTok", ("tiktok.com",)),
@@ -99,11 +106,38 @@ def cookie_salt(user_id: int, host_key: str) -> str:
def authed_format_sig(spec: dict, user_id: int, host_key: str) -> str:
"""The salted `format_sig` an authed download for (user, host) resolves to — the private asset
identity. Both enqueue (to create it) and the worker (to recognise it) compute it HERE."""
"""The salted `format_sig` an authed download for (user, host) resolves to under the current
signature version — the private asset enqueue creates, and one of the identities `is_authed_asset`
recognises. Neither calls it; the tests use it to hold the two to each other."""
return formats.format_sig(spec, salt=cookie_salt(user_id, host_key))
def rebuild_salt(asset_id: int, cookie: str | None) -> str:
"""The salt of a Re-download's fresh asset: its OWN row id (so every rebuild is a new identity no
enqueue can ever compute), prefixed with the user's `cookie_salt` when a cookie applies. The id is
what makes it recognisable — the worker has the asset row, so `is_authed_asset` can recompute this
one, where a salt built from the job's previous asset id could not be."""
return f"{cookie}:rebuild:{asset_id}" if cookie else f"rebuild:{asset_id}"
def is_authed_asset(asset: MediaAsset, spec: dict, user_id: int, url: str | None) -> bool:
"""Whether an asset is `user_id`'s private, cookie-salted identity for `url`'s host — the one
kind of asset a cookie is ever attached to: the one enqueue creates (`authed_format_sig`), or a
Re-download's rebuild of it (`rebuild_salt`). The worker's cookie gate asks it before a download,
and the failure diagnosis asks it after one (a failure on this asset happened WITH the cookie).
Either signature version counts (`formats.format_sigs`): a job queued before a deploy that bumped
the version still downloads into the asset it was given."""
host_key = host_key_for_url(url)
if host_key is None:
return False
salt = cookie_salt(user_id, host_key)
if asset.format_sig in formats.format_sigs(spec, salt=salt):
return True
return asset.id is not None and asset.format_sig in formats.format_sigs(
spec, salt=rebuild_salt(asset.id, salt)
)
def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
"""At enqueue: the cache salt to apply if this user has a cookie for `url`'s host, else None (a
normal shared download). Returns the salt string so the caller passes it straight to
@@ -199,20 +233,19 @@ def status_list(db: Session, user_id: int) -> list[dict]:
# --- worker: resolve + write the cookie file --------------------------------------------------
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset_format_sig: str,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download, or None.
def usable_cookie(
db: Session, user_id: int, url: str | None, asset: MediaAsset, spec: dict
) -> str | None:
"""The user's cookie text a download of `url` into `asset` goes with, or None — the ONE decision
whether a download is signed in. The worker child writes it out for yt-dlp (`resolve_cookiefile`);
the worker asks it again for a job that inherits a failed asset. That second ask sees the
credential as it is when the job is claimed: a cookie the user removed or added since the failed
download changes this job's explanation to fit the cookie its Retry would go with.
The double gate that prevents cross-user leakage:
1. a credential must still exist for `url`'s host (host-scoped via `host_key_for_url`), AND
2. the asset being downloaded into must BE this user's private, salted identity
(`asset_format_sig == authed_format_sig(...)`).
(`is_authed_asset`).
If (2) fails the asset is the SHARED (unsalted) one — attaching the cookie would poison the
global cache with a private-group video, so we attach nothing. An unreadable blob (rotated key →
@@ -224,18 +257,31 @@ def resolve_cookiefile(
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset_format_sig != authed_format_sig(spec, user_id, host_key):
if not is_authed_asset(asset, spec, user_id, url):
return None # shared asset — never attach a cookie to a non-private identity
raw = decrypt_optional(cred.cookies_enc)
if not raw:
log.warning(
"download cookie for user %s host %s could not be decrypted (rotated key?) — "
"proceeding without it",
"treated as no cookie",
user_id,
host_key,
)
return None
return _write_cookiefile(raw, staging)
return raw
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset: MediaAsset,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download (`usable_cookie`), or None."""
raw = usable_cookie(db, user_id, url, asset, spec)
return _write_cookiefile(raw, staging) if raw else None
def _write_cookiefile(raw: str, staging: Path) -> Path:
+177 -5
View File
@@ -1,4 +1,5 @@
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code.
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code — and, where a site's
failure has been measured, into that site's entry of the failure catalog.
The worker stores the raw exception text (truncated) for technical detail, but a raw
`ERROR: [youtube] <id>: Private video. Sign in…` is noise to a user. `classify()` maps that text
@@ -6,21 +7,51 @@ to one of a small set of category codes; the frontend turns the code into a loca
explanation (and decides whether a retry makes sense). The raw text stays available behind a
"details" disclosure. This is the same string-matching approach the worker already trusts for
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
**The failure catalog** (`_CATALOG`, E11.5). A category is generic: "Sign-in required" cannot tell a
Facebook user that the cookie they already gave was refused, and yt-dlp words several different
Facebook walls identically (`Cannot parse data`). A catalog entry is scoped to ONE host and may also
depend on whether a cookie was attached and on the kind of link, and names a `detail` code the UI has
its own sentence for — why it failed on that site, and what the user can do. `diagnose()` tries the
catalog first and falls back to the category. How an entry is added, every time a failure is reported:
1. reproduce it on dev (with and without the user's cookie) and keep the verbatim yt-dlp text;
2. add the entry below, scoped as narrowly as the measurement allows (a marker that is yt-dlp's
generic wording for every site must never match without its host; a marker another module also
decides on — the Facebook Chrome retry's FACEBOOK_PARSE_WALL — is imported, never retyped);
3. pin the real text in `tests/test_download_errors.py`;
4. add `downloads.errors.fail.catalog.<detail>.{short,body}` in EN and HU (a contract test checks it;
the group is not called `detail`, which `actionErrors.contract` reads as a raw `HttpError.detail`).
"""
# Two worker-authored strings that this module owns because both are coupled to `_RULES`.
from collections.abc import Callable
from dataclasses import dataclass
from typing import NamedTuple
from app.downloads.adapters import (
FACEBOOK_PARSE_WALL,
POST_NOT_SHOWN_MESSAGE,
POST_WITHOUT_VIDEO_MESSAGE,
facebook_group_post_id,
)
from app.downloads.credentials import host_key_for_url
# Worker-authored strings that this module owns because each is coupled to `_RULES`.
#
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
# rule drifting apart is exactly how a category silently becomes unreachable.
#
# NETWORK_REDACTED_MESSAGE is the opposite direction: it REPLACES a message after classification, so
# it deliberately has no rule. Nothing re-classifies a stored `job.error` today; if anything ever
# does, it must carry the original code rather than re-deriving one from this text.
# NETWORK_REDACTED_MESSAGE REPLACES a message after classification (`user_error_text`). It is written
# to the shared asset too, and a second job that inherits that failed asset re-diagnoses the stored
# text (`worker._process_job`), so it has a rule of its own that gives the category back.
#
# Other worker-raised texts are still inline literals at their raise sites. The ffmpeg ones and
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
# classifies as `unknown` — it is a refusal, not a diagnosis, and no category would say more.
#
# A source adapter's verdicts on a link (`SourceAdapter.resolve_url`, E11.5 S4) are adapters.py's
# constants, imported above; each has a rule below that gives its category back, and a catalog entry
# that gives the site's sentence.
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
@@ -30,6 +61,12 @@ _RULES: tuple[tuple[str, str], ...] = (
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
# first — it is our wording, so it can never be a false match on an upstream message.
("download storage is full", "quota_exceeded"),
# Our own redaction of a transport failure (see NETWORK_REDACTED_MESSAGE) — our wording too.
(NETWORK_REDACTED_MESSAGE.lower(), "network"),
# Our own verdicts on a post link (adapters.py) — our wording too: its page holds no single video,
# or the site answered the signed-in read with something that is not the post at all.
(POST_WITHOUT_VIDEO_MESSAGE.lower(), "format_unavailable"),
(POST_NOT_SHOWN_MESSAGE.lower(), "login_required"),
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
("sign in to confirm your age", "login_required"),
("confirm your age", "login_required"),
@@ -101,6 +138,120 @@ _RULES: tuple[tuple[str, str], ...] = (
)
class Diagnosis(NamedTuple):
"""What a failure was: the generic `code` (category) and, when a catalog entry matched, its
host-specific `detail`."""
code: str
detail: str | None = None
@dataclass(frozen=True)
class _Entry:
"""One measured failure of one site. It matches when the download URL belongs to `host`
(`credentials.SUPPORTED_HOSTS`), every one of `markers` is in the message, the cookie state is
`cookie` (None: either), and — when `link` is given — it recognises the URL (a truthy result). A
link predicate is the adapter's own, so the catalog and the download agree on what a link is."""
host: str
markers: tuple[str, ...]
code: str
detail: str
cookie: bool | None = None
link: Callable[[str], object] | None = None
_CATALOG: tuple[_Entry, ...] = (
# Facebook answers a signed-out client with a page yt-dlp cannot parse — an 18+ reel, a closed
# group's post (measured 2026-09-13: prod jobs 145, 156, 256 download once the owner's cookie is
# attached). "Cannot parse data" is yt-dlp's generic wording for a broken extractor on EVERY site,
# which is why the entry is scoped to Facebook and never becomes a general rule.
_Entry(
host="facebook",
markers=("[facebook]", FACEBOOK_PARSE_WALL),
code="login_required",
detail="facebook.login_wall",
cookie=False,
),
# The same text WITH a cookie: an expired / signed-out cookie, or a page that also needs
# impersonation (prod job 257). Telling this user to add cookies would be wrong — they have.
_Entry(
host="facebook",
markers=("[facebook]", FACEBOOK_PARSE_WALL),
code="login_required",
detail="facebook.cookie_rejected",
cookie=True,
),
# A group post read with a cookie that Facebook answered with its login page (measured with a
# refused cookie: no post id anywhere on the page) — the same refused cookie, found by the adapter.
_Entry(
host="facebook",
markers=(POST_NOT_SHOWN_MESSAGE.lower(),),
code="login_required",
detail="facebook.cookie_rejected",
cookie=True,
),
# A group post with no single video of its own. With a cookie the adapter reads the post's page
# and says so itself (POST_WITHOUT_VIDEO_MESSAGE); without one yt-dlp is given the post link and
# answers "No video formats found" (measured on prod jobs 66 and 145, whose videos are reshared
# clips). With a cookie yt-dlp only ever gets the post's VIDEO link, so the same text there is about
# that video, not the post — the category says it. Only for group post links (the adapter's own
# test of one): a reel with no formats is a different failure.
_Entry(
host="facebook",
markers=(POST_WITHOUT_VIDEO_MESSAGE.lower(),),
code="format_unavailable",
detail="facebook.group_post_video",
link=facebook_group_post_id,
),
_Entry(
host="facebook",
markers=("[facebook]", "no video formats found"),
code="format_unavailable",
detail="facebook.group_post_video",
cookie=False,
link=facebook_group_post_id,
),
# YouTube gates INDIVIDUAL videos behind "Sign in to confirm you're not a bot" (measured on prod
# job 273, a festival set: all 12 of yt-dlp's player clients fail on that one video while its
# neighbours download fine, the POT sidecar is live and answering, and the newest yt-dlp fails
# identically — so this is the video's own signed-out gate, not a broken stack or a stale yt-dlp).
# The marker is quote-agnostic on purpose: YouTube writes the apostrophe as U+2019, so
# "you're not a bot" with a straight quote never matches the real text.
_Entry(
host="youtube",
markers=("[youtube]", "not a bot"),
code="login_required",
detail="youtube.bot_wall",
cookie=False,
),
# The same wall WITH a cookie. Not yet reproduced with a real YouTube cookie (this user's first
# one is being added now) — but "add cookies" is provably the wrong advice for someone who has,
# and yt-dlp names the likeliest cause itself: "The provided YouTube account cookies are no
# longer valid. They have likely been rotated in the browser as a security measure."
_Entry(
host="youtube",
markers=("[youtube]", "not a bot"),
code="login_required",
detail="youtube.cookie_rejected",
cookie=True,
),
# The same gate one step later: the player response comes back, and then the CDN refuses the
# stream itself ("unable to download video data: HTTP Error 403"). Measured 2026-09-16 on the
# very same video, ~90 minutes after it was answering with the bot message — signed out it
# 403'd on every attempt; with the owner's cookie attached it downloaded in full (519 MB). So
# the category this text carries generically, `network`, is a lie here: nothing was wrong with
# the network. Scoped to a cookie-less YouTube download, where that is what we measured.
_Entry(
host="youtube",
markers=("unable to download video data", "403"),
code="login_required",
detail="youtube.media_forbidden",
cookie=False,
),
)
def classify(msg: str | None) -> str:
"""Return a category code for a raw failure message (``"unknown"`` if nothing matches)."""
if not msg:
@@ -112,6 +263,27 @@ def classify(msg: str | None) -> str:
return "unknown"
def diagnose(
msg: str | None, *, url: str | None = None, cookie_attached: bool = False
) -> Diagnosis:
"""The catalog entry for this failure of this link, else just its category (see the module doc).
`url` is the address yt-dlp was given; `cookie_attached` whether the acting user's cookie went
with it. Without a URL only the category can be known."""
host = host_key_for_url(url) if msg and url else None
if host is not None:
low = msg.lower() # type: ignore[union-attr] — `host` is only set when `msg` is truthy
for e in _CATALOG:
if (
e.host == host
and all(m in low for m in e.markers)
and (e.cookie is None or e.cookie == cookie_attached)
and (e.link is None or e.link(url)) # type: ignore[arg-type] — as `msg` above
):
return Diagnosis(e.code, e.detail)
return Diagnosis(classify(msg))
def user_error_text(source_kind: str, msg: str, code: str) -> str:
"""The failure text to STORE on the job, given the raw message and its category.
+21 -3
View File
@@ -57,6 +57,10 @@ _DEFAULTS = {
# opt-out), so any asset cached under an old non-compat spec — or a stale non-H.264 mp4 that slipped
# the v3 net — must re-derive through the current browser-safe guarantee.
_SIG_VERSION = 4
# Audio-only downloads version on their own, so a change to the audio bytes doesn't re-download every
# cached video. v5: an extracted .mp3/.m4a is stored as itself instead of remuxed into an .mp4 that
# iPad's WebKit refuses to play, so a cached remuxed audio asset must re-derive.
_AUDIO_SIG_VERSION = 5
_SPONSORBLOCK_CATEGORIES = ["sponsor", "selfpromo", "interaction"]
@@ -96,20 +100,34 @@ def normalize(spec: dict) -> dict:
return s
def format_sig(spec: dict, salt: str | None = None) -> str:
def _sig_version(s: dict) -> int:
return _AUDIO_SIG_VERSION if s["mode"] == "a" else _SIG_VERSION
def format_sig(spec: dict, salt: str | None = None, *, version: int | None = None) -> str:
"""Stable short hash of the output-affecting spec — the MediaAsset cache key component.
`salt` forks a distinct, non-dedup identity for the same spec (used by force-redownload so it
always re-fetches into its own asset instead of sharing a cached file with other jobs)."""
always re-fetches into its own asset instead of sharing a cached file with other jobs).
`version` hashes under a signature version other than the current one (see `format_sigs`)."""
s = normalize(spec)
data = {k: s[k] for k in _SIG_KEYS}
data["_v"] = _SIG_VERSION
data["_v"] = _sig_version(s) if version is None else version
if salt is not None:
data["_salt"] = salt
payload = json.dumps(data, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(payload.encode()).hexdigest()[:24]
def format_sigs(spec: dict, salt: str | None = None) -> tuple[str, str]:
"""`format_sig` under the current signature version and under the one before it. An asset keeps
the signature it was created with, so a job queued before a deploy that bumped the version and
run after it downloads into an asset carrying the previous one — which must still be recognised
as that user's private asset (`credentials.is_authed_asset`), or it downloads without its cookie."""
previous = _sig_version(normalize(spec)) - 1
return format_sig(spec, salt), format_sig(spec, salt, version=previous)
def build_ydl_opts(
spec: dict,
outtmpl: str,
+55 -24
View File
@@ -8,7 +8,6 @@ from datetime import date, datetime, timedelta, timezone
from pathlib import Path
import yt_dlp
from sqlalchemy import select
from app import sysconfig
from app.config import settings
@@ -114,9 +113,10 @@ def _fill_asset_meta_early(asset_id: int, info: dict) -> None:
# `or info.get("id")` avoids a blank card while downloading when the cleaned title is empty
# (e.g. a title that was only a social engagement prefix); the completion path later sets
# the final title with its own source_ref fallback.
asset.title = normalize_title(info.get("title")) or info.get("id")
adapter = adapters.resolve_adapter(None, None, info)
asset.title = normalize_title(adapter.title(info)) or info.get("id")
asset.uploader = info.get("uploader") or info.get("channel")
asset.uploader_url = adapters.resolve_adapter(None, None, info).uploader_url(info)
asset.uploader_url = adapter.uploader_url(info)
asset.thumbnail_url = info.get("thumbnail")
asset.duration_s = int(info["duration"]) if info.get("duration") else None
asset.upload_date = _parse_upload_date(info.get("upload_date"))
@@ -323,14 +323,33 @@ def _extract_with_fallback(
):
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
error); other sources make a single attempt. Returns the info dict of the successful attempt.
error); a Facebook download WITH a cookie is retried as Chrome behind "Cannot parse data"; other
sources make a single attempt. Returns the info dict of the successful attempt.
Before the first attempt the adapter may resolve `url` to the link that really holds the video (a
Facebook group post → its video's watch link); when the page shows no single video there, or is not
the post at all (a refused cookie's login page), the adapter raises its own verdict
(adapters.POST_*_MESSAGE) instead of letting yt-dlp guess, and a failed page read fails the
download as itself.
`cookiefile` (S6) is the acting user's per-host cookies.txt, already gated by
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated.
`max_bytes` (SEC-13) is the owner's remaining storage budget, applied to every attempt."""
client_sets = adapter.client_sets()
client_sets = adapter.client_sets(authed=cookiefile is not None)
outtmpl = str(staging / "%(id)s.%(ext)s")
# Resolving may read a page first — network time the stall watchdog must not see as a stall.
_touch_job(job_id)
target = adapter.resolve_url(
url,
formats.build_ydl_opts(
spec,
outtmpl,
_make_progress_hook(job_id, asset_id),
cookiefile=cookiefile,
max_bytes=max_bytes,
),
)
last_exc: Exception | None = None
for i, clients in enumerate(client_sets):
opts = formats.build_ydl_opts(
@@ -348,7 +367,7 @@ def _extract_with_fallback(
# killed — acceptable, as socket_timeout + extractor_retries bound one attempt well under it.)
try:
with yt_dlp.YoutubeDL(opts) as ydl:
return ydl.extract_info(url, download=True)
return ydl.extract_info(target, download=True)
except yt_dlp.utils.DownloadError as exc:
last_exc = exc
if i < len(client_sets) - 1 and adapter.retriable(str(exc)):
@@ -385,8 +404,12 @@ def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict
or just remux when the codecs are fine but the container isn't. **Scope:** only the mp4 compat
profile is normalized; an explicit `vcodec` or a non-mp4 custom profile is the user opting out
of the guarantee and is left as-is (so a `webm` profile can still ship VP9/Opus). Returns the
file to store (converted, or the original) and corrects `info`'s codec fields to match disk."""
if spec.get("container") != "mp4" or spec.get("vcodec"):
file to store (converted, or the original) and corrects `info`'s codec fields to match disk.
An audio download (mode "a") is never touched: yt-dlp already extracted it to .mp3 / .m4a, which
every browser plays as audio/mpeg / audio/mp4. Remuxing it into .mp4 served an mp3 as video/mp4,
which iPad's WebKit refuses to play (prod job 268)."""
if spec.get("mode") == "a" or spec.get("container") != "mp4" or spec.get("vcodec"):
return produced
vcodec, acodec = editmod.probe_codecs(produced)
is_mp4 = produced.suffix.lower() == ".mp4"
@@ -425,6 +448,7 @@ def _download(
) -> None:
staging = _staging_dir(asset_id)
cookiefile: Path | None = None
url: str | None = None # read by the failure branch, which can run before it is resolved
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
@@ -445,12 +469,10 @@ def _download(
# this asset is that user's private, salted identity (the double gate lives in credentials).
remaining_bytes: int | None = None
with SessionLocal() as cdb:
asset_sig = cdb.execute(
select(MediaAsset.format_sig).where(MediaAsset.id == asset_id)
).scalar_one_or_none()
if asset_sig is not None:
target = cdb.get(MediaAsset, asset_id)
if target is not None:
cookiefile = credentials.resolve_cookiefile(
cdb, user_id, url, asset_sig, spec, staging
cdb, user_id, url, target, spec, staging
)
# SEC-13: what this user has left. `check_enqueue` only compared the footprint as it
# stood when the job was QUEUED — a download's size is unknown until it finishes, so a
@@ -484,13 +506,19 @@ def _download(
produced = _produced_file(info, staging)
produced = _ensure_browser_playable(job_id, produced, spec, info)
ext = produced.suffix.lstrip(".").lower()
# Codecs/dims/duration come from a PROBE of the produced file — the ground truth — NOT yt-dlp's
# `info` dict, whose fields are absent for some sources (Facebook among them): deriving
# media_kind from those nulls misclassified a real H.264 video as a download-only 'file', and a
# missing duration hid the trim action. Probed before the sidecars so the .nfo runtime has it
# too. Same probe the upload ingest uses.
vcodec, acodec, width, height, duration_s = probe.resolve_media_facts(produced, info)
meta = storage.MediaMeta(
video_id=info.get("id") or source_ref,
title=normalize_title(info.get("title")) or source_ref,
title=normalize_title(adapter.title(info)) or source_ref,
uploader=info.get("uploader") or info.get("channel") or "Unknown Channel",
upload_date=_parse_upload_date(info.get("upload_date")),
uniqueid_type=adapter.nfo_uniqueid_type(),
duration_s=int(info["duration"]) if info.get("duration") else None,
duration_s=duration_s,
description=info.get("description"),
thumbnail_url=info.get("thumbnail"),
)
@@ -516,12 +544,11 @@ def _download(
asset.rel_path = rel
asset.size_bytes = size
asset.duration_s = meta.duration_s
# Codecs/dims come from a PROBE of the finished file — the ground truth — NOT yt-dlp's
# `info` dict, whose codec fields are absent for some sources (Facebook among them):
# deriving media_kind from those nulls misclassified a real H.264 video as a
# download-only 'file' with no in-app play/edit. Same probe the upload ingest uses.
asset.vcodec, asset.acodec, asset.width, asset.height = probe.resolve_media_facts(
final, info
asset.vcodec, asset.acodec, asset.width, asset.height = (
vcodec,
acodec,
width,
height,
)
asset.container = ext
asset.media_kind = mediakind.derive(asset.vcodec, asset.acodec)
@@ -578,6 +605,7 @@ def _download(
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
error_detail=None,
phase=None,
)
db.commit()
@@ -588,9 +616,11 @@ def _download(
log.info("job %s refused for its owner: %s", job_id, msg)
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
msg = str(exc)[:500]
code = dlerrors.classify(msg)
# The catalog needs the link and whether the user's cookie went with it: the same yt-dlp
# text is Facebook's login wall without one and a refused cookie with one.
diagnosis = dlerrors.diagnose(msg, url=url, cookie_attached=cookiefile is not None)
# SEC-12: redact a transport-level readout for user-supplied URLs (see user_error_text).
msg = dlerrors.user_error_text(source_kind, msg, code)
msg = dlerrors.user_error_text(source_kind, msg, diagnosis.code)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
@@ -603,7 +633,8 @@ def _download(
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=code,
error_code=diagnosis.code,
error_detail=diagnosis.detail,
phase=None,
)
db.commit()
+18 -9
View File
@@ -81,10 +81,12 @@ def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | N
def resolve_media_facts(
path: Path, info: dict
) -> tuple[str | None, str | None, int | None, int | None]:
"""(vcodec, acodec, width, height) for the stored download asset: PROBE the finished file on disk
(the ground truth). yt-dlp's `info` omits codec/dimension fields for some sources (Facebook among
them), so deriving media_kind from it there labelled a real H.264 video a download-only 'file'.
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) for the stored download asset: PROBE the finished
file on disk (the ground truth). yt-dlp's `info` omits codec/dimension fields for some sources
(Facebook among them), so deriving media_kind from it there labelled a real H.264 video a
download-only 'file'. It omits the duration too (a Facebook reel, an Instagram reel), and a NULL
duration hides the Library row's trim action although the file plays for minutes.
Precedence is by whether the probe RAN, not per-field: if ffprobe read the file, its codec verdict
is authoritative — INCLUDING a deliberate None (cover-art-only audio, a still image). A per-field
@@ -92,16 +94,23 @@ def resolve_media_facts(
have (yt-dlp reports a progressive `vcodec` for an audio-mode download whose video it then strips),
mislabelling an audio download as a broken in-app 'video'. Only when ffprobe could NOT read the
file at all do we fall back to `info` (the pre-probe behaviour, a safety net); the cosmetic
dimensions still gap-fill from `info` in either case."""
dimensions and the duration still gap-fill from `info` in either case."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
info_v = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
info_a = info.get("acodec") if info.get("acodec") not in (None, "none") else None
info_w = int(info["width"]) if info.get("width") else None
info_h = int(info["height"]) if info.get("height") else None
info_d = int(info["duration"]) if info.get("duration") else None
if data is None:
# ffprobe could not read the file — fall back to the extractor's info entirely.
return info_v, info_a, info_w, info_h
pv, pa, pw, ph, _dur = _facts_from_probe(data)
return info_v, info_a, info_w, info_h, info_d
pv, pa, pw, ph, pd = _facts_from_probe(data)
# Probe ran → its codecs win outright (a None means "no such track", not "unknown"); info only
# gap-fills the cosmetic width/height the probe couldn't read.
return pv, pa, pw if pw is not None else info_w, ph if ph is not None else info_h
# gap-fills the width/height/duration the probe couldn't read.
return (
pv,
pa,
pw if pw is not None else info_w,
ph if ph is not None else info_h,
pd if pd is not None else info_d,
)
+7 -1
View File
@@ -17,6 +17,8 @@ import shutil
import subprocess
import sys
from app import logs
from app import logsink
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
@@ -52,12 +54,16 @@ def _deprioritize() -> None:
def main() -> None:
logging.basicConfig(level=logging.INFO)
logs.configure_logging("runner")
_deprioritize()
if len(sys.argv) != 2:
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
raise SystemExit(2)
job_id = int(sys.argv[1])
# This whole process exists to run ONE job — tag every line it prints with it, so the child's
# output joins up with the worker's and the API's view of the same download.
logs.set_job(job_id)
logsink.start() # flushed at exit (atexit) — this process has no shutdown hook of its own
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
+64 -8
View File
@@ -7,6 +7,7 @@ handles everything else. State transitions used by the routes (pause/resume/canc
the worker live in `worker.py` / the routes; this module owns enqueue + asset bookkeeping.
"""
import uuid
from datetime import datetime, timezone
from sqlalchemy import func, select, update
@@ -80,11 +81,13 @@ def get_or_create_asset(
status="pending",
ref_count=0,
)
db.add(asset)
try:
db.flush()
# A SAVEPOINT: losing the race undoes only this insert, never the caller's own pending writes
# (Retry clears the job's error before it gets here; a whole-session rollback threw that away).
with db.begin_nested():
db.add(asset)
db.flush()
except IntegrityError:
db.rollback()
asset = db.execute(stmt).scalar_one()
return asset
@@ -341,13 +344,20 @@ def pause(db: Session, job: DownloadJob) -> None:
def resume(db: Session, job: DownloadJob) -> None:
"""Requeue a paused/errored job for a fresh attempt, clearing its error. If the SHARED asset itself
failed, reset it to pending so the worker actually re-downloads — otherwise the requeued job would
instantly re-inherit the asset's stale error (the worker short-circuits an errored asset)."""
instantly re-inherit the asset's stale error (the worker short-circuits an errored asset).
A cookie the user added AFTER the job was queued moves the job onto their private asset first
(`_follow_the_cookie`): the worker only ever attaches a cookie to that identity, so without the move
the Retry a login-wall message asks for would fail on the same wall."""
if job.status in (JobStatus.PAUSED, JobStatus.ERROR):
job.progress = 0
job.error = None
job.error_code = None
job.error_detail = None
if job.asset_id:
asset = db.get(MediaAsset, job.asset_id)
if asset is not None:
asset = _follow_the_cookie(db, job, asset)
if asset and asset.status == "error": # MediaAsset.status (no enum), not a JobStatus
asset.status = "pending"
asset.error = None
@@ -355,6 +365,32 @@ def resume(db: Session, job: DownloadJob) -> None:
db.commit()
def _follow_the_cookie(db: Session, job: DownloadJob, asset: MediaAsset) -> MediaAsset:
"""Re-point a requeued job at the asset enqueue would pick NOW, when the user has a cookie for the
link's host but the job still holds an asset the worker won't attach it to (queued before the
cookie existed). Returns the asset the job holds afterwards.
Only a `pending` / `error` asset moves — neither has a file nor a download in flight. A `ready` one
already has the file, and a `downloading` one may be this job's own child still being stopped by
the pause; the next Retry after it settles moves the job then. Removing a cookie never moves a job
back: the worker attaches nothing without a credential, so the private asset is as good as any."""
if asset.status not in ("pending", "error"):
return asset
url = source_url(job.source_kind, job.source_ref)
salt = credentials.salt_for_enqueue(db, job.user_id, url)
spec = formats.normalize(job.profile_snapshot)
if salt is None or credentials.is_authed_asset(asset, spec, job.user_id, url):
return asset
target = get_or_create_asset(
db, job.source_kind, job.source_ref, formats.format_sig(spec, salt=salt)
)
populate_from_catalog(db, target)
release_asset(db, job) # drop the old hold BEFORE re-pointing — it keys off job.asset_id
incr_ref_count(db, target.id)
job.asset_id = target.id
return target
def cancel(db: Session, job: DownloadJob) -> None:
"""Cancel a job unless it's already done/canceled, releasing its asset hold WHILE it still counts
as holding — `canceled` ∉ HOLDING, so a later delete won't double-release."""
@@ -373,13 +409,31 @@ def delete(db: Session, job: DownloadJob) -> None:
def redownload(db: Session, job: DownloadJob) -> None:
"""Rebuild a finished/failed job's file into a FRESH, private asset (the cache sig is salted with
the job id + current asset id, so it never reuses the dedup pool) and requeue the SAME job row —
the new asset's own id, so it never reuses the dedup pool) and requeue the SAME job row —
so any public share link (DownloadLink -> job_id) keeps resolving to the newly-downloaded file.
The old hold is dropped first (freeing the old file iff this job was its last holder). The caller
has already rejected edit clips + still-in-progress jobs (HTTP 409)."""
has already rejected edit clips + still-in-progress jobs (HTTP 409).
When the user has a cookie for the link's host, the salt carries it (`credentials.rebuild_salt`),
so the worker recognises the rebuild as their private identity and attaches the cookie — a cookie
download, or one the user added after a failure, rebuilds with the cookie."""
spec = formats.normalize(job.profile_snapshot)
sig = formats.format_sig(spec, salt=f"{job.id}:{job.asset_id}")
asset = get_or_create_asset(db, job.source_kind, job.source_ref, sig)
cookie = credentials.salt_for_enqueue(
db, job.user_id, source_url(job.source_kind, job.source_ref)
)
# The salt names the row's own id, which exists only once the row does: insert it under a unique
# placeholder, then write the real signature in the same transaction (nobody sees the placeholder).
asset = MediaAsset(
source_kind=job.source_kind,
source_ref=job.source_ref,
format_sig=f"rebuild:{uuid.uuid4().hex}",
status="pending",
ref_count=0,
)
db.add(asset)
db.flush()
asset.format_sig = formats.format_sig(spec, salt=credentials.rebuild_salt(asset.id, cookie))
db.flush()
populate_from_catalog(db, asset)
# Drop the old hold BEFORE re-pointing the job — release_asset keys off the job's current asset.
release_asset(db, job)
@@ -389,6 +443,8 @@ def redownload(db: Session, job: DownloadJob) -> None:
job.progress = 0
job.phase = None
job.error = None
job.error_code = None # a rebuild that fails again gets its own diagnosis, never the old one
job.error_detail = None
job.speed_bps = None
job.eta_s = None
# Re-derive the display name from the freshly cleaned title on completion (else an older, un-cleaned
+685
View File
@@ -0,0 +1,685 @@
"""Structured logging for every Siftlode process — the ONE place logging is configured.
Before this module a single deployment printed FOUR different log formats: uvicorn's
`log_config.json` for the API, a hand-rolled StreamHandler in `main.py` for everything that did not
start under uvicorn (tests, scripts), `logging.basicConfig()` in the worker and in the download
child (no timestamp at all), and alembic's `fileConfig`. None of them carried a correlation id, and
uvicorn's access log printed the query string of every request (`?account=1`, `?source=feed`) — so
the logs were both unjoinable and leaky. `configure_logging()` replaces all four.
Event shape — the OpenTelemetry log data model, without the SDK; one JSON object per line:
ts · level · sev · source · logger · event · msg · attrs
plus whatever correlation the context carries (`request_id`, `user_id`, `debug_session_id`,
`job_id`) and the app version. That shape is what E22.6's admin console filters on and what a Gitea
issue attachment will carry, so it is deliberately flat and stable.
Call sites stay ordinary stdlib logging:
log = logging.getLogger("siftlode.downloads")
log.info("job finished", extra=logs.event("download.job.done", job_id=7, bytes=n))
`event()` is optional — a bare `log.info("…")` still comes out well-formed — but the dotted code is
what makes a line queryable later, so add one wherever the line matters.
REDACTION IS APPLIED AT THE HANDLER, not at the call site: every message, every attr value and
every traceback passes `redact()` on the way out. That is the whole point — a secret cannot be
leaked by forgetting to redact it at one of ~125 call sites, and a new call site is safe by
default. Never logged, at any level: passwords, tokens (session, reset/verify, setup, share,
OAuth), cookie blobs, secret config values, and email addresses.
"""
import contextlib
import contextvars
import json
import logging
import logging.config
import re
import sys
import threading
import time
import uuid
from datetime import datetime, timezone
from typing import Any
from urllib.parse import parse_qsl
from app.config import settings
from app.ratelimit import RateLimiter, client_key
# One level below DEBUG for the "Teljes / TRACE" diagnostics level (search strings, redacted request
# bodies). Registered on import so `logging.getLogger(...).log(logs.TRACE, …)` and a `LOG_LEVEL=TRACE`
# both work in every process, including one that never calls configure_logging (a test).
TRACE = 5
logging.addLevelName(TRACE, "TRACE")
# OpenTelemetry severity numbers for our level names. Kept beside the level because the number is
# what a log backend sorts and filters on; the name is for humans reading a terminal.
SEVERITY: dict[str, int] = {
"TRACE": 1,
"DEBUG": 5,
"INFO": 9,
"WARNING": 13,
"ERROR": 17,
"CRITICAL": 21,
}
# Which process this is. Set by configure_logging; every line carries it so one Loki/DB query can
# separate the API's view of a job from the worker's.
_source = "api"
# --- correlation context -------------------------------------------------------------------
# contextvars, like quota's actor attribution (app/quota.py): set at an entry point, read by the
# formatter, never threaded through call signatures. A child task copies the context at creation,
# so anything set before `await call_next(...)` is visible to the whole request.
_request_id: contextvars.ContextVar[str | None] = contextvars.ContextVar(
"log_request_id", default=None
)
_user_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_user_id", default=None)
_debug_session: contextvars.ContextVar[str | None] = contextvars.ContextVar(
"log_debug_session", default=None
)
_job_id: contextvars.ContextVar[int | None] = contextvars.ContextVar("log_job_id", default=None)
_CONTEXT_VARS = {
"request_id": _request_id,
"user_id": _user_id,
"debug_session_id": _debug_session,
"job_id": _job_id,
}
class _RequestScope:
"""What the auth dependency learns about a request, for every line logged after it.
A MUTABLE object behind a contextvar, not plain contextvars, because of how a request actually
runs: FastAPI resolves a sync dependency (`auth.current_user`) in a threadpool on a COPY of the
context, so a `ContextVar.set()` there is gone by the time the handler runs — measured: that is
how the account never reached a handler's own lines. `log_requests` installs one scope per
request; every copied context shares the same object, so what the dependency writes is seen by
the handler and by the request line itself."""
__slots__ = ("user_id", "level")
def __init__(self) -> None:
self.user_id: int | None = None
# The user's consented diagnostics level as a logging number; None at the base level.
self.level: int | None = None
_scope: contextvars.ContextVar[_RequestScope | None] = contextvars.ContextVar(
"log_request_scope", default=None
)
def set_user(user_id: int | None, level: str = "INFO", until: datetime | None = None) -> None:
"""Attribute the rest of this request's log lines to `user_id`, and log them at the user's
consented diagnostics `level` until `until` (E22 S5; "INFO" = no elevation). Called from the
auth dependency — the one place the session is actually resolved into an account — rather than
from the request middleware, which runs OUTSIDE the session middleware."""
_user_id.set(user_id)
scope = _scope.get()
if scope is None: # outside log_requests (a script, a unit test): this context only
scope = _RequestScope()
_scope.set(scope)
elevated = level != "INFO" and until is not None and until > datetime.now(timezone.utc)
scope.user_id = user_id
scope.level = resolve_level(level) if elevated else None
if user_id is not None:
_track_elevation(user_id, scope.level, until)
def set_job(job_id: int | None) -> None:
"""Attribute this process's log lines to a download job — for the download CHILD, whose whole
life is one job (the worker uses `bind()` per job instead, since it juggles several)."""
_job_id.set(job_id)
@contextlib.contextmanager
def bind(**fields: Any):
"""Bind correlation fields for the duration of a block (a worker job, a scheduler run):
with logs.bind(job_id=job.id):
...
Unknown field names are ignored rather than raising — a caller binding a field this module does
not know about is a typo, not a reason to take down a download."""
tokens = [(_CONTEXT_VARS[k].set(v)) for k, v in fields.items() if k in _CONTEXT_VARS]
keys = [k for k in fields if k in _CONTEXT_VARS]
try:
yield
finally:
for key, token in zip(keys, tokens):
_CONTEXT_VARS[key].reset(token)
def event(code: str, **attrs: Any) -> dict[str, Any]:
"""`extra=` payload for a log call: a stable dotted event code plus structured attributes.
Returns a plain dict so it composes with stdlib logging and costs nothing when the level is off
(the dict is built either way; keep the attrs cheap — no DB reads inside the call)."""
return {"event": code, "attrs": attrs}
# --- redaction -----------------------------------------------------------------------------
# Substring matching, not exact keys: the same secret travels as `password`, `current_password`,
# `new_password`, `smtp_password`, `plex_server_token`, `access_token`, … Matching on the PART is
# what makes a key we have never seen safe by default.
_SECRET_KEY_PARTS = (
"password",
"passphrase",
"token",
"secret",
"cookie",
"authorization",
"auth_header",
"ciphertext",
"private_key",
"api_key",
"apikey",
"credential",
"session_key",
)
# Query/fragment parameters that carry a credential in a URL we log (OAuth `code`/`state`, our own
# `?token=` setup and share links, `#verify=` / `#reset=` fragments).
_SECRET_PARAM_NAMES = (
"access_token",
"refresh_token",
"id_token",
"token",
"code",
"state",
"key",
"secret",
"password",
"pwd",
"sig",
"signature",
"verify",
"reset",
"setup",
)
_SECRET_PARAM_RE = re.compile(
r"\b((?:" + "|".join(_SECRET_PARAM_NAMES) + r")=)([^&\s\"']+)",
re.IGNORECASE,
)
# Deliberately loose: over-matching redacts a non-address that looks like one (harmless), while
# under-matching puts a real address in a log we attach to an issue (not harmless).
_EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
REDACTED = "[redacted]"
# `extra={PERSIST: False}` keeps a line out of the `log_events` table while stdout still gets it —
# how `log_requests` caps anonymous traffic per client (app/logsink.py reads it).
PERSIST = "persist"
REDACTED_EMAIL = "[email]"
def redact_text(text: str) -> str:
"""Scrub a free-text string: credential-bearing URL params and email addresses."""
scrubbed = _SECRET_PARAM_RE.sub(lambda m: f"{m.group(1)}{REDACTED}", text)
return _EMAIL_RE.sub(REDACTED_EMAIL, scrubbed)
def redact_query(query: str) -> str:
"""Scrub a raw query string. DECODED first: the text rules match what was typed, and a browser
sends it percent-encoded — `someone%40example.com` is an address `_EMAIL_RE` cannot see. Each
parameter also goes through the key rules, so `?new_password=` goes as `redact` would drop it."""
pairs = parse_qsl(query, keep_blank_values=True)
return "&".join(f"{key}={_redact_param(key, value)}" for key, value in pairs)
def _redact_param(key: str, value: str) -> str:
"""One decoded `key=value` pair. The parameter-NAME rule is applied to the key here rather than
left to `_SECRET_PARAM_RE` over the rebuilt pair: that pattern was written for the raw,
percent-encoded query, where a value holds no literal space, `&` or quote, and its value class
stops at the first one. On decoded text `?state=a%20b` would keep its tail and `?code=%20abc`
would not match at all — the whole credential straight into the line."""
if key.lower() in _SECRET_PARAM_NAMES:
return REDACTED
return str(redact(value, key=key))
def redact(value: Any, *, key: str = "") -> Any:
"""Recursively scrub a structure destined for the log.
A value whose KEY names a secret is dropped whole (we never log a prefix of a token — a prefix
is still a leak and still useless). Everything else is scrubbed as text. Depth is bounded by
the structures we actually log (attrs dicts), and non-JSON types fall through to `str`."""
lowered = key.lower()
if key and any(part in lowered for part in _SECRET_KEY_PARTS):
return REDACTED
if isinstance(value, dict):
return {k: redact(v, key=str(k)) for k, v in value.items()}
if isinstance(value, (list, tuple)):
return [redact(v, key=key) for v in value]
if isinstance(value, str):
return redact_text(value)
return value
# --- formatters ----------------------------------------------------------------------------
# Attributes every LogRecord carries; anything else a caller passed via `extra=` is a structured
# attribute and is merged into `attrs` (so `extra={"job_id": 7}` works as well as `event(...)`).
_RECORD_STANDARD = frozenset(
vars(logging.LogRecord("", 0, "", 0, "", None, None)).keys()
| {"message", "asctime", "taskName", "event", "attrs", PERSIST, "elevated"}
)
class ContextFilter(logging.Filter):
"""Copies the correlation contextvars onto the record. A Filter (not the formatter) so the
values are captured at emit time, before a QueueHandler can move the record to another
thread — the DB sink in S3 depends on that."""
def filter(self, record: logging.LogRecord) -> bool:
for name, var in _CONTEXT_VARS.items():
value = var.get()
if value is not None and not hasattr(record, name):
setattr(record, name, value)
scope = _scope.get()
if scope is not None and scope.user_id is not None and not hasattr(record, "user_id"):
record.user_id = scope.user_id
return True
class LevelGate(logging.Filter):
"""The configured level, enforced at the HANDLER, with one exception: the lines of a request
whose user raised their diagnostics level (E22 S5).
The root logger's level cannot do this alone: it is one number for the whole process, and the
point is to log ONE user's requests in detail while everyone else's stay at the base level. So
the root is lowered to the lowest level any active elevation needs (`_track_elevation`), and
this gate drops what only an elevation let through unless the line belongs to that elevated
request. It also marks such a line `elevated`, which is what makes the `log_events` sink keep
every line of that request, not just its warnings."""
def filter(self, record: logging.LogRecord) -> bool:
scope = _scope.get()
if scope is not None and scope.level is not None:
record.elevated = True
return record.levelno >= min(scope.level, _base_level)
return record.levelno >= _base_level
def _record_attrs(record: logging.LogRecord) -> dict[str, Any]:
extra = {k: v for k, v in vars(record).items() if k not in _RECORD_STANDARD}
attrs = dict(getattr(record, "attrs", None) or {})
attrs.update({k: v for k, v in extra.items() if k not in _CONTEXT_VARS})
return attrs
def _safe_message(record: logging.LogRecord) -> str:
"""`record.getMessage()` raises if the args don't match the format string. A broken log call
must not take down the caller, so the failure is reported IN the line instead."""
try:
return record.getMessage()
except Exception as exc: # noqa: BLE001 — any formatting error, reported not raised
return f"{record.msg!r} (log formatting failed: {exc})"
def record_fields(record: logging.LogRecord) -> dict[str, Any]:
"""The event as structured fields, REDACTED — the one definition both the stdout line and the
`log_events` row are built from, so the two sinks can never disagree about what is safe to keep.
Absent optional fields are left out; `ts` is an aware datetime."""
fields: dict[str, Any] = {
"ts": datetime.fromtimestamp(record.created, timezone.utc),
"level": record.levelname,
"sev": SEVERITY.get(record.levelname, 9),
"source": _source,
"logger": record.name,
"msg": redact_text(_safe_message(record)),
}
code = getattr(record, "event", None)
if code:
fields["event"] = code
for name in _CONTEXT_VARS:
value = getattr(record, name, None)
if value is not None:
fields[name] = value
attrs = _record_attrs(record)
if attrs:
fields["attrs"] = redact(attrs)
if record.exc_info:
fields["exc"] = redact_text(logging.Formatter().formatException(record.exc_info))
elif record.exc_text:
# A QueueHandler-style hand-off formats the traceback into exc_text and drops exc_info.
fields["exc"] = redact_text(record.exc_text)
fields["version"] = settings.app_version
return fields
class JsonFormatter(logging.Formatter):
"""One JSON object per line — the shape Loki and the `log_events` table both ingest."""
def format(self, record: logging.LogRecord) -> str:
payload = record_fields(record)
payload["ts"] = payload["ts"].isoformat(timespec="milliseconds")
return json.dumps(payload, ensure_ascii=False, default=str)
class TextFormatter(logging.Formatter):
"""Human-readable single line for a terminal (`LOG_FORMAT=text`): the same content, ordered for
reading. Redaction applies here too — a dev log is still a log someone pastes into an issue."""
def format(self, record: logging.LogRecord) -> str:
stamp = datetime.fromtimestamp(record.created, timezone.utc).strftime("%H:%M:%S.%f")[:-3]
head = f"{stamp} {record.levelname:<5} [{record.name}]"
rid = getattr(record, "request_id", None)
if rid:
head += f" {rid}"
code = getattr(record, "event", None)
line = f"{head} {code + ' — ' if code else ''}{redact_text(_safe_message(record))}"
attrs = _record_attrs(record)
if attrs:
line += f" {json.dumps(redact(attrs), ensure_ascii=False, default=str)}"
if record.exc_info:
line += "\n" + redact_text(self.formatException(record.exc_info))
return line
# --- configuration -------------------------------------------------------------------------
# Third-party loggers that are noisy at our own level. Pinned explicitly so the diagnostics knob can
# lower the ROOT level to DEBUG/TRACE without drowning the output in SQL statements and HTTP chatter
# — raising THESE is a deliberate, separate act (E22.6's console).
_THIRD_PARTY_LEVELS = {
"sqlalchemy": "WARNING",
"sqlalchemy.engine": "WARNING",
"apscheduler": "WARNING",
"urllib3": "WARNING",
"httpx": "WARNING",
"httpcore": "WARNING",
"asyncio": "WARNING",
"multipart": "WARNING",
"PIL": "WARNING",
"alembic": "INFO",
# uvicorn's own startup/shutdown/WebSocket lines are worth keeping; its ACCESS log is not —
# it is one line per request with the query string in it, replaced by `log_requests` below.
"uvicorn": "INFO",
"uvicorn.error": "INFO",
"uvicorn.access": "CRITICAL",
}
def resolve_level(name: str) -> int:
"""Level name → number. Both ways in (the `LOG_LEVEL` env and the `log_level` admin knob) are
already held to the `LogLevel` choices, so an unknown name here is a programming error."""
value = logging.getLevelName(name)
if not isinstance(value, int):
raise ValueError(f"unknown log level {name!r}")
return value
def configure_logging(source: str) -> None:
"""Configure logging for THIS process. Call once, first thing, in every entry point:
`main.py` (api), `worker.main`, `downloads/runner.main`, `alembic/env.py`.
`source` names the process in every line. The level starts at `LOG_LEVEL` (the bootstrap default);
the runtime admin knob goes through `set_level()` instead, so changing it needs no restart.
Our loggers deliberately keep `propagate=True` and own no handler: everything reaches the single
root StreamHandler. That is what keeps `caplog` working in tests, and it means a new logger name
is configured the moment it is created."""
global _source
_source = source
formatter = settings.log_format
logging.config.dictConfig(
{
"version": 1,
# Never disable loggers created at import time (module-level `getLogger` calls run
# before this function does).
"disable_existing_loggers": False,
"formatters": {
"json": {"()": f"{__name__}.JsonFormatter"},
"text": {"()": f"{__name__}.TextFormatter"},
},
"filters": {
"context": {"()": f"{__name__}.ContextFilter"},
"gate": {"()": f"{__name__}.LevelGate"},
},
"handlers": {
"stdout": {
"class": "logging.StreamHandler",
"stream": sys.stdout,
"formatter": formatter,
"filters": ["context", "gate"],
}
},
"root": {"level": resolve_level(settings.log_level), "handlers": ["stdout"]},
"loggers": {name: {"level": lvl} for name, lvl in _THIRD_PARTY_LEVELS.items()},
}
)
set_level(settings.log_level)
def set_level(name: str) -> None:
"""Change the running process's level (the `log_level` admin knob). Third-party loggers keep
their pinned levels, so turning on DEBUG shows OUR detail, not SQLAlchemy's."""
global _base_level
_base_level = resolve_level(name)
_apply_root_level(force=True)
def current_level() -> str:
"""The configured level — not the root logger's, which an active elevation may have lowered."""
return logging.getLevelName(_base_level or logging.getLogger().getEffectiveLevel())
# --- per-user elevation (E22 S5) -----------------------------------------------------------
# The base level (`LOG_LEVEL`, then the admin knob). NOTSET until configure_logging runs — a unit
# test, a script — so the gate lets everything through there, as it did before the gate existed.
_base_level = logging.NOTSET
# user id → (level number, expiry) for every elevation this process has seen and not yet expired.
_elevations: dict[int, tuple[int, datetime]] = {}
_elevations_lock = threading.Lock()
_root_applied: int | None = None
def _track_elevation(user_id: int, level: int | None, until: datetime | None) -> None:
with _elevations_lock:
if level is None or until is None:
_elevations.pop(user_id, None)
else:
_elevations[user_id] = (level, until)
active = bool(_elevations)
if active or _root_applied != _base_level:
_apply_root_level()
def _apply_root_level(force: bool = False) -> None:
"""Root level = the lowest of the base and every unexpired elevation (expired ones are pruned
here). `setLevel` clears every logger's level cache, so it runs only when the number changes."""
global _root_applied
if _base_level == logging.NOTSET:
return # not configured: the root keeps whatever the process set up
now = datetime.now(timezone.utc)
# The level is SET under the lock too, not just computed: two threads that each computed a
# different answer would otherwise apply them in either order, and the older one could win —
# the root back at the base while an elevation is still active.
with _elevations_lock:
for user_id in [u for u, (_, until) in _elevations.items() if until <= now]:
del _elevations[user_id]
lowest = min([_base_level, *(lvl for lvl, _ in _elevations.values())])
if force or lowest != _root_applied:
logging.getLogger().setLevel(lowest)
_root_applied = lowest
# --- erased accounts ------------------------------------------------------------------------
# `auth.purge_user` deletes an account's `log_events` rows, but a line about that account can still
# be on its way: the erasing request's own access line is logged AFTER the handler committed, a queued
# row waits for the sink's writer, a background task logs later. Those rows are stored without the
# account (`logsink` asks here). Remembered for an hour in this process — a line later than that has
# no request left to come from, and the retention job is the backstop for any other process.
ERASED_MEMORY_SECONDS = 3600.0
_erased: dict[int, float] = {}
_erased_lock = threading.Lock()
# Held by whoever checks rows against `is_erased` AND writes them (`erasure_guard`). `forget_user`
# takes it too, so an erasure waits for a write that already passed the check to commit — and the
# purge's DELETE, which runs after `forget_user`, then removes those rows as well. Without it a row
# checked a moment before the erasure could be inserted a moment after the DELETE.
_erasure_guard = threading.Lock()
def erasure_guard() -> threading.Lock:
"""Hold this around "drop erased accounts from these rows, insert them, commit"."""
return _erasure_guard
def forget_user(user_id: int) -> None:
"""Mark `user_id` erased: no line logged about it from now on is stored under its id. Call BEFORE
deleting its rows (see `_erasure_guard`)."""
now = time.monotonic()
with _erasure_guard, _erased_lock:
for stale in [u for u, at in _erased.items() if now - at > ERASED_MEMORY_SECONDS]:
del _erased[stale]
_erased[user_id] = now
with _elevations_lock:
_elevations.pop(user_id, None)
# That pop can have removed the lowest elevation, and the root level is the lowest of the base
# and every unexpired one. Recomputed here rather than left to whichever later request happens
# to call `_track_elevation` — the invariant `_apply_root_level` documents holds at every pop.
_apply_root_level()
def is_erased(user_id: int | None) -> bool:
if user_id is None:
return False
with _erased_lock:
at = _erased.get(user_id)
return at is not None and time.monotonic() - at <= ERASED_MEMORY_SECONDS
def current_source() -> str:
"""The process name `configure_logging` was given (api, worker, runner, alembic)."""
return _source
# --- request logging -----------------------------------------------------------------------
# A client may supply its own request id so a browser log line and a server line join up (E22 S4).
# Accepted only in this shape: it ends up in a log field, and an unbounded header would be a way to
# write arbitrary text into the log.
_REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9._-]{8,64}$")
# Paths whose success is logged at DEBUG rather than INFO: the container healthcheck runs every 15s
# (5.7k lines/day saying nothing) and hashed static assets are served by the hundred per page load.
# A FAILURE on either still logs at WARNING/ERROR — this lowers the floor, it does not hide.
_QUIET_PREFIXES = ("/healthz", "/assets/", "/static/")
_request_log = logging.getLogger("siftlode.http")
async def log_requests(request, call_next):
"""The single access-log line, replacing uvicorn's.
What changes versus `uvicorn.access`: the ROUTE TEMPLATE instead of the concrete path (so
`/api/videos/{video_id}` groups), never the query string (it carried `?account=`, and at TRACE
the redacted query belongs in attrs, not in the URL), a request id echoed back in
`X-Request-Id` (with the template in `X-Route`), the duration, and the account — read from `request.state`, which the auth
dependency fills, because this middleware sits outside both the session middleware and the
downstream task's context."""
incoming = request.headers.get("x-request-id", "")
rid = incoming if _REQUEST_ID_RE.match(incoming) else uuid.uuid4().hex[:16]
token = _request_id.set(rid)
scope = _RequestScope()
scope_token = _scope.set(scope)
request.state.request_id = rid
started = time.perf_counter()
try:
response = await call_next(request)
except Exception:
# The unhandled-exception path: FastAPI's ServerErrorMiddleware re-raises after its own
# 500, so without this the ONE request that most needs a log line is the one with no
# access-log entry at all.
_request_log.exception(
"request failed",
extra={
**event(
"http.request",
method=request.method,
path=_route_template(request),
duration_ms=round((time.perf_counter() - started) * 1000, 1),
),
**_request_owner(request, logging.ERROR),
},
)
_scope.reset(scope_token)
_request_id.reset(token)
raise
duration_ms = round((time.perf_counter() - started) * 1000, 1)
status = response.status_code
quiet = request.url.path.startswith(_QUIET_PREFIXES)
if status >= 500:
level = logging.ERROR
elif status >= 400:
level = logging.WARNING
else:
level = logging.DEBUG if quiet else logging.INFO
attrs: dict[str, Any] = {
"method": request.method,
"path": _route_template(request),
"status": status,
"duration_ms": duration_ms,
}
# Teljes (TRACE) the user consented to: the query string too — redacted, and in attrs only, never
# in the message.
if request.url.query and _traced(scope):
attrs["query"] = redact_query(request.url.query)
_request_log.log(
level,
"%s %s %s",
request.method,
_route_template(request),
status,
extra={**event("http.request", **attrs), **_request_owner(request, level)},
)
response.headers["X-Request-Id"] = rid
# The template, so the browser's copy of this line groups the same way (frontend lib/http.ts).
# Only a MATCHED route's: a template never carries a path parameter's value (a share token stays
# `{token}`), while the raw-path fallback of an unmatched request could — the browser then
# templates the path itself.
matched = _matched_route(request)
if matched:
response.headers["X-Route"] = matched
_scope.reset(scope_token)
_request_id.reset(token)
return response
def _traced(scope: _RequestScope) -> bool:
"""Whether this request's query string is logged: only under its OWN user's Teljes consent. An
admin setting the whole process to TRACE gets more detail, not every user's search terms — the
privacy policy promises those are kept only for a user who agreed to it."""
return scope.level is not None and scope.level <= TRACE
# Anonymous request lines kept per client per minute. Signed-in traffic is not capped: it is a known
# account and exactly what an admin needs to see. Anonymous traffic is where the volume can come from
# nobody in particular — a scanner walking `/wp-login.php`, a bot on a share link — and each of its
# 404s is a WARNING that would otherwise land in `log_events` without limit. The client key is an
# in-memory counter key only; no address is ever written to a log line or a row.
ANONYMOUS_LINES_PER_MINUTE = 60
_anonymous_lines = RateLimiter(ANONYMOUS_LINES_PER_MINUTE, 60.0)
def _request_owner(request, level: int) -> dict[str, Any]:
"""`extra=` fields naming who the request belongs to, and whether its line may be stored. The
account comes from `request.state` (filled by the auth dependency), so it lands in the line's
top-level `user_id` — the correlation field — rather than in `attrs`."""
user_id = getattr(request.state, "user_id", None)
if user_id is not None:
return {"user_id": user_id}
if level < logging.INFO:
return {} # a quiet line is never stored anyway — don't spend the client's allowance on it
return {} if _anonymous_lines.allow(client_key(request)) else {PERSIST: False}
def _matched_route(request) -> str | None:
"""The matched route's template (`/api/videos/{video_id}`), or None when nothing matched."""
return getattr(request.scope.get("route"), "path", None)
def _route_template(request) -> str:
"""The matched route's template, else the raw path for a 404 or a static file. Never the query
string."""
return _matched_route(request) or request.url.path
+315
View File
@@ -0,0 +1,315 @@
"""The `log_events` sink (E22 S3): the stored copy of the log the diagnostics console reads.
stdout stays the complete log (docker keeps 3×50 MB of it); this table keeps the part worth querying
for `log_retention_days`:
* every WARNING and above,
* the `http.request` line of every request at INFO or above (not the DEBUG-level health checks and
static files),
* every line of a request whose user raised their diagnostics level (E22 S5 — `logs.LevelGate`
marks them), which is the detail they consented to have kept,
* nothing a caller marked `extra={logs.PERSIST: False}` — how the request log caps anonymous
traffic per client.
What the table must never do is slow down or break the code that logs. So a line is redacted and
turned into a row on the CALLER's thread (the same `logs.record_fields` the stdout line is built
from), put on a bounded queue without blocking, and written in batches by one background thread
on its own connection — never inside a request's transaction. A full queue drops the row and
counts it; a failed write drops the batch and says so on stderr. Both are reported rather than
retried: a log that retries into a sick database is how a small outage becomes a big one.
Flood control is the second half. On 2026-09-16 a YouTube RSS outage made prod log 11k near-identical
warnings in four hours. stdout keeps them all; the table keeps `REPEATS_PER_MINUTE` of each message
template per minute and one `log.repeats_suppressed` row saying how many more there were.
Core layer: imports `logs`, `db`, `models`, `config` — nothing above them.
"""
import atexit
import json
import logging
import queue
import sys
import threading
import time
from datetime import datetime, timedelta, timezone
from typing import Any
from sqlalchemy import delete, insert
from sqlalchemy.engine import Engine
from sqlalchemy.orm import Session
from app import logs
from app.config import settings
from app.db import engine
from app.models import LogEvent
QUEUE_MAX = 10_000
BATCH_MAX = 500
FLUSH_SECONDS = 1.0
REPEATS_PER_MINUTE = 30
REPEAT_WINDOW_SECONDS = 60.0
# Distinct templates tracked per window. Past this many, a new template is stored without counting
# — the limiter must not become the unbounded thing it exists to prevent.
_REPEAT_KEYS_MAX = 2_000
# Column widths from models.LogEvent. A value past its column would fail the WHOLE batch it rides in.
_LIMITS = {
"level": 10,
"source": 16,
"logger": 128,
"event": 64,
"request_id": 64,
"debug_session_id": 64,
"version": 64,
"msg": 8_000,
"exc": 32_000,
}
_log = logging.getLogger("siftlode.logsink")
def should_store(record: logging.LogRecord) -> bool:
"""Whether this record belongs in `log_events` (see the module docstring)."""
if getattr(record, logs.PERSIST, True) is False:
return False
if record.levelno >= logging.WARNING or getattr(record, "elevated", False):
return True
return getattr(record, "event", None) == "http.request" and record.levelno >= logging.INFO
def _without_nul(value: Any) -> Any:
"""`value` with every NUL character replaced, in keys and nested values alike. Postgres refuses
NUL in text and `\\u0000` in JSONB, and a refused row fails the WHOLE batch it rides in — every
user's rows with it. Anything logged can carry one (`GET /assets/%00` logs the decoded path)."""
if isinstance(value, str):
return value.replace("\x00", "�")
if isinstance(value, dict):
return {_without_nul(k): _without_nul(v) for k, v in value.items()}
if isinstance(value, list):
return [_without_nul(v) for v in value]
return value
def to_row(fields: dict[str, Any]) -> dict[str, Any]:
"""`logs.record_fields` output → a `log_events` row: known columns only, text clipped to its
column and free of NUL, attrs made JSON-safe (anything exotic becomes its `str`, as on stdout)."""
row = {
column: fields.get(column) for column in LogEvent.__table__.columns.keys() if column != "id"
}
for column, limit in _LIMITS.items():
value = row.get(column)
if isinstance(value, str):
value = _without_nul(value)
row[column] = value[: limit - 1] + "…" if len(value) > limit else value
if row["attrs"] is not None:
row["attrs"] = _without_nul(
json.loads(json.dumps(row["attrs"], ensure_ascii=False, default=str))
)
return row
def without_erased(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""The rows, with the account taken off any whose account was erased meanwhile
(`logs.forget_user`). Checked right before the INSERT, under `logs.erasure_guard()` held until
the commit: a row built before the erasure can still be waiting for its batch."""
for row in rows:
if logs.is_erased(row.get("user_id")):
row["user_id"] = None
return rows
def _notice(code: str, msg: str, **attrs: Any) -> dict[str, Any]:
"""A row the sink writes about ITSELF. Built directly, not logged: logging it would route it back
through this sink (and the writer thread's own records are deliberately ignored)."""
return to_row(
{
"ts": datetime.now(timezone.utc),
"level": "WARNING",
"sev": logs.SEVERITY["WARNING"],
"source": logs.current_source(),
"logger": _log.name,
"event": code,
"msg": msg,
"attrs": attrs,
"version": settings.app_version,
}
)
class _Repeats:
"""Per-template admission for one minute at a time. Used by the writer thread only (unlocked)."""
def __init__(self, per_window: int, window_seconds: float):
self.per_window = per_window
self.window = window_seconds
self.started = time.monotonic()
self.seen: dict[tuple[str, str, int], int] = {}
def admit(self, key: tuple[str, str, int] | None) -> bool:
if key is None:
return True
count = self.seen.get(key)
if count is None and len(self.seen) >= _REPEAT_KEYS_MAX:
return True
self.seen[key] = (count or 0) + 1
return self.seen[key] <= self.per_window
def roll(self, now: float) -> list[dict[str, Any]]:
"""Close the window if it is over: one notice per template that went past the limit."""
if now - self.started < self.window:
return []
notices = [
_notice(
"log.repeats_suppressed",
f"{count - self.per_window} more like this were not stored",
logger=logger,
template=logs.redact_text(template),
suppressed=count - self.per_window,
window_seconds=int(self.window),
)
for (logger, template, _level), count in self.seen.items()
if count > self.per_window
]
self.seen.clear()
self.started = now
return notices
class LogSink(logging.Handler):
"""The handler plus its writer. `start()` runs the writer thread; a test drives `write_pending()`
directly instead, on its own thread, with no timing involved."""
def __init__(self, bind: Engine = engine):
super().__init__(level=logging.NOTSET)
# Correlation fields are contextvars: capture them now, on the thread that logged.
self.addFilter(logs.ContextFilter())
self._bind = bind
self._queue: queue.Queue[tuple[tuple[str, str, int] | None, dict[str, Any]]] = queue.Queue(
QUEUE_MAX
)
self._dropped = 0
self._dropped_lock = threading.Lock()
self._repeats = _Repeats(REPEATS_PER_MINUTE, REPEAT_WINDOW_SECONDS)
self._stopping = threading.Event()
self._writer: threading.Thread | None = None
# --- caller side ---------------------------------------------------------------------------
def emit(self, record: logging.LogRecord) -> None:
if threading.current_thread() is self._writer:
return # the writer's own trouble (a driver warning mid-insert) must not feed itself
if not should_store(record):
return
try:
row = to_row(logs.record_fields(record))
except Exception: # noqa: BLE001 — a record we cannot turn into a row is reported, not raised
self.handleError(record)
return
# Request lines are one per request by nature; only the repeated SAME message is a flood.
key = (
None
if row["event"] == "http.request"
else (record.name, str(record.msg), record.levelno)
)
try:
self._queue.put_nowait((key, row))
except queue.Full:
with self._dropped_lock:
self._dropped += 1
# --- writer side ---------------------------------------------------------------------------
def write_pending(self, wait: float = 0.0) -> int:
"""Write one batch: whatever is queued (up to BATCH_MAX), waiting up to `wait` seconds for the
first item. Returns the number of rows written, notices included."""
rows: list[dict[str, Any]] = []
try:
item = self._queue.get(timeout=wait) if wait > 0 else self._queue.get_nowait()
except queue.Empty:
item = None
while item is not None:
key, row = item
if self._repeats.admit(key):
rows.append(row)
if len(rows) >= BATCH_MAX:
break
try:
item = self._queue.get_nowait()
except queue.Empty:
item = None
rows.extend(self._repeats.roll(time.monotonic()))
with self._dropped_lock:
dropped, self._dropped = self._dropped, 0
if dropped:
rows.append(
_notice("log.sink.dropped", f"{dropped} log events were dropped", dropped=dropped)
)
if not rows:
return 0
try:
# The connection is checked out BEFORE the guard, and every request path takes them in
# that order too (its session already holds a connection when it reaches the guard). The
# other way round, this thread would hold a process-global lock while waiting up to
# `pool_timeout` for a connection, and the requests that could return one are the ones
# blocked on the lock. The guard still spans the check, the insert and the commit.
with self._bind.connect() as conn, logs.erasure_guard(), conn.begin():
conn.execute(insert(LogEvent), without_erased(rows))
except Exception as exc: # noqa: BLE001 — the database is down or refused the batch
print(
f"log sink: could not store {len(rows)} log events: {logs.redact_text(str(exc))[:300]}",
file=sys.stderr,
)
return 0
return len(rows)
def _run(self) -> None:
while not self._stopping.is_set():
self.write_pending(wait=FLUSH_SECONDS)
while self.write_pending(): # drain what arrived before the stop
pass
def start(self) -> None:
self._writer = threading.Thread(target=self._run, name="log-sink", daemon=True)
self._writer.start()
def stop(self, timeout: float = 5.0) -> None:
self._stopping.set()
if self._writer is not None:
self._writer.join(timeout)
_sink: LogSink | None = None
def start() -> None:
"""Attach the sink to the root logger and start its writer. Call after `logs.configure_logging`
(which replaces the root handlers) and once the schema exists. A no-op when `LOG_DB_SINK` is off
or it is already running."""
global _sink
if _sink is not None or not settings.log_db_sink:
return
_sink = LogSink()
# The same level gate as stdout: the root may be lowered for one user's elevated requests, and
# everyone else's detail must not reach the table through that gap either.
_sink.addFilter(logs.LevelGate())
logging.getLogger().addHandler(_sink)
_sink.start()
# A short-lived process (the download child) exits without a lifespan: flush on the way out.
atexit.register(stop)
def stop() -> None:
"""Detach the sink and write what is still queued."""
global _sink
if _sink is None:
return
logging.getLogger().removeHandler(_sink)
_sink.stop()
_sink = None
def gc(db: Session, retention_days: int) -> int:
"""Stage the deletion of rows older than `retention_days` (at least one day — stored logs always
expire). The caller commits. Returns the number of rows removed."""
cutoff = datetime.now(timezone.utc) - timedelta(days=max(1, retention_days))
result = db.execute(delete(LogEvent).where(LogEvent.ts < cutoff))
return result.rowcount or 0
+2 -2
View File
@@ -175,7 +175,7 @@ MAIL_TEXT: dict[str, dict[str, str]] = {
"en": (
"Hi,\n\n"
"Someone has opened up Siftlode for you — a quiet, self-hosted reader for your YouTube "
"subscriptions. Every upload from the channels you follow lands in one feed you can "
"subscriptions. Every upload from the channels you follow lands in one Feed you can "
"actually filter and sort, with no algorithm deciding what you see, and no Shorts or "
"livestreams unless you want them.\n\n"
"{next_step}\n\n"
@@ -186,7 +186,7 @@ MAIL_TEXT: dict[str, dict[str, str]] = {
"Szia,\n\n"
"Valaki megnyitotta előtted a Siftlode-ot — ez egy csendes, saját üzemeltetésű olvasó "
"a YouTube-feliratkozásaidhoz. A követett csatornák minden feltöltése egyetlen "
"hírfolyamba érkezik, amit tényleg tudsz szűrni és rendezni; nincs algoritmus, ami "
"Feedbe érkezik, amit tényleg tudsz szűrni és rendezni; nincs algoritmus, ami "
"eldönti, mit látsz, és nincs Shorts vagy élő adás, hacsak nem kéred.\n\n"
"{next_step}\n\n"
"Nem sürget semmi — az ajtó nyitva marad.\n\n"
+32 -11
View File
@@ -1,7 +1,6 @@
import asyncio
import logging
import math
import sys
from contextlib import asynccontextmanager, suppress
from pathlib import Path
@@ -9,16 +8,13 @@ from fastapi import FastAPI, HTTPException, Request
from fastapi.encoders import jsonable_encoder
from fastapi.exceptions import RequestValidationError
# When started via uvicorn --log-config the "siftlode" logger is already configured
# (see log_config.json). This block is a timestamped fallback for other entrypoints
# (tests, scripts) so our logs are never silently dropped.
_siftlode_logger = logging.getLogger("siftlode")
if not _siftlode_logger.handlers:
_handler = logging.StreamHandler(sys.stdout)
_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)-5s [%(name)s] %(message)s"))
_siftlode_logger.addHandler(_handler)
_siftlode_logger.setLevel(logging.INFO)
_siftlode_logger.propagate = False
# Logging is configured HERE, before the route modules are imported, so every logger they create at
# import time is already covered. ONE format for the api, the worker, the download child and alembic
# alike (app/logs.py) — this replaces the old uvicorn --log-config + the hand-rolled fallback that
# used to live here, and uvicorn's access log is replaced by the `http.request` middleware below.
from app.logs import configure_logging, current_level, log_requests, set_level
configure_logging("api")
log = logging.getLogger("siftlode.app")
from fastapi.middleware.cors import CORSMiddleware
@@ -27,7 +23,9 @@ from fastapi.staticfiles import StaticFiles
from sqlalchemy.exc import DataError
from starlette.middleware.sessions import SessionMiddleware
from app import logsink
from app import state
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.plex import stream as plex_stream
@@ -38,6 +36,7 @@ from app.routes import (
auth as auth_routes,
channels,
config as config_routes,
diagnostics as diagnostics_routes,
discover as discover_routes,
downloads,
feed,
@@ -63,9 +62,21 @@ from app.scheduler import shutdown_scheduler, start_scheduler
@asynccontextmanager
async def lifespan(app: FastAPI):
log.info("Siftlode starting up")
# The stored copy of the log (log_events). The entrypoint has already migrated the schema.
logsink.start()
# First-run: if the instance isn't configured yet, mint a fresh one-time setup token and print
# the wizard URL so the operator can open it (the app runs in setup mode until they finish).
with SessionLocal() as db:
# The admin's runtime level (sysconfig `log_level`) wins over the LOG_LEVEL bootstrap
# default, so a level chosen in the UI survives a restart without touching the environment.
# Best-effort on purpose: a LOGGING knob must never be the thing that stops the app from
# booting — an unreachable or not-yet-migrated database has to start into the env level.
try:
set_level(sysconfig.get_str(db, "log_level"))
except Exception as exc: # noqa: BLE001 — any DB/config failure keeps the bootstrap level
log.warning(
"could not read the stored log level (%s); staying at %s", exc, current_level()
)
if not state.is_configured(db):
token = state.rotate_setup_token(db)
url = f"{settings.app_base}/setup?token={token}"
@@ -100,6 +111,7 @@ async def lifespan(app: FastAPI):
# Kills a running subtitle extraction: its worker would otherwise hold interpreter exit for
# the rest of the read (see plex_subtitles.shutdown).
plex_subtitles.shutdown()
logsink.stop() # last: write what the shutdown itself just logged
# How often idle HLS sessions are reaped (see plex/stream.py `reap_idle`).
@@ -243,6 +255,14 @@ async def security_headers(request, call_next):
return response
# The access log (app/logs.py): request id, route template, status, duration — one line, never a
# query string. Registered LAST so it is the OUTERMOST http middleware and therefore times the whole
# request, including what the three above do. Being outermost also puts it OUTSIDE SessionMiddleware,
# which is why the account comes from `request.state.user_id` (set by the auth dependency, the one
# place a session is actually resolved into an account) rather than from `request.session`.
app.middleware("http")(log_requests)
app.include_router(health.router)
app.include_router(auth_routes.router)
app.include_router(sync.router)
@@ -251,6 +271,7 @@ app.include_router(feed.router)
app.include_router(discover_routes.router)
app.include_router(search_routes.router)
app.include_router(me.router)
app.include_router(diagnostics_routes.router)
app.include_router(notifications.router)
app.include_router(messages.router)
app.include_router(channels.router)
+44
View File
@@ -12,6 +12,7 @@ from sqlalchemy import (
Index,
Integer,
JSON,
SmallInteger,
String,
Text,
UniqueConstraint,
@@ -102,6 +103,16 @@ class User(Base, TimestampMixin, CanonicalEmailMixin):
# current_user rejects any cookie whose epoch is stale, so a stolen/copied cookie dies on those
# events (client-side signed cookies otherwise stay valid until they expire).
session_epoch: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# When this account last signed in (Google or password), stamped by the one sign-in path,
# routes/auth._establish_session. NULL = not since the column existed (0085; no backfill is
# possible). The shared demo account is never stamped — its "last login" would mean nobody.
last_login_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The diagnostics level this user consented to (E22 S5) and when it lapses back to the base
# level. Both NULL = base. Written only by `PUT /api/me/diagnostics`, which computes the expiry
# itself; read through `app.diagnostics.effective_level`, never directly (an expired level is
# not cleared eagerly — it simply stops counting).
diagnostics_level: Mapped[str | None] = mapped_column(String(8))
diagnostics_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
display_name: Mapped[str | None] = mapped_column(String(255))
avatar_url: Mapped[str | None] = mapped_column(String(1024))
role: Mapped[str] = mapped_column(String(16), default="user", server_default="user")
@@ -539,6 +550,37 @@ class AuditLog(Base):
)
class LogEvent(Base):
"""One stored log event (E22 S3) — the `log_events` table the diagnostics console reads. Rows are
written by app/logsink.py's background writer, never inside a request's transaction, already
redacted (the same `logs.record_fields` the stdout line is built from).
`user_id` has NO foreign key on purpose: the writer runs behind the event, so a line about an
account can arrive after that account was purged, and a constraint would make it fail the whole
batch it travels in. `auth.purge_user` deletes an account's rows explicitly instead, after
`logs.forget_user` makes every later write in this process store such a line without the account;
the retention job (`log_retention_days`) bounds whatever another process writes late."""
__tablename__ = "log_events"
__table_args__ = (Index("ix_log_events_user_ts", "user_id", "ts"),)
id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
ts: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
level: Mapped[str] = mapped_column(String(10))
sev: Mapped[int] = mapped_column(SmallInteger)
source: Mapped[str] = mapped_column(String(16))
logger: Mapped[str] = mapped_column(String(128))
event: Mapped[str | None] = mapped_column(String(64))
msg: Mapped[str] = mapped_column(Text)
attrs: Mapped[dict | None] = mapped_column(JSONB)
exc: Mapped[str | None] = mapped_column(Text)
request_id: Mapped[str | None] = mapped_column(String(64), index=True)
user_id: Mapped[int | None] = mapped_column(Integer)
debug_session_id: Mapped[str | None] = mapped_column(String(64))
job_id: Mapped[int | None] = mapped_column(Integer)
version: Mapped[str | None] = mapped_column(String(64))
class AppState(Base):
"""Single-row global app state (admin-controlled)."""
@@ -871,6 +913,8 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
error_code: Mapped[str | None] = mapped_column(
String(24)
) # category, see downloads.errors.classify
# The host-specific failure-catalog entry on top of the category (`facebook.login_wall`), or NULL.
error_detail: Mapped[str | None] = mapped_column(String(48)) # see downloads.errors.diagnose
queue_pos: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
+2 -2
View File
@@ -404,7 +404,7 @@ def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) ->
source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
if not transcode:
return (False, False)
mode = (sysconfig.get_str(db, "plex_transcode_hw") or "auto").lower()
mode = sysconfig.get_str(db, "plex_transcode_hw")
if mode == "off" or not _render_node_present():
return (False, False)
if mode == "on":
@@ -664,7 +664,7 @@ def start_session(
# remux ignore preset/crf/max_height, and remux must not touch the DB for them).
preset, crf, max_height = "veryfast", 20, 0
if transcode and not hw:
preset = sysconfig.get_str(db, "plex_transcode_preset") or "veryfast"
preset = sysconfig.get_str(db, "plex_transcode_preset")
crf = sysconfig.get_int(db, "plex_transcode_crf")
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
+65 -11
View File
@@ -1,21 +1,30 @@
"""A tiny in-process sliding-window rate limiter.
Generic groundwork: keyed by an arbitrary string (e.g. a client IP), so it can throttle any
Generic groundwork: keyed by an arbitrary string (e.g. a client — `client_key`), so it can throttle any
endpoint. Single-worker uvicorn → in-memory state is sufficient; it resets on restart, which
is fine for abuse throttling (not for anything that must survive a deploy). Not shared across
processes — if we ever run multiple workers, swap the backing store for Redis behind the same
``allow()`` interface.
"""
import ipaddress
import threading
import time
from fastapi import Request
from app.config import settings
class RateLimiter:
def __init__(self, max_events: int, window_seconds: float):
"""At most `max_events` attempts per `window_seconds` per key — or, when each attempt passes a
`cost`, at most that much total cost (a byte budget counts bytes, not calls)."""
def __init__(self, max_events: float, window_seconds: float):
self.max_events = max_events
self.window = window_seconds
self._hits: dict[str, list[float]] = {}
# key → (monotonic time, cost) of every counted attempt still inside the window
self._hits: dict[str, list[tuple[float, float]]] = {}
self._lock = threading.Lock()
def reset(self) -> None:
@@ -25,27 +34,72 @@ class RateLimiter:
with self._lock:
self._hits.clear()
def allow(self, key: str) -> bool:
"""Record an attempt for ``key`` and return whether it is within the limit.
def exhausted(self, key: str) -> bool:
"""Whether ``key``'s window is already full (``allow`` would refuse it), WITHOUT counting an
attempt — for a limit that counts only some outcomes: ask it first, ``allow`` the ones that
count."""
cutoff = time.monotonic() - self.window
with self._lock:
return sum(c for t, c in self._hits.get(key, []) if t > cutoff) >= self.max_events
True -> under the cap (the attempt is counted).
False -> the cap for the current window is already reached (attempt NOT counted, so a
def allow(self, key: str, cost: float = 1) -> bool:
"""Record an attempt costing ``cost`` for ``key`` and return whether it is within the limit.
True -> within the cap (the attempt is counted).
False -> the attempt would pass the cap for the current window (attempt NOT counted, so a
blocked caller can't keep pushing the window forward)."""
now = time.monotonic()
cutoff = now - self.window
with self._lock:
hits = [t for t in self._hits.get(key, []) if t > cutoff]
if len(hits) >= self.max_events:
hits = [(t, c) for t, c in self._hits.get(key, []) if t > cutoff]
if sum(c for _, c in hits) + cost > self.max_events:
self._hits[key] = hits
return False
hits.append(now)
hits.append((now, cost))
self._hits[key] = hits
# Opportunistic cleanup so abandoned keys don't accumulate unboundedly.
if len(self._hits) > 4096:
for k in list(self._hits):
fresh = [t for t in self._hits[k] if t > cutoff]
fresh = [(t, c) for t, c in self._hits[k] if t > cutoff]
if fresh:
self._hits[k] = fresh
else:
del self._hits[k]
return True
def _norm_ip(s: str) -> ipaddress.IPv4Address | ipaddress.IPv6Address | None:
"""Parse an IP for comparison, unwrapping IPv4-mapped IPv6 (`::ffff:10.0.0.1` → `10.0.0.1`) so a
plain-IPv4 allowlist still matches a peer that surfaces in mapped form. None for a non-IP string."""
try:
ip = ipaddress.ip_address(s.strip())
except ValueError:
return None
return ip.ipv4_mapped if getattr(ip, "ipv4_mapped", None) else ip
def client_key(request: Request) -> str:
"""Best-effort client identity for rate limiting: its IP, or for IPv6 the /64 the address sits in — a
subscriber is handed a whole /64, so keyed on the full address one client would have endless identities
to spread its attempts over. X-Forwarded-For is client-controlled, so we trust it
ONLY when the request actually arrived from a configured reverse proxy (settings.trusted_proxy_ips
— the address the proxy connects FROM, e.g. the VPS Caddy's WireGuard peer IP). Our proxy APPENDS
the client it saw to XFF, so the RIGHTMOST entry is the real client and can't be forged by a
client pre-seeding the header. Any other peer (someone hitting the app port directly, bypassing the
proxy) is untrusted — we use its real socket IP, so it can't spoof its rate-limit identity.
SAFETY: this relies on `request.client` being the TRUE socket peer. uvicorn is started WITHOUT
`--proxy-headers` (see entrypoint.sh) precisely so it never rewrites `request.client` from XFF —
do NOT add that flag, or the trust check below would compare against a forgeable value."""
peer = request.client.host if request.client else "unknown"
client = peer
trusted = {_norm_ip(p) for p in settings.trusted_proxy_ip_set} - {None}
if _norm_ip(peer) in trusted:
xff = request.headers.get("x-forwarded-for")
parts = [p.strip() for p in (xff or "").split(",") if p.strip()]
if parts:
client = parts[-1]
ip = _norm_ip(client)
if isinstance(ip, ipaddress.IPv6Address):
return str(ipaddress.IPv6Network((ip, 64), strict=False))
return client
+2
View File
@@ -408,6 +408,8 @@ def _serialize_user(u: User) -> dict:
# confusing sign-in.
"google_email": u.google_email if u.google_email != u.email else None,
"created_at": iso(u.created_at),
# NULL = no sign-in since the column existed (0085), or the shared demo account.
"last_login_at": iso(u.last_login_at),
}
+19 -51
View File
@@ -3,7 +3,6 @@
Split out of app/auth.py (R12 S2); the core auth surface (current_user, scopes, token
helpers) stays in app/auth.py, imported below."""
import ipaddress
import secrets
from datetime import datetime, timedelta, timezone
from typing import NamedTuple
@@ -40,7 +39,7 @@ from app.auth import (
from app.config import settings
from app.db import SessionLocal, get_db
from app.models import AuthToken, DemoWhitelist, Invite, OAuthToken, User
from app.ratelimit import RateLimiter
from app.ratelimit import RateLimiter, client_key
from app.security import encrypt, hash_password, hash_token, verify_password
from app.utils import canonical_email, valid_email
@@ -172,37 +171,6 @@ def is_demo_allowed(db: Session, email: str) -> bool:
)
def _norm_ip(s: str) -> ipaddress.IPv4Address | ipaddress.IPv6Address | None:
"""Parse an IP for comparison, unwrapping IPv4-mapped IPv6 (`::ffff:10.0.0.1` → `10.0.0.1`) so a
plain-IPv4 allowlist still matches a peer that surfaces in mapped form. None for a non-IP string."""
try:
ip = ipaddress.ip_address(s.strip())
except ValueError:
return None
return ip.ipv4_mapped if getattr(ip, "ipv4_mapped", None) else ip
def _client_ip(request: Request) -> str:
"""Best-effort client IP for rate limiting. X-Forwarded-For is client-controlled, so we trust it
ONLY when the request actually arrived from a configured reverse proxy (settings.trusted_proxy_ips
— the address the proxy connects FROM, e.g. the VPS Caddy's WireGuard peer IP). Our proxy APPENDS
the client it saw to XFF, so the RIGHTMOST entry is the real client and can't be forged by a
client pre-seeding the header. Any other peer (someone hitting the app port directly, bypassing the
proxy) is untrusted — we use its real socket IP, so it can't spoof its rate-limit identity.
SAFETY: this relies on `request.client` being the TRUE socket peer. uvicorn is started WITHOUT
`--proxy-headers` (see entrypoint.sh) precisely so it never rewrites `request.client` from XFF —
do NOT add that flag, or the trust check below would compare against a forgeable value."""
peer = request.client.host if request.client else "unknown"
trusted = {_norm_ip(p) for p in settings.trusted_proxy_ip_set} - {None}
if _norm_ip(peer) in trusted:
xff = request.headers.get("x-forwarded-for")
parts = [p.strip() for p in (xff or "").split(",") if p.strip()]
if parts:
return parts[-1]
return peer
class InviteRequest(NamedTuple):
"""What an access request settled on.
@@ -511,16 +479,8 @@ async def callback(
db.flush()
_store_token(db, user, token)
_establish_session(request, user)
db.commit()
# Multi-session: remember every account that has authenticated in this browser so the
# user can switch between them without going through Google again. Only accounts that
# actually completed OAuth here land in this list (it's the proof they may be switched to).
accounts = [a for a in (request.session.get("account_ids") or []) if a != user.id]
accounts.append(user.id)
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
request.session["user_id"] = user.id
_remember_epoch(request, user)
log.info("Login: %s (id=%s, role=%s)", email, user.id, user.role)
return RedirectResponse(url="/")
@@ -631,6 +591,8 @@ async def logout(request: Request):
out in one tab doesn't disturb another tab running a different account. If the account being
removed was the session default, promote the most recent remaining account (or clear the
session when none remain)."""
# A tab still pinned to an account already signed out in another tab resolves to nobody, so it
# signs nobody out — it used to fall back to the default and sign THAT account out in its place.
active, is_default = resolved_user_id(request)
remaining = [a for a in (request.session.get("account_ids") or []) if a != active]
request.session["account_ids"] = remaining
@@ -672,7 +634,7 @@ async def request_access(
email = (body.email or "").strip().lower()
if not valid_email(email):
raise HTTPException(status_code=400, detail="Enter a valid email address.")
if not _request_access_limiter.allow(_client_ip(request)):
if not _request_access_limiter.allow(client_key(request)):
return {"status": "pending"}
if is_allowed(db, email):
return {"status": "pending"} # already allowed — don't reveal it
@@ -689,7 +651,7 @@ def demo_login(body: EmailIn, request: Request, db: Session = Depends(get_db)) -
so there is no visible 'demo login' button. Rate-limited per IP; when blocked it answers
exactly like a non-match (authenticated=false) so the field can't be hammered as an
enumeration oracle. On a match it sets the session and the client reloads into the app."""
if not _demo_limiter.allow(_client_ip(request)):
if not _demo_limiter.allow(client_key(request)):
return {"authenticated": False}
email = (body.email or "").strip().lower()
if valid_email(email) and is_demo_allowed(db, email):
@@ -705,13 +667,18 @@ def demo_login(body: EmailIn, request: Request, db: Session = Depends(get_db)) -
def _establish_session(request: Request, user: User) -> None:
"""Sign `user` in for this browser session, mirroring the Google callback's multi-account
handling so password sign-in joins the same account switcher."""
"""Sign `user` in for this browser session — the ONE sign-in path, shared by the Google callback
and password login, so both join the same account switcher and both stamp `last_login_at` (the
caller commits). Multi-session: every account that authenticated in this browser is remembered so
the user can switch between them; only accounts that actually proved themselves HERE land in that
wallet (it is the proof they may be switched to). The demo entry deliberately does not come
through here: it clears the wallet instead, and its shared account has no meaningful last login."""
accounts = [a for a in (request.session.get("account_ids") or []) if a != user.id]
accounts.append(user.id)
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
request.session["user_id"] = user.id
_remember_epoch(request, user)
user.last_login_at = datetime.now(timezone.utc)
def _app_base() -> str:
@@ -835,7 +802,7 @@ def register(
)
if not sysconfig.get_bool(db, "allow_registration"):
raise HTTPException(status_code=403, detail="Registration is currently closed.")
if not _register_limiter.allow(_client_ip(request)):
if not _register_limiter.allow(client_key(request)):
return {"status": "ok"} # silently throttle; uniform response
# Do the existence check + account creation OFF the response path (a background task with its own
@@ -1003,7 +970,7 @@ def verify_email_resend(
and re-registering an already-registered email is a silent no-op, so a user who lost the first
mail had no way back. Uniform response + timing regardless of whether the email has a pending
account (the lookup runs off the response path), so it can't probe for registered emails."""
if not _verify_resend_limiter.allow(_client_ip(request)):
if not _verify_resend_limiter.allow(client_key(request)):
return {"status": "ok"} # silently throttle; uniform response
email = (body.email or "").strip().lower()
if valid_email(email):
@@ -1062,7 +1029,7 @@ def password_login(
"""Email+password sign-in. Wrong email/password give a single uniform 401 (no enumeration).
Once the password is proven correct, the owner gets a specific reason if their account is
still pending or suspended — that's not an enumeration leak (they already hold the password)."""
if not _login_limiter.allow(_client_ip(request)):
if not _login_limiter.allow(client_key(request)):
raise _login_refused(429, "rate_limited")
email = (body.email or "").strip().lower()
password = body.password or ""
@@ -1081,6 +1048,7 @@ def password_login(
if not user.is_active:
raise _login_refused(403, "pending_approval", user)
_establish_session(request, user)
db.commit()
log.info("Password login: %s (id=%s, role=%s)", email, user.id, user.role)
return {"ok": True}
@@ -1093,7 +1061,7 @@ def password_reset_request(
) -> dict:
"""Request a password-reset link. Uniform response + timing regardless of whether the email has a
password account (the lookup runs off the response path), so it can't probe for registered emails."""
if not _reset_limiter.allow(_client_ip(request)):
if not _reset_limiter.allow(client_key(request)):
return {"status": "ok"}
email = (body.email or "").strip().lower()
# Do the account lookup + token issue + email OFF the response path (a background task with its
@@ -1142,7 +1110,7 @@ def password_reset_check(body: TokenIn, request: Request, db: Session = Depends(
told `true`, never `false`: this check may only ever save a user a wasted form, so when it cannot
answer it must degrade to the old behaviour instead of declaring a live link dead.
"""
if not _reset_check_limiter.allow(_client_ip(request)):
if not _reset_check_limiter.allow(client_key(request)):
return {"ok": True}
return {"ok": any(_peek_token(db, body.token, k) is not None for k in _PASSWORD_TOKEN_KINDS)}
+11
View File
@@ -10,6 +10,7 @@ from sqlalchemy.orm import Session
from app import audit
from app import email as email_mod
from app import logs
from app import mailtext
from app import sysconfig
from app.audit import AuditAction
@@ -29,6 +30,14 @@ def _redact(v):
return _URL_USERINFO.sub(r"\1***@", v) if isinstance(v, str) else v
def _apply_runtime(db: Session, key: str) -> None:
"""Some knobs change how THIS process behaves, not just what a later read returns — they have to
take effect now, without a restart. One dispatch for both call sites (set and reset), so the next
such key can't be wired into one of them and forgotten in the other."""
if key == "log_level":
logs.set_level(sysconfig.get_str(db, "log_level"))
@router.get("")
def get_config(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> dict:
return sysconfig.describe(db)
@@ -80,6 +89,7 @@ def set_config(
after=None if s.secret else {"value": _redact(new)},
)
db.commit()
_apply_runtime(db, key)
return sysconfig.describe(db)
@@ -104,6 +114,7 @@ def reset_config(
before=None if s.secret else {"value": _redact(old)},
)
db.commit()
_apply_runtime(db, key)
return sysconfig.describe(db)
+179
View File
@@ -0,0 +1,179 @@
"""The browser's log, sent while its user has a raised diagnostics level (E22 S5).
The client keeps its log in memory (frontend `lib/log.ts`). Only after the user consented to Részletes
or Teljes in Settings → Diagnosztika does it stream that log here, in batches, and each batch lands in
`log_events` beside the server's own rows: `source="web"`, the account from the SESSION (never from
the body), and the request id each event carries, which is what joins a browser line to the server
line of the same request.
Nothing from the body is trusted as-is:
* only a JSON Content-Type is read, and the whole body is capped before it is parsed, and so are
the number of events and each event's attributes;
* the pace is capped per account, and so is what an account may store per hour;
* everything is redacted AGAIN with the server's own rules — the client redacts at capture, but a
modified client would not;
* an event below the consented level is dropped (a Részletes consent does not store what only
Teljes may carry), and so is the whole batch once the level has lapsed;
* the timestamp is clamped to the consent window, so a row cannot dodge the retention job by
claiming to come from the future.
"""
import json
from datetime import datetime, timedelta, timezone
from typing import Any, Literal
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel, Field, ValidationError
from sqlalchemy import insert
from sqlalchemy.orm import Session
from app import diagnostics, logs, logsink
from app.auth import current_user
from app.db import get_db
from app.models import LogEvent, User
from app.ratelimit import RateLimiter
router = APIRouter(prefix="/api/diagnostics", tags=["diagnostics"])
BODY_MAX_BYTES = 512 * 1024
EVENTS_MAX = 200
ATTRS_MAX_BYTES = 16 * 1024
# The client flushes every few seconds and when its buffer fills; this is room for a burst, not a
# pace anything legitimate keeps up for a minute.
_batches = RateLimiter(max_events=30, window_seconds=60)
# What one account may STORE per hour. The pace cap alone still lets a self-consented account write
# ~15 MiB a minute into the one shared database, kept for the whole retention window; a real session
# at Teljes stores a few hundred kB an hour. Past the budget a batch is accepted and not stored (a
# refusal would only make the client resend it), and one notice per hour says so.
STORED_BYTES_PER_HOUR = 5 * 1024 * 1024
_stored_bytes = RateLimiter(max_events=STORED_BYTES_PER_HOUR, window_seconds=3600)
_budget_notices = RateLimiter(max_events=1, window_seconds=3600)
ClientLevel = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
class ClientEvent(BaseModel):
ts: datetime
level: ClientLevel
event: str = Field(max_length=64, pattern=r"^[a-z][a-z0-9_]*(\.[a-z0-9_]+)*$")
msg: str = Field(max_length=8_000)
attrs: dict[str, Any] | None = None
request_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9._-]{8,64}$")
tab_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9_-]{1,32}$")
version: str | None = Field(default=None, max_length=64)
class ClientEventsIn(BaseModel):
events: list[ClientEvent] = Field(max_length=EVENTS_MAX)
async def _capped_body(request: Request) -> bytes:
"""The request body, refused the moment it passes BODY_MAX_BYTES while still arriving. Counted,
not taken from Content-Length: a declared length can be absent (chunked) or wrong, and counting
stops reading at the same point a declared-length check would have."""
body = bytearray()
async for chunk in request.stream():
body += chunk
if len(body) > BODY_MAX_BYTES:
raise HTTPException(status_code=413, detail={"code": "too_large"})
return bytes(body)
async def _parsed_events(request: Request) -> ClientEventsIn:
"""The body as the model — parsed only after the size cap, which is why this is a dependency and
not a plain body parameter (FastAPI reads a declared body in full before anything else runs).
JSON only, by its Content-Type, as a declared body would be. Parsing any body as JSON would let a
CORS-simple `text/plain` form post from a same-site page write events into a signed-in user's
log; the app's own client always sends `application/json`, the unload beacon included."""
media_type = request.headers.get("content-type", "").split(";")[0].strip().lower()
if media_type != "application/json" and not media_type.endswith("+json"):
raise HTTPException(status_code=415, detail={"code": "json_required"})
raw = await _capped_body(request)
try:
return ClientEventsIn.model_validate_json(raw)
except ValidationError as exc:
raise HTTPException(status_code=422, detail=json.loads(exc.json())) from exc
def _row(item: ClientEvent, user: User, now: datetime, window_start: datetime) -> dict[str, Any]:
attrs = logs.redact(item.attrs) if item.attrs else None
if attrs is not None:
size = len(json.dumps(attrs, ensure_ascii=False, default=str).encode())
if size > ATTRS_MAX_BYTES:
attrs = {"attrs_dropped_bytes": size}
if item.tab_id:
attrs = {**(attrs or {}), "tab_id": item.tab_id}
ts = item.ts if item.ts.tzinfo else item.ts.replace(tzinfo=timezone.utc)
return logsink.to_row(
{
"ts": min(max(ts, window_start), now),
"level": item.level,
"sev": logs.SEVERITY[item.level],
"source": "web",
"logger": "web",
"event": item.event,
"msg": logs.redact_text(item.msg),
"attrs": attrs,
"request_id": item.request_id,
"user_id": user.id,
"version": item.version,
}
)
@router.post("/events")
def ingest_events(
# The account first: dependencies resolve in order, so an anonymous caller is refused before a
# byte of its body is read.
user: User = Depends(current_user),
body: ClientEventsIn = Depends(_parsed_events),
db: Session = Depends(get_db),
) -> dict:
level = diagnostics.effective_level(user)
if level == diagnostics.BASE:
# Lapsed or withdrawn: the client stops streaming when it sees this.
raise HTTPException(status_code=409, detail={"code": "diagnostics_off"})
if not _batches.allow(str(user.id)):
raise HTTPException(status_code=429, detail={"code": "rate_limited"})
now = datetime.now(timezone.utc)
window_start = now - timedelta(hours=diagnostics.ELEVATION_HOURS)
floor = logs.SEVERITY[level]
rows = [
_row(item, user, now, window_start)
for item in body.events
if logs.SEVERITY[item.level] >= floor
]
if not rows:
return {"ok": True, "stored": 0}
size = sum(len(json.dumps(row, ensure_ascii=False, default=str).encode()) for row in rows)
stored = len(rows)
if not _stored_bytes.allow(str(user.id), size):
stored = 0
if not _budget_notices.allow(str(user.id)):
return {"ok": True, "stored": 0}
rows = [_budget_notice(user, now)]
# Under the erasure guard until the commit: an account erased meanwhile gets no rows back.
with logs.erasure_guard():
db.execute(insert(LogEvent), logsink.without_erased(rows))
db.commit()
return {"ok": True, "stored": stored}
def _budget_notice(user: User, now: datetime) -> dict[str, Any]:
return logsink.to_row(
{
"ts": now,
"level": "WARNING",
"sev": logs.SEVERITY["WARNING"],
"source": "web",
"logger": "web",
"event": "diagnostics.budget_spent",
# What the budget does, exactly: it is a SLIDING hour, and a refused batch is not
# charged to it, so smaller batches keep landing while there is room for them. Saying
# "the rest of the hour is not kept" would be a row contradicting the rows beneath it.
"msg": "The browser log passed its hourly storage budget; a batch bigger than what is left of it is not kept",
"attrs": {"budget_bytes": STORED_BYTES_PER_HOUR},
"user_id": user.id,
}
)
+35 -7
View File
@@ -107,6 +107,7 @@ def _serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
"phase": job.phase,
"error": job.error,
"error_code": job.error_code,
"error_detail": job.error_detail,
"queue_pos": job.queue_pos,
"created_at": iso(job.created_at),
"source_kind": job.source_kind,
@@ -172,6 +173,32 @@ def _own_job(db: Session, user: User, job_id: int) -> DownloadJob:
return job
def _own_job_for_update(db: Session, user: User, job_id: int) -> DownloadJob:
"""`_own_job` for a transition that moves its asset's ref count (resume, cancel, delete, redownload)
or writes the status those decide on (pause). The job is re-read under a row lock, so two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`service.finalize_if_running`), so the two can never deadlock."""
while True:
asset_id = _own_job(db, user, job_id).asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
raise HTTPException(status_code=404, detail="Unknown download")
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
# --- profiles ------------------------------------------------------------------------------
@@ -605,7 +632,7 @@ def update_download_meta(
def pause_download(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
job = _own_job(db, user, job_id)
job = _own_job_for_update(db, user, job_id)
service.pause(db, job)
return _serialize_job(db, job)
@@ -614,7 +641,7 @@ def pause_download(
def resume_download(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
job = _own_job(db, user, job_id)
job = _own_job_for_update(db, user, job_id)
service.resume(db, job)
return _serialize_job(db, job)
@@ -629,12 +656,13 @@ def redownload_download(
resolves to the newly-downloaded file — a re-download never breaks a link you already sent out
(only a *failed* re-download leaves the job in `error`, which is the intended broken-link case).
The rebuild always re-fetches into a FRESH, private asset (the cache sig is salted with the job
id + current asset id) rather than the dedup pool. That means it (a) always genuinely rebuilds
The rebuild always re-fetches into a FRESH, private asset (the cache sig is salted with the new
asset's own id, and the user's cookie salt when one applies — `credentials.rebuild_salt`) rather
than the dedup pool. That means it (a) always genuinely rebuilds
under the current format/naming rules — never silently reuses a shared cached file — and (b)
never deletes a file another job/user still shares: the old asset is only removed if THIS job
was its last holder."""
job = _own_job(db, user, job_id)
job = _own_job_for_update(db, user, job_id)
if job.job_kind in ("edit", "upload"):
# Neither has a re-fetchable source: an editor clip is derived locally, an upload IS the
# source. Re-running yt-dlp against source_kind='upload'/'edit' would just fail.
@@ -649,7 +677,7 @@ def redownload_download(
def cancel_download(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
job = _own_job(db, user, job_id)
job = _own_job_for_update(db, user, job_id)
service.cancel(db, job)
return _serialize_job(db, job)
@@ -658,7 +686,7 @@ def cancel_download(
def delete_download(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
job = _own_job(db, user, job_id)
job = _own_job_for_update(db, user, job_id)
service.delete(db, job)
return {"deleted": job_id}
+51 -12
View File
@@ -1,5 +1,7 @@
from typing import Any
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
from pydantic import BaseModel
from pydantic import BaseModel, RootModel
from sqlalchemy import func, select
from sqlalchemy.orm import Session
@@ -7,6 +9,7 @@ from app.auth import (
count_admins,
count_pending_activations,
current_user,
drop_from_wallet,
google_enabled,
has_read_scope,
has_write_scope,
@@ -14,7 +17,9 @@ from app.auth import (
optional_current_user,
purge_user,
)
from app import diagnostics
from app import email as email_mod
from app import logs
from app import sysconfig
from app.db import get_db
from app.models import Invite, User
@@ -65,24 +70,26 @@ def switch_account(
target = body.user_id
if target not in (request.session.get("account_ids") or []):
raise HTTPException(status_code=403, detail="Not an account you've signed into here.")
# An account `_session_user` would refuse — gone, deactivated, suspended, or its session revoked — is
# refused here too, and dropped from the switch list the way `_session_user` drops it. Made the session
# default instead, the next request's current_user would refuse it as the DEFAULT and clear the whole
# wallet session, logging out every account here. (B-core-9)
u = db.get(User, target)
if u is None:
drop_from_wallet(request.session, target)
raise HTTPException(status_code=404, detail="That account no longer exists.")
if not is_allowed(db, u.email):
raise HTTPException(status_code=403, detail="That account no longer has access.")
if not u.is_active:
drop_from_wallet(request.session, target)
raise HTTPException(status_code=403, detail="That account no longer has access.")
if u.is_suspended:
# Don't make a suspended account the active one — the next request's current_user would
# see the suspension and clear the WHOLE wallet session, logging out every account here.
drop_from_wallet(request.session, target)
raise HTTPException(status_code=403, detail="That account is suspended.")
# Same reasoning for a REVOKED session (a password reset/change or "log out everywhere" bumped the
# account's epoch): making it the session default would trip current_user's epoch check on the next
# request and clear the whole wallet. Mirror the suspend guard — drop the stale account from the
# switch list and refuse, rather than logging everyone out. (B-core-9)
# Revoked: a password reset/change or "log out everywhere" bumped the account's epoch.
epochs = request.session.get("epochs") or {}
if int(epochs.get(str(target), 0)) != (u.session_epoch or 0):
request.session["account_ids"] = [
a for a in (request.session.get("account_ids") or []) if a != target
]
drop_from_wallet(request.session, target)
raise HTTPException(
status_code=403, detail="That account was signed out elsewhere — sign in again."
)
@@ -144,6 +151,9 @@ def get_me(
"pending_activations": pending_activations,
"email_configured": email_configured,
"preferences": user.preferences or {},
# The consented diagnostics level in force and when it lapses (E22 S5). The client applies
# it to its own log and streams that log while it is raised.
"diagnostics": diagnostics.state(db, user),
}
@@ -185,16 +195,45 @@ def delete_my_account(
return {"deleted": True}
class PreferencesIn(RootModel[dict[str, Any]]):
"""Free-form UI preferences — a JSON object of any keys (theme, filters, …), merged into the
stored ones. A model rather than a bare `dict` parameter so a non-object body is a 422 at the
boundary like every other request body."""
@router.put("/preferences")
def update_preferences(
preferences: dict,
preferences: PreferencesIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
# Merge so partial updates don't wipe other keys.
merged = dict(user.preferences or {})
merged.update(preferences)
merged.update(preferences.root)
user.preferences = merged
db.add(user)
db.commit()
return {"preferences": merged}
class DiagnosticsIn(BaseModel):
level: diagnostics.DiagnosticsLevel
@router.put("/diagnostics")
def update_diagnostics(
body: DiagnosticsIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Raise the diagnostics level (the user consented in Settings → Diagnosztika) or return to the
base. The expiry is the server's: a raised level always lapses after
`diagnostics.ELEVATION_HOURS`, whatever the client asks. The shared demo account cannot raise
it — its log would interleave every visitor's session under one account."""
if user.is_demo and body.level != diagnostics.BASE:
raise HTTPException(status_code=403, detail={"code": "demo", "reason": "diagnostics"})
diagnostics.set_level(user, body.level)
db.commit()
# The rest of THIS request, and the process's elevation registry, follow the new level now.
logs.set_user(user.id, diagnostics.effective_level(user), user.diagnostics_until)
return {"ok": True, "diagnostics": diagnostics.state(db, user)}
+5 -4
View File
@@ -46,7 +46,7 @@ WELCOME = {
"This is your private message inbox. Conversations with other members are end-to-end "
"encrypted — only you and the person you're chatting with can read them, not even an "
"admin. To start messaging someone, set up secure messaging with a passphrase first.\n\n"
"Enjoy your feed!"
"Enjoy your Feed!"
),
"hu": (
"Üdv a Siftlode-on! 👋\n\n"
@@ -423,9 +423,10 @@ async def messages_ws(ws: WebSocket) -> None:
"""Live push channel: authenticated by the session cookie, registers the connection so
sent messages reach this user's open tabs instantly. We don't consume client→server frames
(sending goes through POST /api/messages); the receive loop only detects disconnect."""
# Which signed-in account this socket acts as — the SHARED per-tab resolution (wallet-gated
# ?account= override, session default otherwise). A browser WebSocket can't send the
# X-Siftlode-Account header, so resolved_user_id falls through to the ?account= query param.
# Which signed-in account this socket acts as — the SHARED per-tab resolution (a wallet-gated
# ?account=, the session default when none is named, nobody for an account no longer signed in
# here). A browser WebSocket can't send the X-Siftlode-Account header, so resolved_user_id falls
# through to the ?account= query param.
uid, _ = resolved_user_id(ws)
if not uid:
await ws.close(code=1008)
+20 -6
View File
@@ -6,7 +6,7 @@ the app or any account data. Password-protected links exchange the password once
a short-lived signed grant that the media URL carries (a `<video src>` can't send a header).
"""
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import select
@@ -17,13 +17,21 @@ from app.db import get_db
from app.downloads import links as linksmod
from app.downloads import storage
from app.models import DownloadJob, DownloadLink, MediaAsset
from app.ratelimit import RateLimiter
from app.ratelimit import RateLimiter, client_key
from app.security import verify_password
router = APIRouter(prefix="/api/public", tags=["public"])
# Slow password guessing per link (the token itself is 192-bit, so it needs no throttle).
# Slow password guessing on a link (the token itself is 192-bit, so it needs no throttle). Counted per
# viewer — the link AND the client (`client_key`) — so a client guessing wrong shuts out only itself, and a
# busy link's own unlocks don't count against its next viewer.
_unlock_limit = RateLimiter(max_events=10, window_seconds=300)
# ...under a ceiling on WRONG passwords for the link from every client together, so guessing spread over
# many addresses stays slow too; unlocking with the password never adds to it. The price of the ceiling:
# someone with the link but not its password, guessing from 50 / 10 = 5 networks, can keep the link shut
# for everyone until the window passes. It is checked before the password and counted after it, so guesses
# already running when it fills are still checked — as many as the route's threadpool runs at once.
_unlock_failures = RateLimiter(max_events=50, window_seconds=300)
def _link_or_404(db: Session, token: str) -> DownloadLink:
@@ -83,11 +91,16 @@ class WatchUnlockIn(BaseModel):
@router.post("/watch/{token}/unlock")
def watch_unlock(token: str, body: WatchUnlockIn, db: Session = Depends(get_db)) -> dict:
def watch_unlock(
token: str, body: WatchUnlockIn, request: Request, db: Session = Depends(get_db)
) -> dict:
"""Exchange the link password for a signed grant carried by the media URL."""
if not _unlock_limit.allow(token):
raise HTTPException(status_code=429, detail="Too many attempts. Try again later.")
# The link first: a token naming none must leave no limiter entry behind, or posting made-up tokens
# would grow the limiter without bound.
link = _link_or_404(db, token)
viewer = f"{token}|{client_key(request)}"
if _unlock_failures.exhausted(token) or not _unlock_limit.allow(viewer):
raise HTTPException(status_code=429, detail="Too many attempts. Try again later.")
if not link.password_hash:
# Not a password link — nothing to unlock; just hand back the plain meta.
job, asset = _ready_target(db, link)
@@ -96,6 +109,7 @@ def watch_unlock(token: str, body: WatchUnlockIn, db: Session = Depends(get_db))
**linksmod.public_meta(link, job, asset, _file_url(token), _poster_url(token, asset)),
}
if not verify_password(body.password, link.password_hash):
_unlock_failures.allow(token)
raise HTTPException(status_code=403, detail="Wrong password.")
job, asset = _ready_target(db, link)
link.view_count += 1
+10 -1
View File
@@ -22,6 +22,7 @@ from app.scheduler import (
trigger_all,
trigger_job,
)
from app.sync.explore import followed_channel
from app.sync.runner import estimate_deep_backfill
from app.routes._common import not_found
@@ -137,9 +138,17 @@ def get_scheduler(_: User = Depends(admin_user), db: Session = Depends(get_db))
queue = {
"channels_recent_pending": db.scalar(
select(func.count()).select_from(Channel).where(Channel.recent_synced_at.is_(None))
# The same selection run_recent_backfill works through, or the card counts channels
# the job will never touch and never reaches zero.
select(func.count())
.select_from(Channel)
.where(Channel.recent_synced_at.is_(None), followed_channel())
)
or 0,
# The whole opted-in BACKLOG, deliberately not `run_deep_backfill`'s next-pass selection
# (which also wants `recent_synced_at`): the same figure /api/sync/my-status shows the user,
# and a channel missing its recent pull is one the same job tick fetches first, not one the
# job will never touch. Narrowing it here would under-report work that is genuinely queued.
"channels_deep_pending": deep_eta["channels_pending"],
"deep_eta_seconds": deep_eta["eta_seconds"],
"videos_pending_enrich": db.scalar(
+1 -3
View File
@@ -231,9 +231,7 @@ def search_youtube(
limit = max(1, min(limit, 100))
# Source: "scrape" (InnerTube, no API quota) or "api" (search.list, 100 units/page).
source = (sysconfig.get_str(db, "search_source") or "scrape").lower()
if source != "scrape":
source = "api"
source = sysconfig.get_str(db, "search_source")
# Per-user daily cap applies to both sources (anti-abuse); with scrape it's the only guard
# since the search itself spends no quota.
+16 -1
View File
@@ -10,7 +10,7 @@ from typing import Callable
from apscheduler.schedulers.background import BackgroundScheduler
from sqlalchemy import select
from app import audit, progress, quota
from app import audit, logsink, progress, quota
from app.audit import AuditAction
from app.config import settings
from app.db import SessionLocal
@@ -72,6 +72,7 @@ JOB_INTERVALS: dict[str, int] = {
"plex_watch_reconcile": settings.plex_watch_reconcile_interval_min,
"tmdb_enrich": settings.tmdb_enrich_interval_min,
"audit_gc": settings.audit_gc_minutes,
"log_gc": settings.log_gc_minutes,
}
# Sane bounds for an admin-set interval (minutes).
@@ -381,6 +382,19 @@ def _audit_gc_job() -> None:
_job("audit_gc", work)
def _log_gc_job() -> None:
# Prune log_events rows older than log_retention_days — the limit the privacy policy states, so
# this runs hourly rather than daily. Pure DB, no quota.
def work(db):
from app import sysconfig
reaped = logsink.gc(db, sysconfig.get_int(db, "log_retention_days"))
db.commit()
return {"reaped": reaped}
_job("log_gc", work)
# job_id -> wrapper. The single source of truth for which jobs exist and how to run one,
# shared by start_scheduler (recurring registration) and trigger_job (manual "run now").
JOB_FUNCS: dict[str, Callable[[], None]] = {
@@ -400,6 +414,7 @@ JOB_FUNCS: dict[str, Callable[[], None]] = {
"plex_watch_reconcile": _plex_watch_reconcile_job,
"tmdb_enrich": _tmdb_enrich_job,
"audit_gc": _audit_gc_job,
"log_gc": _log_gc_job,
}
+19 -6
View File
@@ -11,7 +11,7 @@ grace period (mirrors the maintenance job's grace-then-delete safety)."""
import logging
from datetime import timedelta
from sqlalchemy import delete, select
from sqlalchemy import delete, or_, select
from sqlalchemy.orm import Session
from app import sysconfig
@@ -31,6 +31,23 @@ from app.youtube.client import YouTubeClient
log = logging.getLogger("siftlode.explore")
def followed_channel():
"""WHERE clause: a channel somebody follows — a subscription, or an exploration still inside its
grace window (`purge_unkept_explored` deletes the expired ExploredChannel rows first). ONE
definition for both halves of the question: what the scheduled RSS poll and recent backfill cover
(sync/runner.py) and what the explore purge may delete.
The poll and backfill cover only these (owner decision, 2026-09-17). They used to cover EVERY
channel row, but rows also come from search results, playlist sync and dropped subscriptions, and
nothing deletes those: on prod 582 of 979 polled channels had no follower, three of them dead
feeds answering 404 on every poll. The cost of narrowing is deliberate: the shared Library stops
receiving new uploads from channels nobody follows."""
return or_(
select(Subscription.id).where(Subscription.channel_id == Channel.id).exists(),
select(ExploredChannel.id).where(ExploredChannel.channel_id == Channel.id).exists(),
)
def _enrich_new(db: Session, yt: YouTubeClient, video_ids: list[str]) -> int:
"""Enrich just the not-yet-enriched videos among `video_ids` (videos.list, 1 unit/50)."""
if not video_ids:
@@ -98,12 +115,8 @@ def purge_unkept_explored(db: Session) -> dict:
).rowcount
db.commit()
sub_exists = select(Subscription.id).where(Subscription.channel_id == Channel.id).exists()
exp_exists = select(ExploredChannel.id).where(ExploredChannel.channel_id == Channel.id).exists()
orphan_ids = (
db.execute(
select(Channel.id).where(Channel.from_explore.is_(True), ~sub_exists, ~exp_exists)
)
db.execute(select(Channel.id).where(Channel.from_explore.is_(True), ~followed_channel()))
.scalars()
.all()
)
+59 -17
View File
@@ -7,16 +7,18 @@ public reads.
import logging
import threading
from collections import Counter
from concurrent.futures import ThreadPoolExecutor, as_completed
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import progress, quota, sysconfig
from app import logs, progress, quota, sysconfig
log = logging.getLogger("siftlode.sync")
from app.db import SessionLocal
from app.models import Channel, OAuthToken, Subscription, User
from app.sync.explore import followed_channel
from app.sync.subscriptions import import_subscriptions
from app.sync.videos import (
apply_rss_feed,
@@ -32,6 +34,8 @@ from app.youtube.rss import RssError, fetch_channel_feed
# RSS feeds are free and network-bound, so fetch many at once; DB writes stay single-threaded.
RSS_POLL_WORKERS = 16
# Failed channels named in a poll's summary warning; the rest are counted by reason.
RSS_FAILURE_EXAMPLES = 5
def token_users(db: Session) -> list[User]:
@@ -52,7 +56,7 @@ def get_service_user(db: Session) -> User | None:
def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str, int]:
"""Poll every channel's free RSS feed. Returns `{"new": inserted, "failed": channels_that_did
"""Poll the free RSS feed of every followed channel (`followed_channel`). Returns `{"new": inserted, "failed": channels_that_did
_not_complete}` — the failure count is part of the RESULT, not just a log line, so the scheduler
card and the audit summary can tell "no new uploads" apart from "nothing worked" (an egress
outage, or a database that can't accept writes, otherwise renders identically to a quiet hour).
@@ -67,7 +71,7 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
carries the FIRST underlying error, because on a small instance "all channels failed" is also
what a single dead feed id looks like."""
if channels is None:
channels = db.execute(select(Channel)).scalars().all()
channels = db.execute(select(Channel).where(followed_channel())).scalars().all()
total = len(channels)
new = 0
done = 0
@@ -76,6 +80,27 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
# outage from one permanently dead feed id — which is the whole difference on an instance with
# a single subscription, where BOTH render as 100% failed.
first_error: str | None = None
# What failed, as counts per reason (`http_404`, `transport`, `apply`, …) plus a few examples —
# the ONE warning a poll writes. One line per failed channel was what turned YouTube's RSS outage
# of 2026-09-16 into 11k warnings in four hours; each channel's own line is DEBUG now.
reasons: Counter[str] = Counter()
examples: list[str] = []
def fail(reason: str, detail: str, *, unexpected: str | None = None) -> None:
"""Count a channel that did not get polled. `unexpected` is the message for a failure that
is a bug on our side: its FIRST occurrence per reason gets a traceback at ERROR, the rest
DEBUG — a read-only database fails every channel the same way, and 979 identical tracebacks
tell nobody anything the first one didn't. Call it from inside the `except` block."""
nonlocal failed, first_error
if unexpected is not None:
level = logging.ERROR if reasons[reason] == 0 else logging.DEBUG
log.log(level, unexpected, exc_info=True)
failed += 1
reasons[reason] += 1
first_error = first_error or detail
if len(examples) < RSS_FAILURE_EXAMPLES:
examples.append(detail)
# Fetch feeds concurrently (network-bound, no DB), then apply inserts on this thread as
# each fetch returns — a SQLAlchemy session isn't thread-safe. Pass the channel id (a
# plain str) to the workers; the ORM instance is touched only here on the main thread.
@@ -90,16 +115,15 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
# A failed poll is NOT an empty feed: skip the apply so `last_rss_at` stays where it
# was (stamping it made a permanently-broken feed look freshly polled). The shared
# counter/progress tail lives after the statement, so the two outcomes can stay two
# clauses: an RssError is EXPECTED (404, timeout) and gets one warning line; anything
# else is a bug on our side and earns a traceback.
# clauses: an RssError is EXPECTED (404, timeout) and is only counted; anything else is a
# bug on our side and earns a traceback (the first of its kind per poll).
except RssError as exc:
log.warning("RSS poll skipped for channel %s: %s", channel.id, exc)
failed += 1
first_error = first_error or f"{channel.id}: {exc}"
log.debug("RSS poll skipped for channel %s: %s", channel.id, exc)
fail(exc.reason, f"{channel.id}: {exc}")
except Exception as exc:
log.exception("RSS fetch failed for channel %s", channel.id)
failed += 1
first_error = first_error or f"{channel.id}: {exc}"
fail(
"error", f"{channel.id}: {exc}", unexpected=f"RSS fetch failed for {channel.id}"
)
else:
try:
new += apply_rss_feed(db, channel, entries)
@@ -107,14 +131,27 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
# A write failure (read-only DB, failover) leaves the channel just as un-polled
# as an unreachable feed — count it, or a wholly broken run reports "failed=0".
db.rollback()
failed += 1
first_error = first_error or f"{channel.id} (apply): {exc}"
log.exception("RSS apply failed for channel %s", channel.id)
fail(
"apply",
f"{channel.id} (apply): {exc}",
unexpected=f"RSS apply failed for {channel.id}",
)
# One place that advances the counter, whatever happened above.
done += 1
progress.report(done, total, "rss_poll")
if failed:
log.warning("RSS poll: %d/%d channels could not be polled", failed, total)
log.warning(
"RSS poll: %d/%d channels could not be polled",
failed,
total,
extra=logs.event(
"sync.rss.failed",
failed=failed,
total=total,
reasons=dict(reasons),
examples=examples,
),
)
if total and failed == total:
# Nothing was polled at all — an egress outage (the tinyproxy-after-reboot incident), a
# dead DNS, a read-only database. Returning normally would record the run as `ok` with the
@@ -174,7 +211,11 @@ def run_recent_backfill(
return {"channels_processed": 0, "videos_added": 0, "reason": "no service user"}
if channels is None:
channels = (
db.execute(select(Channel).where(Channel.recent_synced_at.is_(None))).scalars().all()
db.execute(
select(Channel).where(Channel.recent_synced_at.is_(None), followed_channel())
)
.scalars()
.all()
)
if max_channels:
channels = channels[:max_channels]
@@ -204,7 +245,8 @@ def run_deep_backfill(db: Session, max_channels: int = 10, max_pages: int = 5) -
if user is None:
return {"channels_processed": 0, "videos_added": 0, "reason": "no service user"}
# Demand-driven: only deep-backfill channels at least one user has opted into
# (Subscription.deep_requested). Recent backfill stays unconditional (cheap).
# (Subscription.deep_requested). Recent backfill needs no opt-in: it covers every followed channel
# (`followed_channel`).
deep_requested = (
select(Subscription.id)
.where(
+48 -19
View File
@@ -14,7 +14,7 @@ from dataclasses import dataclass
from sqlalchemy.orm import Session
from app import security
from app.config import settings
from app.config import Settings, canonical_choice, choices_of, settings
from app.models import SystemConfig
log = logging.getLogger("siftlode.config")
@@ -27,7 +27,7 @@ _decrypt_warned: set[str] = set()
@dataclass(frozen=True)
class ConfigSpec:
key: str
type: str # "int" | "str" | "bool"
type: str # "int" | "str" | "bool" | "choice"
group: str # admin-UI grouping (logically related keys stay together)
default_attr: str # attribute on `settings` holding the env/config default
secret: bool = False
@@ -35,16 +35,33 @@ class ConfigSpec:
max: int | None = None
allow_empty: bool = False # str keys: a stored "" is an explicit empty value, not "unset"
@property
def choices(self) -> tuple[str, ...]:
"""A choice key's allowed values, read from its setting's Literal type in config.py — the one
place the list is written. The admin page renders them as a picker."""
return choices_of(Settings.model_fields[self.default_attr].annotation)
# Registry of DB-overridable keys. Add a key here + wire its read through get_*() to move it
# off env into the admin UI; the admin page renders it automatically from this list.
# off env into the admin UI; the admin page renders it automatically from this list. The ORDER is the
# admin page's tab order (first appearance of each group), so keep a group's keys together.
SPECS: tuple[ConfigSpec, ...] = (
# --- Access / registration ---
ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
# --- Email / SMTP (one logical group; the password is the only secret) ---
ConfigSpec("smtp_host", "str", "email", "smtp_host"),
ConfigSpec("smtp_port", "int", "email", "smtp_port", min=1, max=65535),
ConfigSpec("smtp_user", "str", "email", "smtp_user", allow_empty=True),
ConfigSpec("smtp_from", "str", "email", "smtp_from"),
ConfigSpec("smtp_password", "str", "email", "smtp_password", secret=True),
# --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
# Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
# --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
# install wizard / Configuration page so no restart is needed when the creds change. ---
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
# --- Quota (shared YouTube Data API daily budget) ---
ConfigSpec("quota_daily_budget", "int", "quota", "quota_daily_budget", min=1, max=1_000_000),
ConfigSpec(
@@ -59,7 +76,7 @@ SPECS: tuple[ConfigSpec, ...] = (
max=100_000,
),
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
ConfigSpec("search_source", "str", "quota", "search_source"),
ConfigSpec("search_source", "choice", "quota", "search_source"),
# --- Backfill (recent-first first pass per channel) ---
ConfigSpec(
"backfill_recent_max_videos",
@@ -96,16 +113,6 @@ SPECS: tuple[ConfigSpec, ...] = (
ConfigSpec("explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650),
# Days an un-kept live-search result video is kept before the cleanup job removes it.
ConfigSpec("search_grace_days", "int", "explore", "search_grace_days", min=0, max=3_650),
# --- YouTube Data API key (secret; optional — public reads prefer it over OAuth) ---
ConfigSpec("youtube_api_key", "str", "youtube", "youtube_api_key", secret=True),
# Optional egress proxy for YouTube API calls (fixed-IP host for an IP-restricted key).
ConfigSpec("youtube_api_proxy", "str", "youtube", "youtube_api_proxy", allow_empty=True),
# --- Google OAuth client (Sign in with Google; both encrypted, env fallback). Set from the
# install wizard / Configuration page so no restart is needed when the creds change. ---
ConfigSpec("google_client_id", "str", "google", "google_client_id", secret=True),
ConfigSpec("google_client_secret", "str", "google", "google_client_secret", secret=True),
# --- Access / registration ---
ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
# --- Download center (yt-dlp) — per-user defaults + retention/GC + layout ---
ConfigSpec("download_total_max_bytes", "int", "downloads", "download_total_max_bytes", min=0),
ConfigSpec(
@@ -137,7 +144,7 @@ SPECS: tuple[ConfigSpec, ...] = (
ConfigSpec(
"download_gc_grace_days", "int", "downloads", "download_gc_grace_days", min=0, max=3_650
),
ConfigSpec("download_layout", "str", "downloads", "download_layout"),
ConfigSpec("download_layout", "choice", "downloads", "download_layout"),
ConfigSpec(
"download_worker_concurrency",
"int",
@@ -150,6 +157,13 @@ SPECS: tuple[ConfigSpec, ...] = (
# --- Audit log ---
# Days to keep admin audit-log rows before the audit_gc job prunes them (0 = keep forever).
ConfigSpec("audit_retention_days", "int", "audit", "audit_retention_days", min=0, max=3650),
# --- Diagnostics (E22) ---
# Runtime log level for the API process. Applied without a restart — see routes/config.py and
# main.py's lifespan.
ConfigSpec("log_level", "choice", "diagnostics", "log_level"),
# Days the log_events table keeps a row (the log_gc job prunes). No "0 = forever": stored logs
# carry account-linked request history, and the privacy policy names a limit.
ConfigSpec("log_retention_days", "int", "diagnostics", "log_retention_days", min=1, max=90),
# --- Plex integration (optional; local-file playback + Plex metadata) ---
ConfigSpec("plex_enabled", "bool", "plex", "plex_enabled"),
ConfigSpec("plex_server_url", "str", "plex", "plex_server_url"),
@@ -159,10 +173,10 @@ SPECS: tuple[ConfigSpec, ...] = (
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
# (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
# Transcode re-encode tuning (E6 S1). hw = "auto"|"on"|"off" (free-text, like search_source); the
# libx264 knobs apply only to the software fallback. crf 0–51 (x264 range); max_height 0 = source.
ConfigSpec("plex_transcode_hw", "str", "plex", "plex_transcode_hw"),
ConfigSpec("plex_transcode_preset", "str", "plex", "plex_transcode_preset"),
# Transcode re-encode tuning (E6 S1). The libx264 knobs apply only to the software fallback.
# crf 0–51 (x264 range); max_height 0 = source.
ConfigSpec("plex_transcode_hw", "choice", "plex", "plex_transcode_hw"),
ConfigSpec("plex_transcode_preset", "choice", "plex", "plex_transcode_preset"),
ConfigSpec("plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51),
ConfigSpec(
"plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
@@ -194,11 +208,23 @@ def _default(s: ConfigSpec):
return getattr(settings, s.default_attr)
def _choice(s: ConfigSpec, raw: object) -> str:
"""The canonical spelling of `raw` among the key's choices (any letter case), or ValueError."""
option = canonical_choice(s.choices, raw)
if option is None:
raise ValueError(f"{s.key} must be one of: {', '.join(s.choices)}")
return option
def _coerce(s: ConfigSpec, raw: str):
if s.type == "int":
return int(raw)
if s.type == "bool":
return raw == "true"
if s.type == "choice":
# A stored value that is no longer a choice (the list shrank) reads as the default, like a
# non-numeric int override does — see get().
return _choice(s, raw)
return raw
@@ -274,6 +300,8 @@ def set_value(db: Session, key: str, value) -> None:
raw = str(v)
elif s.type == "bool":
raw = "true" if value else "false"
elif s.type == "choice":
raw = _choice(s, value)
else:
raw = "" if value is None else str(value)
@@ -316,6 +344,7 @@ def describe(db: Session) -> dict:
"max": s.max,
"is_set": is_set, # an override row exists
"allow_empty": s.allow_empty, # if true, the UI stores "" rather than deleting the row
"choices": list(s.choices), # the picker's options; [] for any other type
}
if s.secret:
item["default_is_set"] = bool(_default(s))
+60 -15
View File
@@ -28,16 +28,18 @@ from datetime import datetime, timezone
from sqlalchemy import text
from app import logs
from app import logsink
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import quota, service
from app.downloads import credentials, formats, quota, service
from app.downloads import errors as dlerrors
from app.downloads.jobdb import (
_edit_staging_dir,
_requeue_if_running,
_reset_asset_pending,
_set_job,
_staging_dir,
)
from app.downloads.status import STOPPED_BY_ROUTE, JobStatus
@@ -253,7 +255,27 @@ def _fail_after_kill(
"""`_stop_and_free` + mark the job errored — for the watchdog's stall/timeout kills and the
supervisor-error fallback."""
_stop_and_free(job_id, asset_id, proc, grace_s)
_set_job(job_id, status="error", error=error, error_code=error_code, phase=None)
_fail_running(job_id, error=error, error_code=error_code)
def _fail_running(
job_id: int, *, error: str, error_code: str | None, error_detail: str | None = None
) -> None:
"""Fail a claimed job — only while it is still `running` (`service.finalize_if_running`), so a
cancel/pause a route landed first is never turned back into `error`, which holds the asset the
route already released. `error_detail` is written every time: a failure never shows an earlier
failure's explanation."""
with SessionLocal() as db:
service.finalize_if_running(
db,
job_id,
status=JobStatus.ERROR,
error=error,
error_code=error_code,
error_detail=error_detail,
phase=None,
)
db.commit()
def _run_supervised(job_id: int, asset_id: int) -> None:
@@ -329,12 +351,10 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
status, _ = _job_liveness(job_id)
if status == "running":
_cleanup_staging(asset_id)
_set_job(
_fail_running(
job_id,
status="error",
error_code="unknown",
phase=None,
error=f"The download process exited unexpectedly (code {proc.returncode}).",
error_code="unknown",
)
log.warning(
"job %s: child exited %s without finishing", job_id, proc.returncode
@@ -399,16 +419,32 @@ def _process_job(job_id: int) -> None:
asset_id = asset.id if asset else None
asset_status = asset.status if asset else None
asset_error = asset.error if asset else None
if asset_status == "error":
# A job that meets an already-failed asset gets that failure's explanation — not a bare
# "Download failed". Only the text is on the asset, so it is diagnosed again, for this
# link, with whether a cookie goes with it asked of the same gate the download child
# used (`credentials.usable_cookie`) — as it stands now, so the advice fits this job's Retry.
url = service.source_url(job.source_kind, job.source_ref)
cookie = credentials.usable_cookie(
db, job.user_id, url, asset, formats.normalize(job.profile_snapshot)
)
diagnosis = dlerrors.diagnose(asset_error, url=url, cookie_attached=cookie is not None)
del cookie # only whether there is one matters here
if asset is None:
_set_job(job_id, status="error", error="No media asset", phase=None)
_fail_running(job_id, error="No media asset", error_code=None)
return
if asset_status == "ready":
_finish_from_cache(job_id, asset_id)
return
if asset_status == "error":
_set_job(job_id, status="error", error=asset_error or "Download failed", phase=None)
_fail_running(
job_id,
error=asset_error or "Download failed",
error_code=diagnosis.code,
error_detail=diagnosis.detail,
)
return
if asset_status == "downloading":
# Another worker won this asset between our claim and here (a narrow race — the claim SQL
@@ -457,15 +493,20 @@ def _worker_loop(idx: int) -> None:
if job_id is None:
_stop.wait(settings.download_poll_seconds)
continue
try:
_process_job(job_id)
except Exception: # noqa: BLE001
log.exception("processing job %s crashed", job_id)
_set_job(job_id, status="error", error="Worker error", phase=None)
# Tag every line this job produces (including the crash below) with the job id: a worker runs
# N jobs at once on N threads, so without it the interleaved output is unreadable. The bind
# wraps the handler too — bound inside the `try`, it was already reset by the time the crash
# line was logged.
with logs.bind(job_id=job_id):
try:
_process_job(job_id)
except Exception: # noqa: BLE001
log.exception("processing job %s crashed", job_id)
_fail_running(job_id, error="Worker error", error_code=None)
def main() -> None:
logging.basicConfig(level=logging.INFO)
logs.configure_logging("worker")
signal.signal(signal.SIGTERM, _handle_signal)
signal.signal(signal.SIGINT, _handle_signal)
@@ -481,6 +522,10 @@ def main() -> None:
if _stop.is_set():
return # asked to shut down while waiting — that IS a clean exit
raise SystemExit(1)
# After the wait, not before: the DB is reachable now. The wait checks `download_jobs`, not
# `log_events`, so on the deploy that first adds the table the worker may start before the API
# has migrated — those few seconds of batches fail and say so on stderr; nothing else breaks.
logsink.start()
_recover_orphans()
n = max(1, settings.download_worker_concurrency)
log.info("Download worker started (concurrency=%d, root=%s).", n, settings.download_root)
+13 -3
View File
@@ -14,7 +14,14 @@ class RssError(Exception):
Deliberately NOT the same as an empty feed: a failed poll used to return `[]`, which the
caller stamped as a successful `last_rss_at` — so a channel whose feed 404s permanently
(deleted, renamed id) looked like it was being polled fine forever, and nothing surfaced it."""
(deleted, renamed id) looked like it was being polled fine forever, and nothing surfaced it.
`reason` is a short stable code (`http_404`, `transport`) so a poll over many channels can report
WHAT failed as counts instead of one line per channel."""
def __init__(self, message: str, reason: str):
super().__init__(message)
self.reason = reason
def fetch_channel_feed(channel_id: str) -> list[dict]:
@@ -24,9 +31,12 @@ def fetch_channel_feed(channel_id: str) -> list[dict]:
try:
resp = httpx.get(url, timeout=20.0, headers={"User-Agent": "Siftlode/1.0"})
except httpx.HTTPError as exc:
raise RssError(f"RSS fetch failed for {channel_id}: {exc}") from exc
raise RssError(f"RSS fetch failed for {channel_id}: {exc}", "transport") from exc
if resp.status_code != 200:
raise RssError(f"RSS fetch for {channel_id} returned HTTP {resp.status_code}")
raise RssError(
f"RSS fetch for {channel_id} returned HTTP {resp.status_code}",
f"http_{resp.status_code}",
)
parsed = feedparser.parse(resp.content)
out: list[dict] = []
+6 -2
View File
@@ -11,8 +11,12 @@ echo "Starting Siftlode API..."
# on the next request -> 502. Non-idempotent POSTs (the player's progress/state writes,
# fired every 5s) can't be auto-retried by the proxy, so those 502s reached the user.
#
# SECURITY: do NOT add --proxy-headers / --forwarded-allow-ips here. auth._client_ip trusts
# SECURITY: do NOT add --proxy-headers / --forwarded-allow-ips here. ratelimit.client_key trusts
# X-Forwarded-For only when the TRUE socket peer is a configured proxy (TRUSTED_PROXY_IPS); those
# flags would make uvicorn overwrite request.client from the forgeable XFF, defeating that check and
# reopening the rate-limit bypass (SA3).
exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --log-config log_config.json --timeout-keep-alive 75
#
# No --log-config: app/logs.py configures logging for every Siftlode process (api, worker, download
# child, alembic) in one place, and it silences uvicorn's access log in favour of our own
# `http.request` line (route template, no query string, request id).
exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --timeout-keep-alive 75
-40
View File
@@ -1,40 +0,0 @@
{
"version": 1,
"disable_existing_loggers": false,
"formatters": {
"default": {
"()": "uvicorn.logging.DefaultFormatter",
"fmt": "%(asctime)s %(levelprefix)s [%(name)s] %(message)s"
},
"access": {
"()": "uvicorn.logging.AccessFormatter",
"fmt": "%(asctime)s %(levelprefix)s %(client_addr)s - \"%(request_line)s\" %(status_code)s"
},
"plain": {
"format": "%(asctime)s %(levelname)-5s [%(name)s] %(message)s"
}
},
"handlers": {
"default": {
"class": "logging.StreamHandler",
"formatter": "default",
"stream": "ext://sys.stdout"
},
"access": {
"class": "logging.StreamHandler",
"formatter": "access",
"stream": "ext://sys.stdout"
},
"plain": {
"class": "logging.StreamHandler",
"formatter": "plain",
"stream": "ext://sys.stdout"
}
},
"loggers": {
"uvicorn": { "handlers": ["default"], "level": "INFO", "propagate": false },
"uvicorn.error": { "handlers": ["default"], "level": "INFO", "propagate": false },
"uvicorn.access": { "handlers": ["access"], "level": "INFO", "propagate": false },
"siftlode": { "handlers": ["plain"], "level": "INFO", "propagate": false }
}
}
+4 -2
View File
@@ -1,7 +1,9 @@
# Test + lint tooling — NEVER in the prod image. Installed only by backend/Dockerfile.test (the
# throwaway test image), which the prod build (root Dockerfile) never references. Pinned so the
# suite is reproducible: pytest for the pure-logic tests, and ruff at the SAME version the gate's
# backend lint lane currently runs on the host (see the run_lint note in siftlode.sh). httpx is
# already an app dependency (requirements.txt), so its TestClient needs nothing extra here.
# backend lint lane runs on the host (the gate fails when the host's ruff is not this one). httpx2:
# Starlette 1.x's TestClient is built on it (`import httpx2 as httpx`) and only falls back to the
# app's own httpx with a deprecation warning on every run.
pytest==8.4.2
ruff==0.15.21
httpx2==2.13.0
+5
View File
@@ -7,6 +7,7 @@ Authlib==1.7.2
Mako==1.4.1
MarkupSafe==3.0.3
PyYAML==6.0.3
Pygments==2.21.0
SQLAlchemy==2.0.52
alembic==1.19.1
annotated-doc==0.0.5
@@ -21,6 +22,7 @@ cffi==2.1.1
charset-normalizer==3.5.1
click==8.4.2
cryptography==50.0.0
curl_cffi==0.15.0
fastapi==0.141.1
feedparser-sgmllib==2.1.0
feedparser==6.0.14
@@ -32,6 +34,8 @@ httpx==0.28.1
idna==3.18
itsdangerous==2.2.0
joserfc==1.7.4
markdown-it-py==4.2.0
mdurl==0.1.2
mutagen==1.48.1
numpy==2.5.2
psycopg-binary==3.3.4
@@ -45,6 +49,7 @@ pydantic_core==2.46.4
python-dotenv==1.2.3
python-multipart==0.0.32
requests==2.34.2
rich==15.0.0
starlette==1.6.0
typing-inspection==0.4.4
typing_extensions==4.16.0
+7
View File
@@ -21,6 +21,13 @@ argon2-cffi>=23.1,<26
# scripts Deno runs for YouTube's n-signature). yt-dlp is release-often (YouTube changes), so
# keep the floor loose and bump it when extraction breaks.
yt-dlp[default]>=2025.5.22
# Browser impersonation (`impersonate`) for Facebook downloads made with the user's cookie: some
# links answer a signed-in plain request with "Cannot parse data" and only parse as Chrome (E11.5,
# measured). The range is YT-DLP's, not ours — yt-dlp 2026.07.04 imports curl_cffi only in
# 0.10.x–0.15.x and silently reports every target "unavailable" outside it. Move it when bumping
# yt-dlp; `tests/test_facebook_impersonation.py::test_the_locked_curl_cffi_can_impersonate_chrome`
# fails the gate if the locked pair can no longer impersonate.
curl_cffi>=0.10,<0.16
# PO Token provider plugin: talks to the bgutil sidecar (brainicism/bgutil-ytdlp-pot-provider)
# to mint YouTube Proof-of-Origin tokens on demand, bypassing "confirm you're not a bot" and
# unlocking high-quality formats. Pin to match the sidecar image tag (versions must align).
+16
View File
@@ -57,6 +57,10 @@ def _db_reachable() -> bool:
_DB_REACHABLE = _db_reachable()
# No background log writer in tests: every TestClient would otherwise start one on app startup and
# write into tables the fixtures truncate between tests. The sink's own tests drive it explicitly.
settings.log_db_sink = False
@pytest.fixture(scope="session")
def _schema():
@@ -210,3 +214,15 @@ def _reset_rate_limiters():
for obj in vars(mod).values():
if isinstance(obj, RateLimiter):
obj.reset()
@pytest.fixture(autouse=True)
def _reset_log_erasures():
"""`logs._erased` is module-level and remembers an erased account for an hour of monotonic time
— longer than any suite run — while the `db` fixture hands serial ids back out from 1 via
TRUNCATE … RESTART IDENTITY. So a test that purges user 1 would otherwise make a LATER test's
rows for user 1 store with `user_id = None`, failing somewhere nowhere near the cause and only
in some file orders. Same reasoning as the limiter sweep above, different global."""
from app import logs
logs._erased.clear()
+92
View File
@@ -0,0 +1,92 @@
{
"_comment": "One redaction contract for both logs: backend/app/logs.py redact() and frontend/src/lib/log.ts redact() must turn every `value` (under `key`) into `expected`. Both test suites run this file (backend tests/test_logs.py, frontend src/lib/log.contract.test.ts), so the two implementations cannot drift apart. Some cases prove a secret goes; others prove ordinary content (a search term, a Plex title, a chat line) is not mangled — what a LEVEL may carry is decided at the call site, not here.",
"cases": [
{
"name": "a password field is dropped whole",
"key": "",
"value": {
"email": "a@b.co",
"password": "hunter2",
"current_password": "x"
},
"expected": {
"email": "[email]",
"password": "[redacted]",
"current_password": "[redacted]"
}
},
{
"name": "a token under any spelling of the key, nested and in lists",
"key": "",
"value": {
"plex": { "plex_server_token": "abc", "Authorization": "Bearer z" },
"items": [{ "access_token": "t1" }, { "id": 3 }]
},
"expected": {
"plex": {
"plex_server_token": "[redacted]",
"Authorization": "[redacted]"
},
"items": [{ "access_token": "[redacted]" }, { "id": 3 }]
}
},
{
"name": "E2EE material and cookies",
"key": "",
"value": {
"ciphertext": "AAAA",
"passphrase": "p",
"private_key": "k",
"cookies": "SID=1"
},
"expected": {
"ciphertext": "[redacted]",
"passphrase": "[redacted]",
"private_key": "[redacted]",
"cookies": "[redacted]"
}
},
{
"name": "a reset link and a verify fragment",
"key": "msg",
"value": "open https://app.example/#reset=QWErty123 or /?verify=abc&lang=hu",
"expected": "open https://app.example/#reset=[redacted] or /?verify=[redacted]&lang=hu"
},
{
"name": "an OAuth callback URL",
"key": "url",
"value": "/auth/callback?code=4/0Ab-xyz&state=s3cr3t&scope=email",
"expected": "/auth/callback?code=[redacted]&state=[redacted]&scope=email"
},
{
"name": "an email address inside a sentence",
"key": "message",
"value": "Invitation sent to Some.One+tag@mail.example.org.",
"expected": "Invitation sent to [email]."
},
{
"name": "a search term is kept (TRACE decides whether it is logged at all)",
"key": "q",
"value": "lo-fi beats to relax",
"expected": "lo-fi beats to relax"
},
{
"name": "a Plex title with an at-sign is not an address",
"key": "title",
"value": "Live @ Wembley 1986",
"expected": "Live @ Wembley 1986"
},
{
"name": "a chat line is text like any other; its only secret part is scrubbed",
"key": "body",
"value": "see you at 8, my mail is me@home.example",
"expected": "see you at 8, my mail is [email]"
},
{
"name": "non-text values pass through",
"key": "",
"value": { "status": 404, "ok": false, "ratio": 0.5, "none": null },
"expected": { "status": 404, "ok": false, "ratio": 0.5, "none": null }
}
]
}
+90 -1
View File
@@ -1,6 +1,8 @@
"""Per-source adapter selection + the source-specific policy each one carries. These lock the
behaviour that used to live inline in worker.build_uploader_url / backfill._should_refetch."""
from app.config import settings
from app.titles import normalize_title
from app.downloads.adapters import (
FacebookAdapter,
RedditAdapter,
@@ -103,12 +105,74 @@ def test_generic_source_url_is_passthrough_youtube_reconstructs():
)
def test_client_sets_and_retriable_are_youtube_only():
def test_client_sets_and_retriable_are_per_source():
# Generic sources make a single attempt and never retry on the YouTube per-client markers.
assert SourceAdapter().client_sets() == [None]
assert SourceAdapter().client_sets(authed=True) == [None]
assert SourceAdapter().retriable("Sign in to confirm you're not a bot") is False
assert YouTubeAdapter().retriable("Sign in to confirm you're not a bot") is True
assert YouTubeAdapter().retriable("some unrelated network blip") is False
# A client list yt-dlp empties because none of its clients take cookies does NOT raise "No
# player clients have been requested" (that raise sits earlier in yt-dlp, before the
# cookie-capability removal). Measured with a real cookie: the emptied request fails with
# "Requested format is not available", which is retriable here already — so the wasted attempt
# is recoverable and `_COOKIE_CAPABLE` exists to avoid spending it at all.
assert (
YouTubeAdapter().retriable(
"ERROR: [youtube] abc: Requested format is not available. Use --list-formats"
)
is True
)
# The media fetch refused after a successful extraction: the format URLs belong to the client
# that minted them, so the next client set is a real second chance (it used to end the job).
assert (
YouTubeAdapter().retriable(
"ERROR: unable to download video data: HTTP Error 403: Forbidden"
)
is True
)
assert (
SourceAdapter().retriable("ERROR: unable to download video data: HTTP Error 403") is False
)
# Facebook: a second, Chrome-impersonating attempt only WITH the user's cookie, and only behind
# the parse wall (without a cookie that wall is the login wall — no disguise gets past it).
fb = FacebookAdapter()
assert fb.client_sets() == [None]
assert len(fb.client_sets(authed=True)) == 2
assert fb.ydl_opts_extra(fb.client_sets(authed=True)[0]) == {}
assert fb.retriable("ERROR: [facebook] 1: Cannot parse data; please report this issue") is True
assert fb.retriable("ERROR: [facebook] 1: No video formats found!") is False
def test_a_signed_in_youtube_download_only_runs_cookie_capable_clients(monkeypatch):
yt = YouTubeAdapter()
monkeypatch.setattr(settings, "download_player_clients", "android_vr")
monkeypatch.setattr(settings, "download_player_clients_fallback", "web_safari,web,android")
# Signed out: the admin's ladder exactly as configured.
assert yt.client_sets() == [["android_vr"], ["web_safari", "web", "android"]]
# Signed in: yt-dlp REMOVES every client that takes no cookies, and a set it empties that way
# raises "No player clients have been requested" — which would fail the download on attempt 1,
# before the POT-backed web fallback the cookie exists for. So the adapter never asks for one:
# the android_vr-only set is gone and `android` is dropped out of the fallback.
assert yt.client_sets(authed=True) == [["web_safari", "web"]]
def test_a_signed_in_download_still_has_clients_when_the_admin_configured_none(monkeypatch):
# Both knobs cookie-incapable: fall back to the POT-backed web pair rather than no attempt at all.
monkeypatch.setattr(settings, "download_player_clients", "android_vr")
monkeypatch.setattr(settings, "download_player_clients_fallback", "android,ios")
assert YouTubeAdapter().client_sets(authed=True) == [["web_safari", "web"]]
def test_the_cookie_capable_clients_match_the_installed_ytdlp():
# The list is yt-dlp's, not ours (INNERTUBE_CLIENTS[...]["SUPPORTS_COOKIES"]). Pinned here so a
# yt-dlp bump that renames or re-flags a client turns the gate red, instead of silently sending
# a signed-in download to a client that drops the cookie.
from yt_dlp.extractor.youtube._base import INNERTUBE_CLIENTS
live = {k for k, v in INNERTUBE_CLIENTS.items() if v.get("SUPPORTS_COOKIES")}
assert live == set(YouTubeAdapter._COOKIE_CAPABLE)
assert set(YouTubeAdapter._AUTHED_DEFAULT) <= live
def test_youtube_ydl_opts_extra_carries_player_client_and_pot():
@@ -130,3 +194,28 @@ def test_wants_metadata_backfill_matches_old_should_refetch():
def test_nfo_uniqueid_type_is_the_adapter_key():
assert YouTubeAdapter().nfo_uniqueid_type() == "youtube"
assert SourceAdapter().nfo_uniqueid_type() == "generic"
def test_title_is_yt_dlps_for_a_source_without_placeholders():
assert SourceAdapter().title({"title": "Video", "description": "text"}) == "Video"
assert SourceAdapter().title({}) is None
def test_facebook_placeholder_title_takes_the_posts_text():
# Measured: signed in, yt-dlp titles every reel "Video" (an 18+ reel read as Chrome: "Facebook"),
# while the post's text is still in the description.
a = FacebookAdapter()
info = {"title": "Video", "description": "\n Tune of The Day\nsecond line"}
assert a.title(info) == "Tune of The Day"
assert a.title({**info, "title": "Facebook"}) == "Tune of The Day"
# A real title (signed out) is left for normalize_title, engagement prefix and all.
assert a.title({**info, "title": "456K views | Tune"}) == "456K views | Tune"
# Nothing better on the page: keep what yt-dlp gave.
assert a.title({"title": "Facebook", "description": None}) == "Facebook"
def test_facebook_title_from_a_long_description_is_the_whole_line_after_cleaning():
# Dev UAT: a cut with "…" came out of normalize_title as "… live at." (NFKC "..." collapsed to "."),
# a sentence that seemed to end. The whole first line survives the cleaning intact instead.
line = 'Tune of The Day, Argy playing his version of the classic "Resurrection" with Paul Oakenfold live at Space'
assert normalize_title(FacebookAdapter().title({"title": "Video", "description": line})) == line
+70
View File
@@ -12,6 +12,7 @@ Note: `/api/me` is the bootstrap probe — it returns 200 `{authenticated: False
from datetime import timedelta
import pytest
from fastapi.testclient import TestClient
from app.auth import ACTIVE_ACCOUNT_HEADER
@@ -57,6 +58,75 @@ def test_a_non_wallet_account_header_falls_back_to_the_default(client, make_user
assert me.json()["email"] == "alice2@example.com" # still Alice, not Mallory
def test_a_tab_pinned_to_an_account_signed_out_elsewhere_acts_as_nobody(client, make_user):
# Two accounts in one browser: a tab pinned to Alice, Bob the default. Alice signs out in another
# tab. The pinned tab used to go on acting as the default, Bob — its player checkpoints and pref
# saves landed in his account, and its own Sign out signed HIM out.
alice = make_user(email="alice3@example.com", password=_PW)
make_user(email="bob3@example.com", password=_PW)
assert _login(client, "alice3@example.com", _PW).status_code == 200
assert _login(client, "bob3@example.com", _PW).status_code == 200 # Bob is now the default
tab = {ACTIVE_ACCOUNT_HEADER: str(alice.id)}
assert client.get("/api/feed", headers=tab).status_code == 200
assert client.post("/auth/logout", headers=tab).status_code == 200 # Alice signs out
assert client.get("/api/feed", headers=tab).status_code == 401
assert client.post("/auth/logout", headers=tab).status_code == 200
assert client.get("/api/feed").status_code == 200 # Bob is still signed in, the session intact
# The bootstrap probe still answers as the default: how the stale tab learns who it is now.
assert client.get("/api/me", headers=tab).json()["email"] == "bob3@example.com"
@pytest.mark.parametrize("ended", ["revoked", "suspended"])
def test_a_tab_pinned_to_an_account_whose_session_ended_comes_back_as_the_default(
client, make_user, db, ended
):
# A tab pinned to Alice, Bob the default. Alice's session then ends while she is still in this
# browser's wallet ("log out everywhere" on another device, or an admin suspends her). The tab's
# bootstrap probe used to answer "signed out": the landing page with Bob still signed in, and no way
# off it but closing the tab (signing in as anyone else there kept the tab pinned to Alice).
alice = make_user(email="alice4@example.com", password=_PW)
make_user(email="bob4@example.com", password=_PW)
assert _login(client, "alice4@example.com", _PW).status_code == 200
assert _login(client, "bob4@example.com", _PW).status_code == 200
tab = {ACTIVE_ACCOUNT_HEADER: str(alice.id)}
assert client.get("/api/feed", headers=tab).status_code == 200
if ended == "revoked":
alice.session_epoch = (alice.session_epoch or 0) + 1
else:
alice.is_suspended = True
db.commit()
assert client.get("/api/me", headers=tab).json()["email"] == "bob4@example.com"
assert client.get("/api/feed", headers=tab).status_code == 401 # still nobody, never Bob
assert client.get("/api/feed").status_code == 200 # Bob's session is intact
@pytest.mark.parametrize("ended", ["revoked", "suspended", "deactivated"])
def test_switching_to_an_account_whose_session_ended_refuses_it_and_drops_it_from_the_list(
client, make_user, db, monkeypatch, ended
):
# The account switcher lists this browser's wallet. A revoked account was dropped from it on a refused
# switch, but a suspended one stayed, refused on every click; a deactivated one was not refused at all —
# made the default, the next request found it inactive and signed every account here out.
monkeypatch.setattr("app.routes.me.is_allowed", lambda _db, _email: True) # both hold an invite
alice = make_user(email="alice5@example.com", password=_PW)
make_user(email="bob5@example.com", password=_PW)
assert _login(client, "alice5@example.com", _PW).status_code == 200
assert _login(client, "bob5@example.com", _PW).status_code == 200
if ended == "revoked":
alice.session_epoch = (alice.session_epoch or 0) + 1
elif ended == "suspended":
alice.is_suspended = True
else:
alice.is_active = False
db.commit()
assert client.post("/api/me/switch", json={"user_id": alice.id}).status_code == 403
assert [a["email"] for a in client.get("/api/me/accounts").json()] == ["bob5@example.com"]
assert client.get("/api/feed").status_code == 200 # Bob is still signed in
def test_logout_others_revokes_the_other_session_but_keeps_the_caller(client, make_user):
# Two independent cookie jars (browsers) signed into the same account. `logout-others` from one
# bumps the account epoch so the OTHER's cookie is rejected on its next request, while the caller
+11 -9
View File
@@ -7,14 +7,14 @@ of the caller's already-translated fallback, so from that commit on the demo use
this gate is UI-gated on `is_demo` either — the comfort shelf is not, so the pin button is live.
A structured `{code: "demo"}` is what `localizeDetail` can turn into `errors.demo` in both locales.
Pinned at all FOUR sites: the shared dependency, the two auth routes that hand-roll the same check,
and the account-delete route in `routes/me.py` — which round 6 missed while its docstring claimed
Pinned at every site: the shared dependency, the two auth routes that hand-roll the same check,
the account-delete route in `routes/me.py` — which round 6 missed while its docstring claimed
all the sites were covered, so the next reader had no reason to look (round 7). An asymmetric fix is
the smell this repo bans, and a docstring asserting symmetry it does not have is worse than none.
The list is derived from the source, not typed out: `test_every_is_demo_refusal_is_on_the_list`
below fails when a fifth `is_demo` 403 appears anywhere in `app/`, so this file cannot fall behind
the code the way round 6's did. Needs the DB.
the code the way round 6's did. (E22 S5 added the fifth: raising the diagnostics level.) Needs the DB.
"""
import re
@@ -33,6 +33,8 @@ def test_every_demo_refusal_carries_the_same_structured_reason(db, make_user, as
("post", "/auth/logout-others", None),
("post", "/auth/set-password", {"password": "a long enough password"}),
("delete", "/api/me/account", None),
# E22 S5: the shared account's log would interleave every visitor's session.
("put", "/api/me/diagnostics", {"level": "DEBUG"}),
]:
r = getattr(client, method)(url, **({"json": body} if body is not None else {}))
assert r.status_code == 403, (url, r.text)
@@ -55,9 +57,9 @@ def test_every_is_demo_refusal_is_on_the_list():
for path in sorted(app_dir.rglob("*.py")):
lines = path.read_text(encoding="utf-8").splitlines()
for i, line in enumerate(lines):
# `if user.is_demo:` / `if u.is_demo:` — the hand-rolled gate, not the ORM filters
# (`User.is_demo.is_(False)`) and not the column definition.
if not re.search(r"^\s*if\s+\w+\.is_demo\s*:", line):
# `if user.is_demo:` / `if u.is_demo and …:` — the hand-rolled gate, not the ORM
# filters (`User.is_demo.is_(False)`) and not the column definition.
if not re.search(r"^\s*if\s+\w+\.is_demo\b[^:]*:", line):
continue
block = "\n".join(lines[i : i + 6])
if "403" not in block:
@@ -67,9 +69,9 @@ def test_every_is_demo_refusal_is_on_the_list():
f"{path.relative_to(app_dir)}:{i + 1} refuses the demo account with an "
f"unstructured detail — the frontend cannot localize it:\n{block}"
)
# auth.py's shared gate + routes/auth.py's two + routes/me.py's delete. A new one is welcome,
# but it has to be added to the loop above in the same commit.
assert len(sites) == 4, sites
# auth.py's shared gate + routes/auth.py's two + routes/me.py's delete and diagnostics level. A
# new one is welcome, but it has to be added to the loop above in the same commit.
assert len(sites) == 5, sites
def test_a_real_account_is_not_refused_by_the_gate(db, make_user, as_user):
+492
View File
@@ -0,0 +1,492 @@
"""E22 S5 — a user's consented diagnostics level, and the browser log it lets in.
Three parts:
* the level itself: the server computes its expiry, a lapsed one reads as the base, the demo
account cannot raise it (DB lane, through the real routes);
* `POST /api/diagnostics/events`: refused at the base level, and nothing in a batch is trusted —
the account comes from the session, everything is redacted again, events under the consented
level are dropped, sizes and pace are capped (DB lane);
* the server side of the elevation: ONE user's requests are logged in detail while everyone
else's stay at the base level, and the account reaches a handler's own lines at all (pure, an
in-process app).
"""
import json
import logging
import threading
from datetime import datetime, timedelta, timezone
import pytest
from fastapi import Depends, FastAPI
from fastapi.testclient import TestClient
from sqlalchemy import select
from app import diagnostics, logs, logsink, sysconfig
from app.config import settings
from app.models import LogEvent, User
from app.routes import diagnostics as diagnostics_routes
_PW = "correct horse battery"
# --- the level ------------------------------------------------------------------------------
def test_a_raised_level_counts_until_it_lapses_and_not_a_moment_longer():
now = datetime.now(timezone.utc)
assert diagnostics.effective_level(User()) == "INFO"
raised = User(diagnostics_level="TRACE", diagnostics_until=now + timedelta(minutes=1))
assert diagnostics.effective_level(raised, now) == "TRACE"
assert diagnostics.effective_level(raised, now + timedelta(minutes=2)) == "INFO"
# A level without an expiry is not a level that never ends — it is no level at all.
assert diagnostics.effective_level(User(diagnostics_level="DEBUG")) == "INFO"
def _signed_in(client, make_user, **kw):
"""A real sign-in: `/api/me` resolves through `optional_current_user`, which the `as_user`
override does not cover (see test_me_email_flag)."""
user = make_user(email="diag@example.com", password=_PW, **kw)
r = client.post("/auth/password-login", json={"email": "diag@example.com", "password": _PW})
assert r.status_code == 200, r.text
return user
def test_raising_the_level_sets_the_servers_expiry_whatever_the_client_sends(db, client, make_user):
user = _signed_in(client, make_user)
r = client.put("/api/me/diagnostics", json={"level": "DEBUG", "until": "2099-01-01T00:00:00Z"})
assert r.status_code == 200, r.text
db.refresh(user)
expected = datetime.now(timezone.utc) + timedelta(hours=diagnostics.ELEVATION_HOURS)
assert abs((user.diagnostics_until - expected).total_seconds()) < 60
assert r.json()["diagnostics"]["level"] == "DEBUG"
assert client.get("/api/me").json()["diagnostics"]["level"] == "DEBUG"
assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
db.refresh(user)
assert (user.diagnostics_level, user.diagnostics_until) == (None, None)
assert client.get("/api/me").json()["diagnostics"] == {
"level": "INFO",
"until": None,
"retention_days": 7,
}
def test_a_lapsed_level_reads_as_the_base_in_me(db, client, make_user):
_signed_in(
client,
make_user,
diagnostics_level="TRACE",
diagnostics_until=datetime.now(timezone.utc) - timedelta(minutes=1),
)
assert client.get("/api/me").json()["diagnostics"] == {
"level": "INFO",
"until": None,
"retention_days": 7,
}
def test_only_the_three_levels_can_be_chosen(db, make_user, as_user):
client = as_user(make_user())
for bogus in ("WARNING", "debug", "EVERYTHING"):
assert client.put("/api/me/diagnostics", json={"level": bogus}).status_code == 422
def test_the_demo_account_can_return_to_the_base_but_never_raise_it(db, make_user, as_user):
client = as_user(make_user(is_demo=True))
r = client.put("/api/me/diagnostics", json={"level": "TRACE"})
assert r.status_code == 403 and r.json()["detail"]["code"] == "demo"
assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
def test_preferences_must_be_a_json_object(db, make_user, as_user):
client = as_user(make_user())
assert client.put("/api/me/preferences", json=["not", "an", "object"]).status_code == 422
r = client.put("/api/me/preferences", json={"theme": "dark"})
assert r.status_code == 200 and r.json()["preferences"]["theme"] == "dark"
# --- the browser's batches ------------------------------------------------------------------
def _event(**over) -> dict:
return {
"ts": datetime.now(timezone.utc).isoformat(),
"level": "INFO",
"event": "nav.view",
"msg": "view",
"tab_id": "a1b2c3",
"version": "0.88.0",
**over,
}
@pytest.fixture
def elevated(db, make_user, as_user):
"""A signed-in client whose user consented to `level` a minute ago, and a reader for its rows."""
diagnostics_routes._batches.reset()
def _make(level: str = "DEBUG"):
user = make_user(
diagnostics_level=level,
diagnostics_until=datetime.now(timezone.utc) + timedelta(hours=23),
)
def rows() -> list[LogEvent]:
db.expire_all()
return list(db.execute(select(LogEvent).order_by(LogEvent.id)).scalars())
return user, as_user(user), rows
yield _make
diagnostics_routes._batches.reset()
def test_nothing_is_accepted_at_the_base_level(db, make_user, as_user):
client = as_user(make_user())
r = client.post("/api/diagnostics/events", json={"events": [_event()]})
assert r.status_code == 409 and r.json()["detail"]["code"] == "diagnostics_off"
assert not db.execute(select(LogEvent)).first()
def test_an_anonymous_caller_is_refused(client):
assert client.post("/api/diagnostics/events", json={"events": [_event()]}).status_code == 401
def test_a_batch_is_stored_as_the_sessions_web_events_and_redacted_again(elevated):
user, client, rows = elevated("DEBUG")
r = client.post(
"/api/diagnostics/events",
json={
"events": [
_event(
level="WARNING",
event="ui.notify",
msg="sent to someone@example.com",
attrs={"password": "hunter2", "url": "/?token=abc123", "user_id": 999},
request_id="0123456789abcdef",
),
_event(level="DEBUG", event="ui.click", attrs={"control": "feed.refresh"}),
],
"user_id": 999, # ignored: the account is the session's
},
)
assert r.json() == {"ok": True, "stored": 2}
warning, click = rows()
assert (warning.source, warning.logger, warning.user_id) == ("web", "web", user.id)
assert warning.msg == "sent to [email]"
assert warning.attrs["password"] == logs.REDACTED
assert "abc123" not in json.dumps(warning.attrs)
assert warning.attrs["tab_id"] == "a1b2c3"
assert warning.request_id == "0123456789abcdef"
assert (warning.sev, click.level, click.version) == (13, "DEBUG", "0.88.0")
def test_an_erased_accounts_batch_is_stored_without_the_account(elevated):
"""The sink's writer is not the only path into `log_events` — this route inserts too, and an
account erased while its browser was mid-flush must not get its rows back through it. Faked at
the `forget_user` seam the purge itself calls: once the User row is actually gone the session
no longer resolves, so the reachable window is exactly a request already past `current_user`.
(`logs._erased` is cleared before every test by conftest's `_reset_log_erasures`.)"""
user, client, rows = elevated("DEBUG")
logs.forget_user(user.id)
r = client.post("/api/diagnostics/events", json={"events": [_event(level="WARNING")]})
assert r.json() == {"ok": True, "stored": 1}
(stored,) = rows()
assert (stored.source, stored.user_id) == ("web", None)
def test_an_event_under_the_consented_level_is_dropped(elevated):
_user, client, rows = elevated("DEBUG")
r = client.post(
"/api/diagnostics/events",
json={"events": [_event(level="TRACE", msg="typed: lofi"), _event(level="DEBUG")]},
)
assert r.json()["stored"] == 1
assert [row.level for row in rows()] == ["DEBUG"]
def test_a_timestamp_outside_the_consent_window_is_clamped(elevated):
_user, client, rows = elevated("DEBUG")
now = datetime.now(timezone.utc)
future = (now + timedelta(days=365)).isoformat()
past = (now - timedelta(days=30)).isoformat()
client.post("/api/diagnostics/events", json={"events": [_event(ts=future), _event(ts=past)]})
late, early = rows()
assert late.ts <= datetime.now(timezone.utc)
assert early.ts >= now - timedelta(hours=diagnostics.ELEVATION_HOURS, minutes=1)
def test_sizes_are_capped(elevated):
_user, client, rows = elevated("DEBUG")
too_many = [_event() for _ in range(diagnostics_routes.EVENTS_MAX + 1)]
assert client.post("/api/diagnostics/events", json={"events": too_many}).status_code == 422
huge = "x" * (diagnostics_routes.BODY_MAX_BYTES + 1)
r = client.post(
"/api/diagnostics/events",
content=json.dumps({"events": [_event(attrs={"blob": huge})]}),
headers={"content-type": "application/json"},
)
assert r.status_code == 413
big_attrs = {"blob": "y" * (diagnostics_routes.ATTRS_MAX_BYTES + 10)}
assert (
client.post("/api/diagnostics/events", json={"events": [_event(attrs=big_attrs)]}).json()[
"stored"
]
== 1
)
(row,) = rows()
assert "blob" not in row.attrs and row.attrs["attrs_dropped_bytes"] > 0
bad_code = client.post("/api/diagnostics/events", json={"events": [_event(event="Nav View")]})
assert bad_code.status_code == 422
def test_the_pace_is_capped_per_user(elevated):
_user, client, _rows = elevated("DEBUG")
codes = [
client.post("/api/diagnostics/events", json={"events": []}).status_code for _ in range(31)
]
assert codes[:30] == [200] * 30 and codes[30] == 429
def test_what_an_account_stores_per_hour_is_capped_and_said_once(elevated, monkeypatch):
user, client, rows = elevated("DEBUG")
monkeypatch.setattr(
diagnostics_routes, "_stored_bytes", diagnostics_routes.RateLimiter(3_000, 3600)
)
batch = {"events": [_event(msg="m" * 1_000)]}
stored = [client.post("/api/diagnostics/events", json=batch).json()["stored"] for _ in range(4)]
assert stored == [1, 1, 0, 0]
events = [row.event for row in rows()]
# Two batches, then ONE notice for the hour, not one per refused batch.
assert events == ["nav.view", "nav.view", "diagnostics.budget_spent"]
assert rows()[-1].user_id == user.id
def test_only_a_json_body_is_read(elevated):
_user, client, rows = elevated("DEBUG")
# What a same-site `<form enctype="text/plain">` can send without a preflight.
r = client.post(
"/api/diagnostics/events",
content=json.dumps({"events": [_event()]}),
headers={"content-type": "text/plain"},
)
assert r.status_code == 415 and r.json()["detail"]["code"] == "json_required"
assert rows() == []
ok = client.post(
"/api/diagnostics/events",
content=json.dumps({"events": [_event()]}),
headers={"content-type": "application/json; charset=utf-8"},
)
assert ok.json()["stored"] == 1
# --- the server side of an elevation ---------------------------------------------------------
class _Capture(logging.Handler):
"""The stdout handler's filters (context + gate), capturing records."""
def __init__(self):
super().__init__(level=logging.NOTSET)
self.records: list[logging.LogRecord] = []
self.addFilter(logs.ContextFilter())
self.addFilter(logs.LevelGate())
def emit(self, record):
self.records.append(record)
@pytest.fixture
def gated():
"""A process configured at INFO with a gated handler on the ROOT (where the real one sits), and
every piece of module state the elevation touches restored afterwards."""
root = logging.getLogger()
saved = (logs._base_level, dict(logs._elevations), logs._root_applied, root.level)
handler = _Capture()
root.addHandler(handler)
logs._elevations.clear()
logs.set_level("INFO")
try:
yield handler
finally:
root.removeHandler(handler)
logs._base_level, logs._root_applied = saved[0], saved[2]
logs._elevations.clear()
logs._elevations.update(saved[1])
root.setLevel(saved[3])
def _app() -> TestClient:
"""What a real request does: the request middleware, then a SYNC auth dependency that resolves
the account (run by FastAPI in a threadpool, on a copy of the context), then a handler."""
app = FastAPI()
app.middleware("http")(logs.log_requests)
handler_log = logging.getLogger("siftlode.difftest")
def account(user_id: int, level: str = "INFO"):
until = datetime.now(timezone.utc) + timedelta(hours=1)
logs.set_user(user_id, level, until)
@app.get("/api/thing/{user_id}")
def _thing(user_id: int, _=Depends(account)):
handler_log.debug("detail only an elevated request keeps")
handler_log.info("an ordinary line")
return {}
return TestClient(app)
def test_the_account_reaches_the_handlers_own_lines(gated):
_app().get("/api/thing/41")
line = next(r for r in gated.records if r.getMessage() == "an ordinary line")
assert getattr(line, "user_id", None) == 41
def test_one_users_elevation_logs_their_requests_in_detail_and_nobody_elses(gated):
client = _app()
client.get("/api/thing/7", params={"level": "DEBUG"})
client.get("/api/thing/8")
detail = [r for r in gated.records if r.getMessage().startswith("detail only")]
assert [r.user_id for r in detail] == [7]
assert getattr(detail[0], "elevated", False) is True
# Every line of the elevated request is marked for storage — the sink keeps it at any level.
ordinary = [r for r in gated.records if r.getMessage() == "an ordinary line"]
assert [logsink.should_store(r) for r in ordinary] == [True, False]
# The process's configured level is still what it reports, however low the root went.
assert logs.current_level() == "INFO" and logging.getLogger().level == logging.DEBUG
def test_the_root_level_returns_to_the_base_when_the_elevation_ends(gated):
client = _app()
client.get("/api/thing/7", params={"level": "TRACE"})
assert logging.getLogger().level == logs.TRACE
client.get("/api/thing/7") # the same user, back at the base
assert logging.getLogger().level == logging.INFO
logs._track_elevation(9, logging.DEBUG, datetime.now(timezone.utc) - timedelta(seconds=1))
assert logging.getLogger().level == logging.INFO # an already-expired one is pruned at once
def test_the_query_string_is_logged_only_for_a_traced_request_and_redacted(gated):
client = _app()
client.get("/api/thing/7", params={"level": "DEBUG", "q": "lofi"})
client.get("/api/thing/7", params={"level": "TRACE", "q": "lofi", "token": "SECRET"})
lines = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert "query" not in lines[0].attrs
assert "q=lofi" in lines[1].attrs["query"] and "SECRET" not in lines[1].attrs["query"]
def test_an_address_typed_into_a_search_is_redacted_though_the_browser_encodes_it(gated):
_app().get("/api/thing/7?level=TRACE&q=someone%40example.com&new_password=hunter2")
(line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert line.attrs["query"] == "level=TRACE&q=[email]&new_password=[redacted]"
@pytest.mark.parametrize(
"raw",
["state=a%20b", "code=%20SECRET", "sig=a%22b", "key=a%26b", "reset=a+b"],
ids=["space", "leading-space", "quote", "ampersand", "plus"],
)
def test_a_credential_param_goes_whole_however_its_value_decodes(raw):
"""Decoding is what lets `_EMAIL_RE` see `%40`, but it also puts spaces and separators into a
value that `_SECRET_PARAM_RE`'s value class stops at — so the name rule decides, not the text."""
name = raw.split("=", 1)[0]
assert logs.redact_query(f"{raw}&q=lofi") == f"{name}={logs.REDACTED}&q=lofi"
def test_an_admin_level_of_trace_does_not_log_other_users_query_strings(gated):
logs.set_level("TRACE")
_app().get("/api/thing/7", params={"q": "something private"})
(line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert "query" not in line.attrs
def test_the_admin_level_applies_to_the_running_process_when_set_and_when_reset(
gated, db, make_user, as_user
):
admin = as_user(make_user(role="admin"))
r = admin.patch("/api/admin/config/log_level", json={"value": "DEBUG"})
assert r.status_code == 200, r.text
assert logs.current_level() == "DEBUG"
assert admin.delete("/api/admin/config/log_level").status_code == 200
assert logs.current_level() == settings.log_level
def test_erasing_the_only_elevated_account_puts_the_process_level_back(gated):
"""The root level is the lowest of the base and every unexpired elevation. `forget_user` drops
an elevation like any other path that drops one, so it owes the same recompute — or the process
keeps building every TRACE record for an account that no longer exists, to throw them away."""
until = datetime.now(timezone.utc) + timedelta(hours=1)
try:
logs.set_user(7, "TRACE", until)
assert logging.getLogger().level == logs.TRACE
logs.forget_user(7)
assert logging.getLogger().level == logging.INFO
finally:
# `set_user` writes the request scope, and outside a request that scope is this process's
# for good — a later test would store its INFO lines as an elevated session's.
logs.set_user(None)
def test_the_stored_admin_level_is_applied_at_boot(gated, db, monkeypatch):
import asyncio
import app.main as app_main
sysconfig.set_value(db, "log_level", "DEBUG")
db.commit()
monkeypatch.setattr(app_main.state, "is_configured", lambda db: True)
monkeypatch.setattr(app_main, "start_scheduler", lambda: None)
monkeypatch.setattr(app_main, "shutdown_scheduler", lambda: None)
monkeypatch.setattr(app_main.plex_subtitles, "shutdown", lambda: None)
async def _no_sweep():
return None
monkeypatch.setattr(app_main, "_sweep_hls_orphans", _no_sweep)
levels: list[str] = []
async def _cycle():
async with app_main.lifespan(app_main.app):
levels.append(logs.current_level())
asyncio.run(_cycle())
assert levels == ["DEBUG"]
def test_the_root_level_is_applied_in_the_order_the_elevations_changed(gated, monkeypatch):
"""Two requests change the elevations at once: A's user leaves (the root may go back to INFO),
B's user arrives (it must go to DEBUG). B's change is the later one, so DEBUG must be what stays
— even when A is paused between working out its level and applying it."""
root = logging.getLogger()
logs._track_elevation(1, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1))
real_set = root.setLevel
other: list[threading.Thread] = []
def paused_set(level):
if level == logging.INFO and not other:
b = threading.Thread(
target=logs._track_elevation,
args=(2, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1)),
)
other.append(b)
b.start()
b.join(0.3) # B runs to completion here unless A's level change holds it off
real_set(level)
monkeypatch.setattr(root, "setLevel", paused_set)
logs._track_elevation(1, None, None) # A
other[0].join(5)
assert root.level == logging.DEBUG and 2 in logs._elevations
def test_a_real_sign_in_carries_the_consented_level_into_the_process(gated, client, db, make_user):
user = _signed_in(client, make_user)
assert client.put("/api/me/diagnostics", json={"level": "DEBUG"}).status_code == 200
logs._elevations.clear() # only a request resolving the account through auth may put it back
client.get("/api/me")
assert logs._elevations[user.id][0] == logging.DEBUG
assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
assert user.id not in logs._elevations
+204 -8
View File
@@ -18,7 +18,7 @@ from cryptography.fernet import Fernet
from app import security
from app.downloads import credentials, formats, service
from app.models import DownloadCredential, MediaAsset, User
from app.models import DownloadCredential, DownloadJob, MediaAsset, User
@pytest.fixture(autouse=True)
@@ -39,13 +39,23 @@ class TestHostKeyForUrl:
assert credentials.host_key_for_url("https://www.facebook.com/watch?v=1") == "facebook"
assert credentials.host_key_for_url("https://fb.watch/abc/") == "facebook"
assert credentials.host_key_for_url("https://x.com/i/status/1") == "twitter"
# YouTube gates individual videos behind its bot check; the cookie is the only way past it.
for url in (
"https://www.youtube.com/watch?v=abc",
"https://youtu.be/abc",
"https://music.youtube.com/watch?v=abc",
"https://www.youtube-nocookie.com/embed/abc",
):
assert credentials.host_key_for_url(url) == "youtube", url
def test_look_alike_host_never_matches(self):
assert credentials.host_key_for_url("https://myfacebook.com/x") is None
assert credentials.host_key_for_url("https://facebook.com.evil.test/x") is None
assert credentials.host_key_for_url("https://notyoutube.com/watch?v=abc") is None
assert credentials.host_key_for_url("https://youtube.com.evil.test/x") is None
def test_unrelated_or_empty_url_has_no_host_key(self):
assert credentials.host_key_for_url("https://www.youtube.com/watch?v=abc") is None
assert credentials.host_key_for_url("https://vimeo.com/123456") is None
assert credentials.host_key_for_url("") is None
assert credentials.host_key_for_url(None) is None
@@ -72,6 +82,21 @@ def test_authed_sig_differs_per_user_and_from_the_shared_sig():
assert a != shared and b != shared and a != b
def test_an_authed_asset_queued_before_a_signature_version_bump_still_gets_its_cookie(monkeypatch):
# A job keeps the asset it was queued with. A deploy that bumps the signature version between the
# queueing and the download used to make the user's private asset stop counting as theirs, so the
# download went without its cookie and failed on the login wall.
spec = formats.normalize({"mode": "a"})
queued = MediaAsset(format_sig=credentials.authed_format_sig(spec, 1, "facebook"))
rebuild = credentials.rebuild_salt(7, credentials.cookie_salt(1, "facebook"))
rebuilt = MediaAsset(id=7, format_sig=formats.format_sig(spec, salt=rebuild))
monkeypatch.setattr(formats, "_AUDIO_SIG_VERSION", formats._AUDIO_SIG_VERSION + 1) # the deploy
for asset in (queued, rebuilt):
assert credentials.is_authed_asset(asset, spec, 1, FB_URL)
assert not credentials.is_authed_asset(asset, spec, 2, FB_URL) # still private to its user
def test_write_cookiefile_is_private_and_survives_the_retry_wipe(tmp_path):
staging = tmp_path / "staging"
staging.mkdir()
@@ -160,6 +185,22 @@ class TestCachePrivacy:
asset = db.get(MediaAsset, job.asset_id)
assert asset.format_sig == formats.format_sig(formats.normalize({}))
def test_a_youtube_job_reaches_the_gate_through_its_rebuilt_watch_url(self, db):
# The one join nothing else makes. A YouTube job stores a bare 11-char ref, and the URL the
# cookie gate keys on is REBUILT by the adapter (service.source_url -> YouTubeAdapter). Both
# ends are pinned apart — the watch-URL shape in test_adapters, the host_key in
# TestHostKeyForUrl — so this welds them: an enqueue that ever handed the gate the raw ref
# would silently stop salting, every YouTube cookie would quietly stop being attached, and
# signed-out downloads would keep working, so nothing else here would go red.
u = _user(db)
credentials.upsert_credential(db, u.id, "youtube", COOKIES)
db.commit()
job = service.enqueue(db, u.id, "youtube", "VIDID12345_", {})
asset = db.get(MediaAsset, job.asset_id)
spec = formats.normalize({})
assert asset.format_sig == credentials.authed_format_sig(spec, u.id, "youtube")
assert asset.format_sig != formats.format_sig(spec)
def test_two_users_never_share_the_authed_asset(self, db):
a = _user(db, "a@example.com")
b = _user(db, "b@example.com")
@@ -189,7 +230,9 @@ class TestResolveCookiefile:
db.commit()
spec = formats.normalize({})
salted = credentials.authed_format_sig(spec, u.id, "facebook")
path = credentials.resolve_cookiefile(db, u.id, FB_URL, salted, spec, tmp_path)
path = credentials.resolve_cookiefile(
db, u.id, FB_URL, self._asset_sig(salted), spec, tmp_path
)
assert path is not None and Path(path).read_text() == COOKIES
def test_never_attaches_to_a_shared_asset(self, db, tmp_path):
@@ -200,23 +243,176 @@ class TestResolveCookiefile:
db.commit()
spec = formats.normalize({})
shared = formats.format_sig(spec)
assert credentials.resolve_cookiefile(db, u.id, FB_URL, shared, spec, tmp_path) is None
assert (
credentials.resolve_cookiefile(
db, u.id, FB_URL, self._asset_sig(shared), spec, tmp_path
)
is None
)
def test_non_matching_host_gets_no_cookie(self, db, tmp_path):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
spec = formats.normalize({})
# A YouTube URL has no supported host_key -> no cookie regardless of what's stored.
yt = "https://www.youtube.com/watch?v=abc"
# A URL of ANOTHER host (supported or not) never gets this credential's cookie. YouTube is
# a supported host in its own right now, so the case needs a host that is not.
other = "https://vimeo.com/123456"
salted = credentials.authed_format_sig(spec, u.id, "facebook")
assert credentials.resolve_cookiefile(db, u.id, yt, salted, spec, tmp_path) is None
assert (
credentials.resolve_cookiefile(db, u.id, other, self._asset_sig(salted), spec, tmp_path)
is None
)
# ...and a YouTube link gets nothing either: the stored credential is Facebook's.
yt = "https://www.youtube.com/watch?v=abc"
assert (
credentials.resolve_cookiefile(db, u.id, yt, self._asset_sig(salted), spec, tmp_path)
is None
)
def test_no_credential_gets_no_cookie(self, db, tmp_path):
u = _user(db)
spec = formats.normalize({})
salted = credentials.authed_format_sig(spec, u.id, "facebook")
assert credentials.resolve_cookiefile(db, u.id, FB_URL, salted, spec, tmp_path) is None
assert (
credentials.resolve_cookiefile(
db, u.id, FB_URL, self._asset_sig(salted), spec, tmp_path
)
is None
)
# --- Retry / Re-download take a cookie added later (DB) -----------------------------------------
# Measured on dev (E11.5): a Facebook reel failed on the login wall, the user added their cookie as
# the failure copy tells them to, pressed Retry — and it failed again, because the job still held the
# shared asset the worker never attaches a cookie to. Re-download forked a salt the gate could not
# recognise, so even a job downloaded WITH a cookie rebuilt without it.
def _held_job(db, u, *, sig, asset_status="error", job_status="error"):
a = MediaAsset(
source_kind="url", source_ref=FB_URL, format_sig=sig, status=asset_status, ref_count=1
)
db.add(a)
db.flush()
j = DownloadJob(
user_id=u.id,
asset_id=a.id,
source_kind="url",
source_ref=FB_URL,
status=job_status,
queue_pos=0,
profile_snapshot={},
)
db.add(j)
db.commit()
return a, j
class TestRequeueTakesTheCookie:
spec = formats.normalize({})
@pytest.mark.parametrize(
("asset_status", "job_status"),
# A failed download; and a job paused before it ever started (its asset never downloaded).
[("error", "error"), ("pending", "paused")],
)
def test_retry_after_adding_a_cookie_moves_the_job_to_the_private_asset(
self, db, tmp_path, asset_status, job_status
):
u = _user(db)
shared, j = _held_job(
db,
u,
sig=formats.format_sig(self.spec),
asset_status=asset_status,
job_status=job_status,
)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
service.resume(db, j)
db.expire_all()
job = db.get(DownloadJob, j.id)
target = db.get(MediaAsset, job.asset_id)
assert job.status == "queued" and target.id != shared.id
assert target.format_sig == credentials.authed_format_sig(self.spec, u.id, "facebook")
assert (target.status, target.ref_count) == ("pending", 1)
assert db.get(MediaAsset, shared.id).ref_count == 0 # the old hold went with the move
assert credentials.resolve_cookiefile(db, u.id, FB_URL, target, self.spec, tmp_path)
def test_retry_without_a_cookie_keeps_the_shared_asset(self, db):
u = _user(db)
shared, j = _held_job(db, u, sig=formats.format_sig(self.spec))
service.resume(db, j)
db.expire_all()
assert db.get(DownloadJob, j.id).asset_id == shared.id
assert db.get(MediaAsset, shared.id).ref_count == 1
@pytest.mark.parametrize(
("asset_status", "job_status"), [("ready", "error"), ("downloading", "paused")]
)
def test_retry_never_moves_off_a_file_or_a_download_in_flight(
self, db, asset_status, job_status
):
u = _user(db)
shared, j = _held_job(
db,
u,
sig=formats.format_sig(self.spec),
asset_status=asset_status,
job_status=job_status,
)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
service.resume(db, j)
db.expire_all()
assert db.get(DownloadJob, j.id).asset_id == shared.id
def test_redownload_with_a_cookie_rebuilds_into_an_asset_the_cookie_goes_with(
self, db, tmp_path
):
u = _user(db)
other = _user(db, "other@example.com")
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
credentials.upsert_credential(db, other.id, "facebook", COOKIES)
authed = credentials.authed_format_sig(self.spec, u.id, "facebook")
first, j = _held_job(db, u, sig=authed, asset_status="ready", job_status="done")
seen = {first.id}
for _ in range(2): # a rebuild of a rebuild is still recognised
service.redownload(db, j)
db.expire_all()
job = db.get(DownloadJob, j.id)
target = db.get(MediaAsset, job.asset_id)
assert target.id not in seen and target.format_sig != authed
seen.add(target.id)
assert credentials.resolve_cookiefile(db, u.id, FB_URL, target, self.spec, tmp_path)
# Private: another user's cookie is never attached to it.
assert not credentials.is_authed_asset(target, self.spec, other.id, FB_URL)
# A Retry of a failed rebuild keeps the rebuild — it is already the cookie's identity.
target.status, job.status = "error", "error"
db.commit()
service.resume(db, job)
db.expire_all()
assert db.get(DownloadJob, j.id).asset_id == target.id
def test_a_rebuild_made_without_a_cookie_is_no_cookie_identity(self, db, tmp_path):
u = _user(db)
_, j = _held_job(db, u, sig=formats.format_sig(self.spec))
service.redownload(db, j)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
db.expire_all()
target = db.get(MediaAsset, db.get(DownloadJob, j.id).asset_id)
assert not credentials.is_authed_asset(target, self.spec, u.id, FB_URL)
# --- API: write-only, never echoes the blob (DB) ----------------------------------------------
+159
View File
@@ -1,5 +1,6 @@
"""Category classification of raw yt-dlp / ffmpeg download-failure strings (S2)."""
from app.downloads import adapters
from app.downloads import errors as dlerrors
from app.downloads.errors import classify
@@ -64,6 +65,164 @@ def test_our_own_worker_refusals_classify_as_themselves():
assert classify(dlerrors.QUOTA_FULL_MESSAGE) == "quota_exceeded"
def test_the_redacted_network_text_classifies_back_to_network():
# The redacted text is also what a failed asset keeps, and a second job that meets that asset is
# diagnosed from it — without its own rule it would read "Download failed".
assert classify(dlerrors.NETWORK_REDACTED_MESSAGE) == "network"
# Verbatim yt-dlp 2026.07.04 output from prod jobs, measured on dev 2026-09-13 (E11.5).
_FB_PARSE = (
"ERROR: [facebook] 1736360027626346: Cannot parse data; please report this issue on "
"https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template."
)
_FB_NO_FORMATS = (
"ERROR: [facebook] 1755988225525547: No video formats found!; please report this issue on "
"https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template."
)
_REEL = "https://www.facebook.com/reel/1736360027626346/?fs=e&fs=e"
_GROUP_POST = "https://www.facebook.com/groups/983554479435596/permalink/1755988225525547/?"
# Verbatim from prod job 273 (2026-09-15). YouTube writes the apostrophe as U+2019, which is why
# neither the rule nor the catalog entry may spell out "you’re" with a straight quote.
_YT_BOT_WALL = (
"ERROR: [youtube] 7JPmS_bWjhI: Sign in to confirm you’re not a bot. Use "
"--cookies-from-browser or --cookies for the authentication. See "
"https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp for how to manually "
"pass cookies."
)
_YT = "https://www.youtube.com/watch?v=7JPmS_bWjhI"
def test_youtube_bot_wall_is_that_video_own_signed_out_gate():
# Measured on prod: all 12 player clients fail on this video while its neighbours download, the
# POT sidecar answers, and the newest yt-dlp fails identically — a cookie is the only way in.
assert classify(_YT_BOT_WALL) == "login_required"
for url in (
_YT,
"https://youtu.be/7JPmS_bWjhI",
"https://www.youtube-nocookie.com/embed/7JPmS_bWjhI",
"https://music.youtube.com/watch?v=7JPmS_bWjhI",
):
assert dlerrors.diagnose(_YT_BOT_WALL, url=url) == (
"login_required",
"youtube.bot_wall",
), url
def test_youtube_bot_wall_with_a_cookie_is_the_cookie_refused():
# Telling a user who already gave a cookie to add one is what this entry exists to avoid.
assert dlerrors.diagnose(_YT_BOT_WALL, url=_YT, cookie_attached=True) == (
"login_required",
"youtube.cookie_rejected",
)
# Verbatim from the isolated test stack (2026-09-16), the SAME video ~90 minutes later: yt-dlp got
# the player response and picked formats, and the CDN then refused the stream.
_YT_MEDIA_403 = "ERROR: unable to download video data: HTTP Error 403: Forbidden"
def test_youtube_media_403_is_the_wall_one_step_later_not_a_network_fault():
# Generically this text is a network failure — and for every other host it stays one.
assert classify(_YT_MEDIA_403) == "network"
assert dlerrors.diagnose(_YT_MEDIA_403, url="https://vimeo.com/1") == ("network", None)
# On a signed-out YouTube download it is the gate: measured, the same video downloaded in full
# once the owner's cookie was attached. Calling that "network error" is the honest-states bug.
assert dlerrors.diagnose(_YT_MEDIA_403, url=_YT) == (
"login_required",
"youtube.media_forbidden",
)
def test_youtube_media_403_with_a_cookie_is_not_claimed_as_the_gate():
# Only the cookie-less case was measured. With a cookie the same text keeps its bare category
# rather than telling a signed-in user to sign in.
assert dlerrors.diagnose(_YT_MEDIA_403, url=_YT, cookie_attached=True) == ("network", None)
def test_the_youtube_entries_never_match_another_site():
# Host-scoped: the same words about a Facebook link stay the bare category.
assert dlerrors.diagnose(
"ERROR: [facebook] 1: Sign in to confirm you’re not a bot",
url="https://www.facebook.com/reel/1/",
) == ("login_required", None)
def test_facebook_cannot_parse_data_is_the_login_wall_without_a_cookie():
for url in (
_REEL,
_GROUP_POST,
"https://fb.watch/Ia_iJYI2Vw/?fs=e",
"https://m.facebook.com/x",
):
assert dlerrors.diagnose(_FB_PARSE, url=url) == ("login_required", "facebook.login_wall")
def test_facebook_cannot_parse_data_with_a_cookie_is_the_cookie_refused():
# Telling a user who already gave a cookie to add one is the failure this entry exists to avoid.
assert dlerrors.diagnose(_FB_PARSE, url=_REEL, cookie_attached=True) == (
"login_required",
"facebook.cookie_rejected",
)
def test_a_facebook_group_post_without_formats_names_the_link():
# Without a cookie yt-dlp is given the post link itself (measured, prod jobs 66 and 145).
assert dlerrors.diagnose(_FB_NO_FORMATS, url=_GROUP_POST) == (
"format_unavailable",
"facebook.group_post_video",
)
# With one it is only ever given the post's VIDEO link, so the text is about that video: the adapter
# found it, and "this post has no video" would be false.
assert dlerrors.diagnose(_FB_NO_FORMATS, url=_GROUP_POST, cookie_attached=True) == (
"format_unavailable",
None,
)
# A reel, or a group link that is not a post, is another failure: only the category.
for url in (_REEL, "https://www.facebook.com/groups/983554479435596/"):
assert dlerrors.diagnose(_FB_NO_FORMATS, url=url) == ("format_unavailable", None), url
def test_our_post_not_shown_verdict_is_the_refused_cookie():
# The adapter's text when a signed-in read of the post got a page without the post (the login page).
msg = adapters.POST_NOT_SHOWN_MESSAGE
assert classify(msg) == "login_required"
assert dlerrors.diagnose(msg, url=_GROUP_POST, cookie_attached=True) == (
"login_required",
"facebook.cookie_rejected",
)
def test_our_post_without_a_video_verdict_names_the_group_link():
# The adapter's own text when the post page holds no single video (S4): the message and its rule and
# catalog entry are pinned together, or an edit to the wording orphans the copy.
msg = adapters.POST_WITHOUT_VIDEO_MESSAGE
assert classify(msg) == "format_unavailable"
assert dlerrors.diagnose(msg, url=_GROUP_POST, cookie_attached=True) == (
"format_unavailable",
"facebook.group_post_video",
)
assert dlerrors.diagnose(msg, url=_REEL, cookie_attached=True) == ("format_unavailable", None)
def test_a_catalog_entry_never_leaves_its_host():
# "Cannot parse data" is yt-dlp's wording for a broken extractor on every site.
other = _FB_PARSE.replace("[facebook]", "[vimeo]")
assert dlerrors.diagnose(other, url="https://vimeo.com/1") == ("unknown", None)
# The Facebook text for a link that is not Facebook's (a look-alike host) is not the Facebook wall.
assert dlerrors.diagnose(_FB_PARSE, url="https://facebook.com.evil/reel/1") == ("unknown", None)
# No link known: the category only.
assert dlerrors.diagnose(_FB_PARSE) == ("unknown", None)
def test_diagnose_falls_back_to_the_category():
msg = "ERROR: [facebook] 123: You must be logged in to see this post"
assert dlerrors.diagnose(msg, url=_REEL) == ("login_required", None)
assert dlerrors.diagnose(None, url=_REEL) == ("unknown", None)
def test_a_url_jobs_network_failure_is_redacted_but_youtube_keeps_its_text():
# SEC-12: the raw transport text for a user-chosen host:port is a port-scanner readout, so it is
# replaced for `url` jobs. Everything else must survive verbatim, or a real diagnosis is lost.
@@ -0,0 +1,444 @@
"""DB-backed tests for the download failure catalog (E11.5 S1) at the places a failure is WRITTEN.
`test_download_errors` pins which entry a message matches; these pin that every writer hands the
matcher what it needs — the link and whether the user's cookie went with it — and that a stored entry
never outlives the failure it describes:
- the download child's failure branch (`pipeline._download`);
- a second job meeting an already-failed asset (`worker._process_job`), which only has the text;
- Retry / Re-download, which requeue the job;
- the `0084` backfill of the rows that failed before the catalog existed;
- the downloads list, which is how the entry reaches the UI.
It also holds the one success-path check of the same child: the duration it stores (E11.5 S3).
"""
import shutil
import subprocess
import pytest
from alembic import command
from alembic.config import Config
from cryptography.fernet import Fernet
from sqlalchemy import text
from app import security
from app.config import settings
from app.db import engine
from app.downloads import credentials, formats, pipeline, service
from app.models import DownloadJob, DownloadQuota, MediaAsset, User
from app.worker import _process_job
_REEL = "https://www.facebook.com/reel/1736360027626346/?fs=e&fs=e"
# Verbatim yt-dlp 2026.07.04, prod job 256.
_FB_PARSE = (
"ERROR: [facebook] 1736360027626346: Cannot parse data; please report this issue on "
"https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template."
)
_COOKIES = "# Netscape HTTP Cookie File\n.facebook.com\tTRUE\t/\tTRUE\t0\tc_user\t123\n"
@pytest.fixture
def _fernet(monkeypatch):
# A throwaway key, so a stored credential encrypts and decrypts (mirrors test_download_credentials).
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
def _user(db, email):
u = User(email=email)
db.add(u)
db.flush()
db.add(
DownloadQuota(
user_id=u.id, max_bytes=10**12, max_concurrent=5, max_jobs=10**6, unlimited=True
)
)
return u
def _asset_and_job(db, user, *, sig, asset_status, job_status, error=None):
a = MediaAsset(
source_kind="url",
source_ref=_REEL,
format_sig=sig,
status=asset_status,
error=error,
ref_count=1,
)
db.add(a)
db.flush()
j = DownloadJob(
user_id=user.id,
asset_id=a.id,
source_kind="url",
source_ref=_REEL,
status=job_status,
queue_pos=0,
profile_snapshot={},
)
db.add(j)
db.commit()
return a, j
def _fail_the_child(monkeypatch, tmp_path, *, cookie: bool):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
cookiefile = tmp_path / "cookies.txt"
cookiefile.write_text("# Netscape HTTP Cookie File\n")
monkeypatch.setattr(
pipeline.credentials,
"resolve_cookiefile",
lambda *a, **k: cookiefile if cookie else None,
)
def extract(*a, **k):
raise RuntimeError(_FB_PARSE)
monkeypatch.setattr(pipeline, "_extract_with_fallback", extract)
def test_the_download_child_diagnoses_with_the_link_and_no_cookie(db, monkeypatch, tmp_path):
u = _user(db, "fb-nocookie@example.com")
a, j = _asset_and_job(db, u, sig="shared", asset_status="downloading", job_status="running")
_fail_the_child(monkeypatch, tmp_path, cookie=False)
pipeline._download(j.id, a.id, "url", _REEL, {}, u.id)
db.expire_all()
job = db.get(DownloadJob, j.id)
assert (job.status, job.error_code, job.error_detail) == (
"error",
"login_required",
"facebook.login_wall",
)
def test_the_download_child_knows_the_cookie_went_with_it(db, monkeypatch, tmp_path):
# The same text with the user's cookie attached: telling them to add cookies would be wrong.
u = _user(db, "fb-cookie@example.com")
a, j = _asset_and_job(db, u, sig="private", asset_status="downloading", job_status="running")
_fail_the_child(monkeypatch, tmp_path, cookie=True)
pipeline._download(j.id, a.id, "url", _REEL, {}, u.id)
db.expire_all()
job = db.get(DownloadJob, j.id)
assert (job.error_code, job.error_detail) == ("login_required", "facebook.cookie_rejected")
def test_a_job_meeting_a_failed_shared_asset_gets_the_same_explanation(db):
# Before the catalog this path wrote the text and NO code: "Download failed" for the second
# job, "Facebook sign-in required" for the first.
u = _user(db, "fb-second@example.com")
_, j = _asset_and_job(
db, u, sig="shared", asset_status="error", job_status="running", error=_FB_PARSE
)
_process_job(j.id)
db.expire_all()
job = db.get(DownloadJob, j.id)
assert (job.status, job.error_code, job.error_detail) == (
"error",
"login_required",
"facebook.login_wall",
)
def test_a_job_meeting_its_users_failed_private_asset_knows_the_cookie_was_used(db, _fernet):
u = _user(db, "fb-private@example.com")
credentials.upsert_credential(db, u.id, "facebook", _COOKIES)
sig = credentials.authed_format_sig(formats.normalize({}), u.id, "facebook")
_, j = _asset_and_job(
db, u, sig=sig, asset_status="error", job_status="running", error=_FB_PARSE
)
_process_job(j.id)
db.expire_all()
job = db.get(DownloadJob, j.id)
assert (job.error_code, job.error_detail) == ("login_required", "facebook.cookie_rejected")
def test_a_private_asset_the_cookie_could_not_go_with_is_explained_as_the_download_had_it(db):
# The cookie was deleted (or can no longer be decrypted) after the jobs were queued: the download
# child had nothing to attach and said "add your cookies". The job meeting that failed asset must say
# the same — the asset's salt alone would have told it "your cookies were refused".
u = _user(db, "fb-gone@example.com")
sig = credentials.authed_format_sig(formats.normalize({}), u.id, "facebook")
_, j = _asset_and_job(
db, u, sig=sig, asset_status="error", job_status="running", error=_FB_PARSE
)
_process_job(j.id)
db.expire_all()
job = db.get(DownloadJob, j.id)
assert (job.error_code, job.error_detail) == ("login_required", "facebook.login_wall")
def test_a_job_canceled_before_it_meets_a_failed_asset_stays_canceled(db):
# The cancel landed between the claim and this write. `error` would hold the asset again after the
# route already released it — a later delete releases twice.
u = _user(db, "fb-canceled@example.com")
_, j = _asset_and_job(
db, u, sig="shared", asset_status="error", job_status="canceled", error=_FB_PARSE
)
_process_job(j.id)
db.expire_all()
job = db.get(DownloadJob, j.id)
assert (job.status, job.error_code, job.error_detail) == ("canceled", None, None)
def test_the_downloads_list_carries_the_catalog_entry(db, as_user):
# The whole explanation rides on this one field: without it every row falls back to the category.
u = _user(db, "fb-list@example.com")
_, j = _asset_and_job(
db, u, sig="shared", asset_status="error", job_status="error", error=_FB_PARSE
)
j.error_code, j.error_detail = "login_required", "facebook.login_wall"
db.commit()
[row] = as_user(u).get("/api/downloads").json()
assert (row["error_code"], row["error_detail"]) == ("login_required", "facebook.login_wall")
@pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
def test_a_download_yt_dlp_gave_no_duration_stores_the_files_duration(db, monkeypatch, tmp_path):
# Facebook reel 257 (E11.5): yt-dlp's info had no duration, the asset stored NULL, and the Library
# row lost its trim action. Through the real finalize, with a real 3-second file.
u = _user(db, "fb-duration@example.com")
a, j = _asset_and_job(db, u, sig="shared", asset_status="downloading", job_status="running")
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
def extract(job_id, asset_id, url, spec, staging, adapter, **kwargs):
media = staging / "1736360027626346.mp4"
subprocess.run(
[
"ffmpeg",
"-y",
"-loglevel",
"error",
"-f",
"lavfi",
"-i",
"testsrc=size=64x64:rate=10",
"-f",
"lavfi",
"-i",
"sine=frequency=440",
"-t",
"3",
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-c:a",
"aac",
str(media),
],
check=True,
)
return {"id": "1736360027626346", "ext": "mp4", "title": "Reel", "filepath": str(media)}
monkeypatch.setattr(pipeline, "_extract_with_fallback", extract)
pipeline._download(j.id, a.id, "url", _REEL, {}, u.id)
db.expire_all()
assert db.get(DownloadJob, j.id).status == "done"
assert db.get(MediaAsset, a.id).duration_s == 3
def test_retry_and_redownload_drop_the_old_explanation(db):
u = _user(db, "fb-requeue@example.com")
for requeue in (service.resume, service.redownload):
_, j = _asset_and_job(
db, u, sig=f"s-{requeue.__name__}", asset_status="error", job_status="error"
)
j.error, j.error_code, j.error_detail = _FB_PARSE, "login_required", "facebook.login_wall"
db.commit()
requeue(db, j)
db.refresh(j)
assert (j.status, j.error_code, j.error_detail) == ("queued", None, None), requeue.__name__
def test_0084_backfills_only_the_facebook_parse_failures_it_measured(db):
cfg = Config("alembic.ini")
command.downgrade(cfg, "0083_comfort_pins")
upgraded = False
try:
with engine.begin() as conn:
uid = conn.execute(
text("INSERT INTO users (email) VALUES ('backfill@example.com') RETURNING id")
).scalar_one()
rows = {
"fb_unknown": ("error", "unknown", _FB_PARSE),
"fb_null": ("canceled", None, _FB_PARSE),
"other_site": ("error", "unknown", _FB_PARSE.replace("[facebook]", "[vimeo]")),
"fb_other_code": ("error", "network", _FB_PARSE),
}
ids = {}
for name, (status, code, error) in rows.items():
ids[name] = conn.execute(
text(
"INSERT INTO download_jobs (user_id, source_kind, source_ref, "
"profile_snapshot, status, error_code, error) "
"VALUES (:u, 'url', :ref, '{}', :s, :c, :e) RETURNING id"
),
{"u": uid, "ref": _REEL, "s": status, "c": code, "e": error},
).scalar_one()
command.upgrade(cfg, "head")
upgraded = True
with engine.connect() as conn:
got = {
name: tuple(
conn.execute(
text("SELECT error_code, error_detail FROM download_jobs WHERE id = :i"),
{"i": i},
).one()
)
for name, i in ids.items()
}
assert got == {
"fb_unknown": ("login_required", "facebook.login_wall"),
"fb_null": ("login_required", "facebook.login_wall"),
"other_site": ("unknown", None),
"fb_other_code": ("network", None),
}
finally:
if not upgraded:
command.upgrade(cfg, "head")
def _ffmpeg_media(path, *codec_args):
subprocess.run(
["ffmpeg", "-y", "-loglevel", "error", "-f", "lavfi", "-i", "sine=frequency=440"]
+ list(codec_args)
+ ["-t", "2", str(path)],
check=True,
)
@pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
def test_a_signed_in_facebook_download_is_titled_with_its_posts_text(db, monkeypatch, tmp_path):
# Prod jobs 265–268 (2026-09-14): with a cookie yt-dlp titles every reel "Video", so the row, the
# file name and the share preview all said "Video". The post's text is in the description.
u = _user(db, "fb-title@example.com")
a, j = _asset_and_job(db, u, sig="shared", asset_status="downloading", job_status="running")
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
def extract(job_id, asset_id, url, spec, staging, adapter, **kwargs):
media = staging / "2131884261040652.m4a"
_ffmpeg_media(media, "-c:a", "aac")
return {
"id": "2131884261040652",
"ext": "m4a",
"title": "Video",
"description": "Tune of The Day, Argy playing his version",
"extractor_key": "Facebook",
"filepath": str(media),
}
monkeypatch.setattr(pipeline, "_extract_with_fallback", extract)
pipeline._download(j.id, a.id, "url", _REEL, {"mode": "a"}, u.id)
db.expire_all()
asset, job = db.get(MediaAsset, a.id), db.get(DownloadJob, j.id)
assert asset.title == job.display_name == "Tune of The Day, Argy playing his version"
assert "Tune_of_The_Day" in asset.rel_path
def test_the_early_title_takes_the_posts_text_too(db):
u = _user(db, "fb-early-title@example.com")
a, _ = _asset_and_job(db, u, sig="shared", asset_status="downloading", job_status="running")
a.title = None
db.commit()
pipeline._fill_asset_meta_early(
a.id,
{
"id": "1",
"title": "Video",
"description": "Almás palacsinta",
"extractor_key": "Facebook",
},
)
db.expire_all()
assert db.get(MediaAsset, a.id).title == "Almás palacsinta"
@pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
def test_an_mp3_download_is_stored_as_an_mp3(db, monkeypatch, tmp_path):
# Prod job 268: the "Audio (MP3)" profile's extracted .mp3 was remuxed into an .mp4 and served as
# video/mp4, which iPad's WebKit will not play. It stays an .mp3 (served as audio/mpeg).
u = _user(db, "fb-mp3@example.com")
a, j = _asset_and_job(db, u, sig="shared", asset_status="downloading", job_status="running")
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
def extract(job_id, asset_id, url, spec, staging, adapter, **kwargs):
media = staging / "2131884261040652.mp3"
_ffmpeg_media(media, "-c:a", "libmp3lame")
return {"id": "2131884261040652", "ext": "mp3", "title": "Reel", "filepath": str(media)}
monkeypatch.setattr(pipeline, "_extract_with_fallback", extract)
pipeline._download(
j.id, a.id, "url", _REEL, formats.normalize({"mode": "a", "audio_format": "mp3"}), u.id
)
db.expire_all()
asset = db.get(MediaAsset, a.id)
assert (asset.container, asset.acodec, asset.media_kind) == ("mp3", "mp3", "audio")
assert asset.rel_path.endswith(".mp3")
@pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
@pytest.mark.parametrize("mode", ["av", "v"])
def test_a_video_download_is_still_made_browser_playable(db, monkeypatch, tmp_path, mode):
# The mp3 fix skips the browser-playable pass for AUDIO only: a video that arrives in a container
# browsers won't play (an .mkv) is still remuxed into an .mp4.
u = _user(db, f"fb-mkv-{mode}@example.com")
a, j = _asset_and_job(
db, u, sig=f"mkv-{mode}", asset_status="downloading", job_status="running"
)
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
def extract(job_id, asset_id, url, spec, staging, adapter, **kwargs):
media = staging / "1736360027626346.mkv"
audio = ["-f", "lavfi", "-i", "sine=frequency=440", "-c:a", "aac"] if mode == "av" else []
subprocess.run(
[
"ffmpeg",
"-y",
"-loglevel",
"error",
"-f",
"lavfi",
"-i",
"testsrc=size=64x64:rate=10",
]
+ audio
+ ["-t", "1", "-c:v", "libx264", "-pix_fmt", "yuv420p", str(media)],
check=True,
)
return {"id": "1736360027626346", "ext": "mkv", "title": "Reel", "filepath": str(media)}
monkeypatch.setattr(pipeline, "_extract_with_fallback", extract)
pipeline._download(j.id, a.id, "url", _REEL, formats.normalize({"mode": mode}), u.id)
db.expire_all()
asset = db.get(MediaAsset, a.id)
assert (asset.container, asset.vcodec) == ("mp4", "h264")
assert asset.rel_path.endswith(".mp4")
+139
View File
@@ -0,0 +1,139 @@
"""Overlapping requests on ONE job (E11.5 review) — DB-backed, two real sessions.
The routes that move a job's asset ref count (resume, cancel, delete, redownload) used to decide on an
unlocked read: two overlapping requests for one job (a double-click, a second tab, a replay after a
gateway timeout) both saw it holding its asset and both released the hold. The count then reaches 0
while another job still holds the asset, and GC or the ready-file delete removes it under that job.
A pause racing a cancel did the same one step later: its unlocked write turned the released job back
into a holder, which a later delete releases again. `_own_job_for_update` makes the second request
wait and decide on the first one's committed result.
"""
import threading
import pytest
from fastapi import HTTPException
from sqlalchemy import select
from app.db import SessionLocal
from app.downloads import service
from app.models import DownloadJob, MediaAsset, User
from app.routes.downloads import (
_own_job_for_update,
cancel_download,
delete_download,
pause_download,
)
_REF = "https://example.com/video"
def _asset_held_by_two_jobs(db, status="error"):
mine, theirs = User(email="lock-mine@example.com"), User(email="lock-theirs@example.com")
db.add_all([mine, theirs])
db.flush()
asset = MediaAsset(
source_kind="url", source_ref=_REF, format_sig="shared", status="error", ref_count=2
)
db.add(asset)
db.flush()
jobs = [
DownloadJob(
user_id=owner.id,
asset_id=asset.id,
source_kind="url",
source_ref=_REF,
status=status if owner is mine else "error",
queue_pos=0,
profile_snapshot={},
)
for owner in (mine, theirs)
]
db.add_all(jobs)
db.commit()
return mine, asset, jobs[0]
@pytest.mark.parametrize(
("status", "transition", "route", "left"),
[
("error", service.cancel, cancel_download, "canceled"),
("error", service.delete, delete_download, None),
("queued", service.cancel, pause_download, "canceled"),
],
)
def test_overlapping_requests_for_one_job_release_its_hold_once(
db, status, transition, route, left
):
user, asset, job = _asset_held_by_two_jobs(db, status)
job_id = job.id # read before the delete case removes the row under the expired instance
first, second = SessionLocal(), SessionLocal()
finished = threading.Event()
def the_other_request():
try:
route(job_id, user=user, db=second)
except HTTPException:
pass # a request after the delete: the job is gone, a 404
finally:
second.close()
finished.set()
try:
locked = _own_job_for_update(first, user, job_id)
other = threading.Thread(target=the_other_request)
other.start()
# The second request waits for the first one's locks instead of deciding on the same row.
assert not finished.wait(0.5), "the second request did not wait for the first"
transition(first, locked)
other.join(10)
assert finished.is_set()
finally:
first.close()
db.expire_all()
mine = select(DownloadJob.status).where(DownloadJob.id == job_id)
assert db.execute(mine).scalar_one_or_none() == left
assert db.get(MediaAsset, asset.id).ref_count == 1 # the other user's job still holds it
def test_losing_the_asset_insert_race_keeps_the_callers_own_writes(db, monkeypatch):
# Retry clears the job's error, THEN may create the cookie asset. When another request inserted that
# asset first, the loser used to roll the whole session back — and requeue the job with its stale
# error, which a stall or crash later leaves showing under the new failure.
user = User(email="race@example.com")
db.add(user)
db.flush()
job = DownloadJob(
user_id=user.id,
source_kind="url",
source_ref=_REF,
status="error",
error="the old failure",
queue_pos=0,
profile_snapshot={},
)
db.add(job)
db.commit()
job.error = None # the caller's pending write
real_flush, raced = db.flush, []
def flush_after_the_other_request_inserted(*args, **kwargs):
if not raced: # the other request commits the same asset between our lookup and our insert
raced.append(True)
with SessionLocal() as other:
other.add(
MediaAsset(source_kind="url", source_ref=_REF, format_sig="raced", ref_count=0)
)
other.commit()
return real_flush(*args, **kwargs)
monkeypatch.setattr(db, "flush", flush_after_the_other_request_inserted)
asset = service.get_or_create_asset(db, "url", _REF, "raced")
db.commit()
assert raced and asset.format_sig == "raced"
db.expire_all()
assert db.get(DownloadJob, job.id).error is None
@@ -0,0 +1,243 @@
"""E11.5 S3 + S4 — the Facebook download path with the user's cookie.
S3, measured on dev (yt-dlp 2026.07.04, the owner's cookie): reel 806379462330494 fails with the cookie
alone and extracts 2560p with `--impersonate chrome`; the reels that already download with the cookie
keep their plain first attempt. These pin the attempt order through the real download loop
(`pipeline._extract_with_fallback`, with yt-dlp faked at the `YoutubeDL` boundary), and that the
LOCKED curl_cffi can actually impersonate — the pair only works inside yt-dlp's own version range, and
outside it every target silently reports "unavailable".
S4, measured on dev and in the owner's Chrome: given a group post's link, yt-dlp finds no formats or
downloads another video from the page's feed. The post's own video is the single Video node in the
page scripts that carry the post id (prod posts 66, 145, 140, 143, 221; text posts have none while the
page holds ~10 other videos), and its watch link downloads.
"""
import pytest
import yt_dlp
from yt_dlp.networking.exceptions import TransportError
from yt_dlp.networking.impersonate import ImpersonateTarget
from app.downloads import adapters, pipeline
from app.downloads import errors as dlerrors
_URL = "https://www.facebook.com/reel/806379462330494/?fs=e&fs=e"
_POST = "1755988225525547"
_GROUP_LINK = f"https://www.facebook.com/groups/983554479435596/permalink/{_POST}/?"
_POST_VIDEO = "1408015447857753"
_FEED_VIDEO = "1056963063816966" # what yt-dlp downloaded from this post's link: a feed video
_CHROME = ImpersonateTarget.from_str("chrome")
def _video_node(video_id: str) -> str:
# The measured shape of a post's attachment in the page JSON.
return '{"media":{"__typename":"Video","id":"%s","__isNode":"Video"},"styles":{}}' % video_id
def _page(*post_videos: str, post: str = _POST) -> str:
attachments = ",".join(_video_node(v) for v in post_videos)
return (
'<html><script type="application/json" data-sjs>'
f'{{"post_id":"{post}","creation_time":1785084554,"attachments":[{attachments}]}}</script>'
'<script type="application/json" data-sjs>'
f'{{"feed":[{_video_node(_FEED_VIDEO)}]}}</script></html>'
)
_WALL = (
"ERROR: [facebook] 806379462330494: Cannot parse data; please report this issue on "
"https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template."
)
_NO_FORMATS = "ERROR: [facebook] 2541212983053999: No video formats found!"
def test_the_locked_curl_cffi_can_impersonate_chrome():
# YoutubeDL refuses to start when the target is unavailable — the same check a download makes.
with yt_dlp.YoutubeDL({"impersonate": ImpersonateTarget.from_str("chrome"), "quiet": True}):
pass
@pytest.fixture
def ydl(monkeypatch):
"""Fake `yt_dlp.YoutubeDL`: records each attempt's options and plays back `outcomes` in order;
`web["page"]` is what a page read returns (`web["fail"]`, when set, is raised instead), and `web`
records the pages opened, the byte cap each read asked for, and the links extracted."""
seen: list[dict] = []
outcomes: list = []
web: dict = {"page": "", "fail": None, "opened": [], "reads": [], "extracted": []}
class FakeYDL:
def __init__(self, opts):
seen.append(opts)
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def urlopen(self, url):
web["opened"].append(url)
if web["fail"] is not None:
raise web["fail"]
body = web["page"].encode()
class Response:
def read(self, amt=None):
web["reads"].append(amt)
return body
return Response()
def extract_info(self, url, download):
web["extracted"].append(url)
outcome = outcomes.pop(0)
if isinstance(outcome, str):
raise yt_dlp.utils.DownloadError(outcome)
return outcome
monkeypatch.setattr(pipeline.yt_dlp, "YoutubeDL", FakeYDL)
monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: None)
monkeypatch.setattr(pipeline, "_set_job", lambda job_id, **fields: None)
return seen, outcomes, web
def _run(tmp_path, cookiefile, url=_URL):
return pipeline._extract_with_fallback(
1, 1, url, {}, tmp_path, adapters.FacebookAdapter(), cookiefile=cookiefile
)
def test_a_cookie_download_at_the_wall_is_retried_as_chrome(ydl, tmp_path):
seen, outcomes, web = ydl
outcomes += [_WALL, {"id": "806379462330494"}]
assert _run(tmp_path, "/cookies.txt") == {"id": "806379462330494"}
assert [o.get("impersonate") for o in seen] == [None, _CHROME]
assert all(o["cookiefile"] == "/cookies.txt" for o in seen) # the retry stays signed in
assert web["opened"] == [] # a reel is not a group post: no page read
def test_without_a_cookie_the_wall_is_the_login_wall_and_final(ydl, tmp_path):
seen, outcomes, _ = ydl
outcomes += [_WALL]
with pytest.raises(yt_dlp.utils.DownloadError):
_run(tmp_path, None)
assert len(seen) == 1 and "impersonate" not in seen[0]
def test_other_facebook_failures_are_not_retried_as_chrome(ydl, tmp_path):
seen, outcomes, _ = ydl
outcomes += [_NO_FORMATS]
with pytest.raises(yt_dlp.utils.DownloadError):
_run(tmp_path, "/cookies.txt")
assert len(seen) == 1 and "impersonate" not in seen[0]
# --- S4: a group post downloads ITS video, or fails saying it has none ----------------------------
def test_a_cookie_group_post_downloads_the_posts_own_video(ydl, tmp_path):
seen, outcomes, web = ydl
web["page"] = _page(_POST_VIDEO)
outcomes += [_WALL, {"id": _POST_VIDEO}] # the watch link needs Chrome too (measured)
assert _run(tmp_path, "/cookies.txt", _GROUP_LINK) == {"id": _POST_VIDEO}
assert web["opened"] == [_GROUP_LINK]
# Never an unbounded read.
assert web["reads"] == [adapters.FacebookAdapter._POST_PAGE_MAX_BYTES]
assert seen[0]["cookiefile"] == "/cookies.txt" # the post page is read signed in
watch = f"https://www.facebook.com/watch/?v={_POST_VIDEO}"
# Never the post link: that is where the feed video comes from.
assert web["extracted"] == [watch, watch]
assert [o.get("impersonate") for o in seen[1:]] == [None, _CHROME]
def test_a_group_post_without_a_single_video_fails_saying_so(ydl, tmp_path):
_, _, web = ydl
web["page"] = _page() # a text post: the only video on the page is the feed's
with pytest.raises(RuntimeError) as exc:
_run(tmp_path, "/cookies.txt", _GROUP_LINK)
assert web["extracted"] == [] # yt-dlp is not left to guess from the post link
assert dlerrors.diagnose(str(exc.value), url=_GROUP_LINK, cookie_attached=True) == (
"format_unavailable",
"facebook.group_post_video",
)
def test_a_group_post_page_without_the_post_is_the_refused_cookie(ydl, tmp_path):
# Measured with a refused cookie (E11.5 review): Facebook answers the post link with its login page —
# a login form, no post id anywhere. Saying "this post has no video" would blame the post.
_, _, web = ydl
web["page"] = '<html><title>ChatGPT magyarul</title><form id="login_form"></form></html>'
with pytest.raises(RuntimeError) as exc:
_run(tmp_path, "/cookies.txt", _GROUP_LINK)
assert web["extracted"] == []
assert dlerrors.diagnose(str(exc.value), url=_GROUP_LINK, cookie_attached=True) == (
"login_required",
"facebook.cookie_rejected",
)
def test_a_failed_post_page_read_fails_the_download_as_itself(ydl, tmp_path):
# A read that failed says nothing about the post: it must not turn into "no video in this post".
_, _, web = ydl
web["fail"] = TransportError("Connection reset by peer")
with pytest.raises(TransportError) as exc:
_run(tmp_path, "/cookies.txt", _GROUP_LINK)
assert web["extracted"] == []
assert dlerrors.diagnose(str(exc.value), url=_GROUP_LINK, cookie_attached=True) == (
"network",
None,
)
def test_without_a_cookie_the_group_post_link_goes_to_ytdlp_unchanged(ydl, tmp_path):
_, outcomes, web = ydl
outcomes += [_NO_FORMATS]
with pytest.raises(yt_dlp.utils.DownloadError):
_run(tmp_path, None, _GROUP_LINK)
assert web["opened"] == [] # signed out the page is the login wall: nothing to read
assert web["extracted"] == [_GROUP_LINK]
def test_the_group_post_link_forms_are_recognised():
for url in (
_GROUP_LINK,
f"https://www.facebook.com/groups/983554479435596/posts/{_POST}/",
f"https://m.facebook.com/groups/983554479435596/permalink/{_POST}/?",
f"https://www.facebook.com/groups/chatgptmagyarul/posts/{_POST}",
):
assert adapters.facebook_group_post_id(url) == _POST, url
for url in (
_URL,
f"https://www.facebook.com/watch/?v={_POST_VIDEO}",
f"https://facebook.com.evil/groups/x/permalink/{_POST}/",
"https://www.facebook.com/groups/983554479435596/",
):
assert adapters.facebook_group_post_id(url) is None, url
assert adapters.SourceAdapter().resolve_url(_GROUP_LINK, {"cookiefile": "/c"}) == _GROUP_LINK
def test_only_the_single_video_of_the_posts_own_blocks_counts():
assert adapters.facebook_post_video_id(_page(_POST_VIDEO), _POST) == _POST_VIDEO
# No video in the post, although the page has one elsewhere.
assert adapters.facebook_post_video_id(_page(), _POST) is None
# Two videos: guessing between them would be the wrong-video bug again.
assert adapters.facebook_post_video_id(_page(_POST_VIDEO, "2063302677911049"), _POST) is None
# A longer number that merely STARTS with the post id is another object.
assert adapters.facebook_post_video_id(_page(_POST_VIDEO, post=_POST + "0"), _POST) is None
+63
View File
@@ -0,0 +1,63 @@
"""The scheduled RSS poll and recent backfill cover FOLLOWED channels only (owner decision, 2026-09-17).
A channel row also comes from a search result, a playlist item or a subscription somebody dropped,
and nothing deletes those rows. Polling all of them was 582 of 979 channels on prod with no follower,
three of them dead feeds answering 404 every 20 minutes. Followed = subscribed by someone, or explored
by someone. The admin card counting "channels to sync" must use the same selection, or it counts
channels the backfill will never touch. DB lane.
"""
from app.models import Channel, ExploredChannel, Subscription
from app.sync import runner
def _channels(db, make_user):
user = make_user()
for cid in ("UC_sub", "UC_explored", "UC_search", "UC_playlist"):
db.add(Channel(id=cid, title=cid, from_search=cid == "UC_search"))
db.flush()
db.add(Subscription(user_id=user.id, channel_id="UC_sub"))
db.add(ExploredChannel(user_id=user.id, channel_id="UC_explored"))
db.commit()
return user
def test_the_rss_poll_skips_channels_nobody_follows(db, make_user, monkeypatch):
_channels(db, make_user)
polled: list[str] = []
monkeypatch.setattr(runner, "fetch_channel_feed", lambda cid: polled.append(cid) or [])
monkeypatch.setattr(runner, "apply_rss_feed", lambda db, ch, entries: 0)
assert runner.run_rss_poll(db) == {"new": 0, "failed": 0}
assert sorted(polled) == ["UC_explored", "UC_sub"]
def test_the_recent_backfill_skips_channels_nobody_follows(db, make_user, monkeypatch):
user = _channels(db, make_user)
done: list[str] = []
class _Client:
def __init__(self, *a):
pass
def __enter__(self):
return self
def __exit__(self, *a):
return False
monkeypatch.setattr(runner, "get_service_user", lambda db: user)
monkeypatch.setattr(runner, "YouTubeClient", _Client)
monkeypatch.setattr(runner.quota, "backfill_reserve_reached", lambda: False)
monkeypatch.setattr(
runner, "backfill_channel_recent", lambda db, yt, ch: done.append(ch.id) or 0
)
monkeypatch.setattr(runner, "reconcile_full_history", lambda db: 0)
runner.run_recent_backfill(db)
assert sorted(done) == ["UC_explored", "UC_sub"]
def test_the_dashboard_counts_the_channels_the_backfill_will_process(db, make_user, as_user):
_channels(db, make_user)
admin = make_user(role="admin")
queue = as_user(admin).get("/api/admin/scheduler").json()["queue"]
assert queue["channels_recent_pending"] == 2
+21
View File
@@ -158,3 +158,24 @@ def test_a_normal_cut_list_is_untouched():
segs = [{"start_s": 0, "end_s": 5}, {"start_s": 10, "end_s": 12}]
out = editmod.normalize_edit_spec({"segments": segs})
assert len(out["segments"]) == 2
def test_audio_versions_its_cache_identity_apart_from_video(monkeypatch):
# The .mp3/.m4a-stays-itself fix must re-derive cached remuxed audio WITHOUT re-downloading every
# cached video: bumping one version moves only its own mode's signature.
audio, video = {"mode": "a", "audio_format": "mp3"}, {"mode": "av"}
before = formats.format_sig(audio), formats.format_sig(video)
monkeypatch.setattr(formats, "_AUDIO_SIG_VERSION", formats._AUDIO_SIG_VERSION + 1)
assert formats.format_sig(audio) != before[0]
assert formats.format_sig(video) == before[1]
monkeypatch.setattr(formats, "_SIG_VERSION", formats._SIG_VERSION + 1)
assert formats.format_sig(video) != before[1]
def test_the_mp3_fix_moved_the_audio_cache_identity(monkeypatch):
# Before the fix every mode hashed with version 4, so an audio signature still at 4 would find the
# cached mp3-remuxed-into-mp4 assets iPad refuses (prod job 268) instead of re-deriving them.
audio = {"mode": "a", "audio_format": "mp3"}
current = formats.format_sig(audio)
monkeypatch.setattr(formats, "_AUDIO_SIG_VERSION", 4)
assert formats.format_sig(audio) != current
+97
View File
@@ -0,0 +1,97 @@
"""`users.last_login_at` (E22 S1), DB-lane.
The roadmap assumed stamping it in `_establish_session` would cover Google and password alike. It
would not have: only password login called it, and the Google callback carried its own copy of the
same four lines. So these pin BOTH doors through the real routes — plus the two that must not stamp
(a refused password, the shared demo account) and the admin list that shows it.
"""
from datetime import datetime, timedelta, timezone
from app.config import settings
from app.models import DemoWhitelist
from app.routes import auth as auth_routes
_PW = "correct horse battery"
def _recent(ts: datetime | None) -> bool:
return ts is not None and datetime.now(timezone.utc) - ts < timedelta(minutes=1)
def test_a_password_sign_in_stamps_last_login(client, db, make_user):
user = make_user(email="pw@example.com", password=_PW)
assert user.last_login_at is None
r = client.post("/auth/password-login", json={"email": "pw@example.com", "password": _PW})
assert r.status_code == 200
db.refresh(user)
assert _recent(user.last_login_at)
def test_a_refused_password_does_not_stamp(client, db, make_user):
user = make_user(email="wrong@example.com", password=_PW)
r = client.post(
"/auth/password-login", json={"email": "wrong@example.com", "password": "not it"}
)
assert r.status_code == 401
db.refresh(user)
assert user.last_login_at is None
class _FakeGoogle:
"""authlib's client, reduced to the one call the callback makes after Google redirects back."""
def __init__(self, userinfo: dict):
self._userinfo = userinfo
async def authorize_access_token(self, _request):
return {"userinfo": self._userinfo, "access_token": "at", "scope": "openid email profile"}
def test_a_google_sign_in_stamps_last_login(client, db, make_user, monkeypatch):
from cryptography.fernet import Fernet
from app import security
# The callback stores the OAuth grant encrypted; a throwaway key stands in for TOKEN_ENCRYPTION_KEY.
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
user = make_user(email="g@example.com", password=None, google_sub="sub-g")
monkeypatch.setattr(settings, "admin_emails", "g@example.com") # the bootstrap allow-list
fake = _FakeGoogle({"sub": "sub-g", "email": "g@example.com", "email_verified": True})
monkeypatch.setattr(auth_routes, "google_oauth", lambda _db: fake)
r = client.get("/auth/callback", follow_redirects=False)
assert r.status_code in (302, 307) and r.headers["location"] == "/"
db.refresh(user)
assert _recent(user.last_login_at)
# …and it is a real sign-in, not only a stamp: the session now answers as this account.
assert client.get("/api/me").json()["id"] == user.id
def test_the_demo_entry_does_not_stamp_the_shared_account(client, db):
db.add(DemoWhitelist(email="visitor@example.com"))
db.commit()
r = client.post("/auth/demo", json={"email": "visitor@example.com"})
assert r.json() == {"authenticated": True}
demo = auth_routes.get_or_create_demo_user(db)
db.refresh(demo)
assert demo.last_login_at is None
def test_the_admin_user_list_carries_last_login(as_user, db, make_user):
admin = make_user(email="boss@example.com", role="admin")
never = make_user(email="never@example.com")
admin.last_login_at = datetime(2026, 9, 16, 8, 30, tzinfo=timezone.utc)
db.commit()
rows = {u["email"]: u for u in as_user(admin).get("/api/admin/users").json()}
assert rows["boss@example.com"]["last_login_at"].startswith("2026-09-16T08:30")
assert rows[never.email]["last_login_at"] is None
+320
View File
@@ -0,0 +1,320 @@
"""E22 S2 — the one logging configuration, its redaction, and the request line that replaced
uvicorn's access log.
Redaction is the part worth guarding hardest: it is applied at the HANDLER, so these cases are the
only thing standing between a future call site and a token in a log we attach to a Gitea issue.
The request-line cases pin what uvicorn's access log got wrong — the query string (`?account=1`,
`?token=…`) was printed on every request, and nothing correlated two lines of the same request.
Pure logic + an in-process ASGI app: no database.
"""
import json
import logging
from pathlib import Path
import pytest
from fastapi import FastAPI, Request
from fastapi.testclient import TestClient
from app import logs
from app.config import LogLevel, choices_of
class _Sink(logging.Handler):
"""Captures records AND their formatted JSON, with the same filter/formatter the real stdout
handler uses — so a test sees exactly the bytes a deployment would write."""
def __init__(self):
super().__init__(level=logging.NOTSET)
self.lines: list[str] = []
self.records: list[logging.LogRecord] = []
self.setFormatter(logs.JsonFormatter())
self.addFilter(logs.ContextFilter())
def emit(self, record):
self.lines.append(self.format(record))
self.records.append(record)
def objects(self) -> list[dict]:
return [json.loads(line) for line in self.lines]
@pytest.fixture
def sink():
"""Attached to `siftlode` (not root) so a `configure_logging` call inside a test — which
replaces the ROOT handlers — can't silently detach it mid-test."""
logger = logging.getLogger("siftlode")
handler = _Sink()
previous = logger.level
logger.addHandler(handler)
logger.setLevel(logs.TRACE)
try:
yield handler
finally:
logger.removeHandler(handler)
logger.setLevel(previous)
# --- redaction ------------------------------------------------------------------------------
@pytest.mark.parametrize(
"text, must_not_contain",
[
("user signed in: someone@example.com", "someone@example.com"),
("callback https://app/auth/callback?code=4/0AY&state=xyz", "4/0AY"),
("mail link https://app/#verify=abc123def", "abc123def"),
("share https://app/s?token=SHARETOKEN", "SHARETOKEN"),
("reset https://app/#reset=RESETTOKEN", "RESETTOKEN"),
],
)
def test_free_text_secrets_never_reach_the_line(text, must_not_contain):
assert must_not_contain not in logs.redact_text(text)
def test_a_key_that_names_a_secret_is_dropped_whole():
# Not truncated to a prefix: a prefix is still a leak and is useless for triage anyway.
out = logs.redact(
{
"password": "hunter2",
"current_password": "hunter2",
"plex_server_token": "abc",
"Cookie": "session=xyz",
"authorization": "Bearer abc",
"nested": {"refresh_token": "rt", "keep": "visible"},
"list": [{"api_key": "k"}],
"count": 7,
}
)
assert out["password"] == logs.REDACTED
assert out["current_password"] == logs.REDACTED
assert out["plex_server_token"] == logs.REDACTED
assert out["Cookie"] == logs.REDACTED
assert out["authorization"] == logs.REDACTED
assert out["nested"]["refresh_token"] == logs.REDACTED
assert out["nested"]["keep"] == "visible" # only the named keys go
assert out["list"][0]["api_key"] == logs.REDACTED
assert out["count"] == 7
def test_an_email_in_an_attribute_value_is_scrubbed_too():
assert logs.redact({"to": "a.b+tag@mail.example.org"})["to"] == logs.REDACTED_EMAIL
# The SAME cases frontend/src/lib/log.contract.test.ts runs against the browser's copy of redact():
# the one thing that keeps the two logs' idea of "safe to keep" from drifting apart.
_REDACTION_CASES = json.loads(
(Path(__file__).parent / "fixtures" / "log_redaction_cases.json").read_text(encoding="utf-8")
)["cases"]
@pytest.mark.parametrize("case", _REDACTION_CASES, ids=[c["name"] for c in _REDACTION_CASES])
def test_the_shared_redaction_cases(case):
assert logs.redact(case["value"], key=case["key"]) == case["expected"]
def test_redaction_runs_on_the_message_the_attrs_and_the_traceback(sink):
log = logging.getLogger("siftlode.test")
try:
raise RuntimeError("boom for someone@example.com")
except RuntimeError:
log.exception(
"failed for %s",
"someone@example.com",
extra=logs.event("test.failed", password="hunter2", url="https://x/?token=T0K3N"),
)
line = sink.lines[-1]
assert "someone@example.com" not in line
assert "hunter2" not in line
assert "T0K3N" not in line
payload = json.loads(line)
assert payload["attrs"]["password"] == logs.REDACTED
assert "RuntimeError" in payload["exc"] # the diagnosis survives the scrubbing
# --- event shape ----------------------------------------------------------------------------
def test_every_line_carries_the_agreed_fields(sink):
logging.getLogger("siftlode.test").info("hello", extra=logs.event("test.hello", n=1))
payload = sink.objects()[-1]
for field in ("ts", "level", "sev", "source", "logger", "msg", "event", "version"):
assert field in payload, field
assert payload["level"] == "INFO"
assert payload["sev"] == 9
assert payload["event"] == "test.hello"
assert payload["attrs"] == {"n": 1}
def test_a_plain_log_call_still_produces_a_well_formed_line(sink):
"""An event code is optional — 125 existing call sites have none yet (E22.5 adds them), and they
must not be worse off than before this module."""
logging.getLogger("siftlode.test").warning("just a message")
payload = sink.objects()[-1]
assert payload["msg"] == "just a message"
assert payload["level"] == "WARNING"
assert "event" not in payload
def test_a_broken_format_call_reports_itself_instead_of_raising():
"""A record is formatted directly rather than logged: a genuinely broken call ALSO breaks
pytest's own capture handler (stdlib formatting raises there too), which would fail the test for
the wrong reason. What is under test is our formatter's refusal to raise."""
record = logging.LogRecord(
"siftlode.test", logging.INFO, __file__, 0, "two args %s %s", ("only-one",), None
)
payload = json.loads(logs.JsonFormatter().format(record))
assert "log formatting failed" in payload["msg"]
def test_bound_correlation_appears_and_is_released(sink):
log = logging.getLogger("siftlode.test")
with logs.bind(job_id=42):
log.info("inside")
log.info("outside")
assert sink.objects()[-2]["job_id"] == 42
assert "job_id" not in sink.objects()[-1]
def test_every_selectable_level_resolves_to_a_real_logging_level():
# The picker's options come from `LogLevel`; each must be a level `logging` knows, TRACE included.
assert [logs.resolve_level(n) for n in choices_of(LogLevel)] == [
logs.TRACE,
logging.DEBUG,
logging.INFO,
logging.WARNING,
logging.ERROR,
]
with pytest.raises(ValueError):
logs.resolve_level("nonsense")
def test_configure_logging_silences_the_uvicorn_access_log_and_keeps_ours_propagating():
logs.configure_logging("api")
assert logging.getLogger("uvicorn.access").getEffectiveLevel() >= logging.CRITICAL
# propagate stays on so every line reaches the ONE root handler — and so pytest's caplog works.
assert logging.getLogger("siftlode.anything").propagate is True
assert len(logging.getLogger().handlers) == 1
# --- the request line -----------------------------------------------------------------------
def _client() -> TestClient:
app = FastAPI()
app.middleware("http")(logs.log_requests)
@app.get("/api/videos/{video_id}")
def _video(video_id: str, q: str | None = None):
return {"ok": video_id}
@app.get("/healthz")
def _health():
return {"ok": True}
@app.get("/boom")
def _boom():
raise RuntimeError("kaboom")
return TestClient(app, raise_server_exceptions=False)
def test_the_request_line_has_the_route_template_and_no_query_string(sink):
logging.getLogger("siftlode.http").setLevel(logs.TRACE)
r = _client().get("/api/videos/abc123?q=cats&token=SECRET")
assert r.status_code == 200
payload = sink.objects()[-1]
assert payload["event"] == "http.request"
# The TEMPLATE, so every video request groups into one queryable path...
assert payload["attrs"]["path"] == "/api/videos/{video_id}"
# ...and the query string — uvicorn's access log printed it, `?account=1` and all — is gone.
assert "SECRET" not in sink.lines[-1]
assert "cats" not in sink.lines[-1]
assert payload["attrs"]["status"] == 200
assert isinstance(payload["attrs"]["duration_ms"], float)
def test_anonymous_request_lines_are_capped_per_client_for_storage_but_still_logged(sink):
# A scanner's 404s are WARNINGs, and WARNINGs are stored. Past the cap they stay on stdout but are
# marked not-for-storage; a signed-in request is never capped. No address appears in any line.
logs._anonymous_lines.reset()
client = _client()
for _ in range(logs.ANONYMOUS_LINES_PER_MINUTE + 3):
client.get("/wp-login.php")
lines = sink.objects()[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
assert len(lines) == logs.ANONYMOUS_LINES_PER_MINUTE + 3 # stdout keeps every one
records = sink.records[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
stored = [getattr(r, logs.PERSIST, True) for r in records]
assert stored.count(False) == 3 and all(stored[: logs.ANONYMOUS_LINES_PER_MINUTE])
assert "testclient" not in "".join(sink.lines) # the client key never reaches the line
logs._anonymous_lines.reset()
def test_a_signed_in_request_is_never_capped_and_names_its_account(sink):
logs._anonymous_lines.reset()
app = FastAPI()
app.middleware("http")(logs.log_requests)
@app.get("/api/me")
def _me(request: Request):
request.state.user_id = 42 # what the auth dependency does
return {}
client = TestClient(app)
for _ in range(logs.ANONYMOUS_LINES_PER_MINUTE + 3):
client.get("/api/me")
records = sink.records[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
assert all(getattr(r, logs.PERSIST, True) for r in records)
assert sink.objects()[-1]["user_id"] == 42 and "user_id" not in sink.objects()[-1]["attrs"]
def test_the_request_id_is_echoed_and_shared_by_every_line_of_that_request(sink):
client = _client()
r = client.get("/api/videos/abc123")
rid = r.headers["X-Request-Id"]
assert rid and sink.objects()[-1]["request_id"] == rid
def test_the_route_template_goes_back_to_the_browser_but_never_a_raw_path(sink):
# The browser logs its own copy of each request, grouped by this template (frontend lib/http.ts).
client = _client()
assert client.get("/api/videos/SeCrEt-VaLuE").headers["X-Route"] == "/api/videos/{video_id}"
# Unmatched: the only template there is the raw path, which may carry a value — so none is sent.
assert "X-Route" not in client.get("/api/share/SeCrEt-VaLuE").headers
def test_a_client_supplied_id_is_reused_only_when_it_is_well_formed(sink):
client = _client()
good = client.get("/api/videos/x", headers={"X-Request-Id": "web-0123456789ab"})
assert good.headers["X-Request-Id"] == "web-0123456789ab"
# An unbounded header would be a way to write arbitrary text into the log; short/odd ones are
# replaced with our own id rather than trusted.
for bogus in ("short", "has spaces and \n newline", "x" * 200):
r = client.get("/api/videos/x", headers={"X-Request-Id": bogus})
assert r.headers["X-Request-Id"] != bogus
def test_the_healthcheck_is_logged_at_debug_so_it_cannot_drown_the_log(sink):
http = logging.getLogger("siftlode.http")
http.setLevel(logs.TRACE)
_client().get("/healthz")
assert sink.objects()[-1]["level"] == "DEBUG"
# ...but it is still logged when something is wrong with it: the floor drops, nothing is hidden.
http.setLevel(logging.INFO)
before = len(sink.lines)
_client().get("/healthz")
assert len(sink.lines) == before
def test_a_failing_request_is_logged_by_status_and_an_unhandled_error_is_logged_at_all(sink):
client = _client()
client.get("/api/videos/missing/deeper") # 404 from the router
assert sink.objects()[-1]["level"] == "WARNING"
client.get("/boom")
payload = sink.objects()[-1]
# The request that most needs a line is the one that raised: uvicorn's access log wrote it, but
# our middleware sees the exception instead of a response, so it has to log it explicitly.
assert payload["level"] == "ERROR"
assert payload["event"] == "http.request"
assert "RuntimeError" in payload["exc"]
+260
View File
@@ -0,0 +1,260 @@
"""E22 S3 — the `log_events` sink: what reaches the table, in what shape, and what it refuses to do.
The sink is driven synchronously (`write_pending()` on the test's own thread) so nothing here waits on
a timer. It is attached to a private logger rather than the root, for the same reason test_logs.py's
capture handler is: a `configure_logging` call elsewhere replaces the root handlers.
DB lane (`db` fixture): the rows are read back from a real Postgres.
"""
import logging
import threading
from datetime import datetime, timedelta, timezone
import pytest
from fastapi import BackgroundTasks
from sqlalchemy import create_engine, func, select
from app import logs, logsink, scheduler
from app.auth import purge_user
from app.db import engine
from app.models import LogEvent
@pytest.fixture
def stored(db):
"""A LogSink on the logger `siftlode.sinktest`, and a reader for the rows it wrote."""
sink = logsink.LogSink(engine)
logger = logging.getLogger("siftlode.sinktest")
logger.addHandler(sink)
logger.setLevel(logs.TRACE)
def rows() -> list[LogEvent]:
sink.write_pending()
db.expire_all()
return list(db.execute(select(LogEvent).order_by(LogEvent.id)).scalars())
try:
yield sink, logger, rows
finally:
logger.removeHandler(sink)
def test_warnings_and_request_lines_are_stored_and_everything_else_is_not(stored):
_sink, log, rows = stored
log.info("an ordinary info line")
log.debug("GET /healthz 200", extra=logs.event("http.request", path="/healthz", status=200))
log.info("GET /api/feed 200", extra=logs.event("http.request", path="/api/feed", status=200))
log.warning("scanner", extra={**logs.event("http.request", status=404), logs.PERSIST: False})
log.warning("disk nearly full")
assert [r.msg for r in rows()] == ["GET /api/feed 200", "disk nearly full"]
def test_a_row_is_redacted_exactly_like_the_stdout_line(stored):
_sink, log, rows = stored
try:
raise ValueError("reset link https://x.test/#reset=abc123")
except ValueError:
log.exception(
"sign-in failed for someone@example.com",
extra=logs.event("auth.failed", password="hunter2", reason="bad_password"),
)
(row,) = rows()
assert row.level == "ERROR" and row.sev == logs.SEVERITY["ERROR"]
assert row.event == "auth.failed"
assert "someone@example.com" not in row.msg and logs.REDACTED_EMAIL in row.msg
assert row.attrs == {"password": logs.REDACTED, "reason": "bad_password"}
assert "abc123" not in row.exc and "ValueError" in row.exc
assert row.source == logs.current_source() and row.version
def test_correlation_is_captured_when_the_line_is_logged_not_when_it_is_written(stored):
_sink, log, rows = stored
with logs.bind(job_id=7, request_id="req-0123456789"):
log.warning("inside the job")
# The writer runs later, outside the bind — the row still belongs to job 7.
(row,) = rows()
assert (row.job_id, row.request_id) == (7, "req-0123456789")
def test_a_flood_of_one_message_is_stored_up_to_the_limit_and_then_summarised(stored):
sink, log, rows = stored
limit = logsink.REPEATS_PER_MINUTE
for i in range(limit + 70):
log.warning("RSS poll skipped for channel %s: HTTP 404", f"UC{i}")
log.warning("a different message")
assert len(rows()) == limit + 1 # the flood capped; the other message untouched
sink._repeats.started -= logsink.REPEAT_WINDOW_SECONDS # the minute is over
notice = rows()[-1]
assert notice.event == "log.repeats_suppressed"
assert notice.attrs["suppressed"] == 70
assert notice.attrs["template"] == "RSS poll skipped for channel %s: HTTP 404"
def test_request_lines_are_never_treated_as_a_flood(stored):
_sink, log, rows = stored
for _ in range(logsink.REPEATS_PER_MINUTE + 5):
log.info("%s %s %s", "GET", "/api/feed", 200, extra=logs.event("http.request"))
assert len(rows()) == logsink.REPEATS_PER_MINUTE + 5
def test_a_full_queue_drops_and_says_how_many(db, monkeypatch):
monkeypatch.setattr(logsink, "QUEUE_MAX", 3)
sink = logsink.LogSink(engine)
log = logging.getLogger("siftlode.sinktest.small")
log.addHandler(sink)
try:
for i in range(5):
log.warning("message number %d", i)
finally:
log.removeHandler(sink)
assert sink.write_pending() == 4
notice = db.execute(select(LogEvent).where(LogEvent.event == "log.sink.dropped")).scalar_one()
assert notice.attrs == {"dropped": 2}
def test_an_unreachable_database_costs_the_batch_not_the_caller(capsys):
dead = create_engine(
"postgresql+psycopg://x:y@127.0.0.1:1/none", connect_args={"connect_timeout": 1}
)
sink = logsink.LogSink(dead)
log = logging.getLogger("siftlode.sinktest.dead")
log.addHandler(sink)
try:
log.error("this has nowhere to go") # must not raise
finally:
log.removeHandler(sink)
assert sink.write_pending() == 0
assert "could not store 1 log events" in capsys.readouterr().err
def test_the_writer_thread_does_not_store_its_own_records(stored):
sink, log, rows = stored
sink._writer = threading.current_thread() # as if this line came from inside the writer
log.error("a driver warning raised while inserting")
sink._writer = None
assert rows() == []
def test_gc_keeps_the_retention_window_and_never_keeps_forever(db):
now = datetime.now(timezone.utc)
def add(age: timedelta) -> None:
db.add(
LogEvent(ts=now - age, level="WARNING", sev=13, source="api", logger="t", msg=str(age))
)
for age in (timedelta(hours=1), timedelta(days=2), timedelta(days=8)):
add(age)
db.commit()
assert logsink.gc(db, 7) == 1
assert logsink.gc(db, 0) == 1 # 0 is not "forever": stored logs always expire (floor: 1 day)
db.commit()
assert [r.msg for r in db.execute(select(LogEvent)).scalars()] == [str(timedelta(hours=1))]
def test_log_gc_is_a_scheduled_job():
assert "log_gc" in scheduler.JOB_FUNCS and scheduler.JOB_INTERVALS["log_gc"] >= 1
def test_deleting_an_account_deletes_its_stored_log_events(db, make_user):
gone, kept = make_user(), make_user()
for uid in (gone.id, kept.id, gone.id):
db.add(
LogEvent(
ts=datetime.now(timezone.utc),
level="INFO",
sev=9,
source="api",
logger="t",
msg="x",
user_id=uid,
)
)
db.commit()
purge_user(db, gone, BackgroundTasks())
assert [r.user_id for r in db.execute(select(LogEvent)).scalars()] == [kept.id]
def test_an_erased_accounts_late_lines_are_stored_without_it(stored, db, make_user):
"""The DELETE in purge_user cannot reach a row still on its way: one queued before the erasure,
or the erasing request's own access line, logged after the handler committed."""
_sink, log, rows = stored
user = make_user()
# `logs._erased` outlives the test; conftest's `_reset_log_erasures` clears it before each one.
log.warning("queued before the erasure", extra={"user_id": user.id})
purge_user(db, user, BackgroundTasks())
log.info("DELETE /api/me/account 200", extra={**logs.event("http.request"), "user_id": user.id})
assert [(r.msg, r.user_id) for r in rows()] == [
("queued before the erasure", None),
("DELETE /api/me/account 200", None),
]
def test_every_path_into_the_table_drops_an_erased_account():
"""The two behavioural tests above cover the two writers there are today (this sink and
`POST /api/diagnostics/events`). This one covers the THIRD one nobody has written yet: a new
`insert(LogEvent)` that forgets the filter would leave an erased account's rows named, and no
test of an existing path can notice. Source-level on purpose — it is a rule about call sites."""
import pathlib
root = pathlib.Path(__file__).resolve().parents[1] / "app"
unfiltered = [
f"{path.relative_to(root)}:{n}"
for path in root.rglob("*.py")
for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1)
if "insert(LogEvent)" in line and "without_erased" not in line
]
assert unfiltered == [], f"insert(LogEvent) without logsink.without_erased at {unfiltered}"
def test_a_nul_character_costs_nothing_but_itself(stored):
"""Postgres refuses NUL in text and JSONB, and a refused row fails its whole batch — every other
user's rows with it. `GET /assets/%00` logs exactly that."""
_sink, log, rows = stored
log.warning("before")
log.warning("GET /assets/\x00 404", extra=logs.event("http.request", path="/assets/\x00"))
log.warning("after")
before, nul, after = rows()
assert (before.msg, after.msg) == ("before", "after")
assert nul.msg == "GET /assets/� 404" and nul.attrs["path"] == "/assets/�"
def test_the_writer_drains_what_was_queued_when_it_is_stopped(db):
sink = logsink.LogSink(engine)
log = logging.getLogger("siftlode.sinktest.drain")
log.addHandler(sink)
try:
for i in range(logsink.BATCH_MAX + 5):
log.warning(f"line {i}") # distinct templates: no flood limiter involved
finally:
log.removeHandler(sink)
sink._stopping.set()
sink._run() # stopped before its first wait: everything goes through the drain
assert (
db.execute(select(func.count()).select_from(LogEvent)).scalar_one() == logsink.BATCH_MAX + 5
)
def test_the_running_sink_keeps_the_base_level_for_everyone_the_elevation_is_not_for(
db, monkeypatch
):
"""`logsink.start()` as production runs it. One user's elevation lowers the ROOT logger, so an
ordinary request line below the base level reaches every handler — the sink's own level gate is
what keeps it out of the table, as stdout's keeps it off the terminal."""
root = logging.getLogger()
saved = (logs._base_level, dict(logs._elevations), logs._root_applied, root.level)
monkeypatch.setattr(logsink.settings, "log_db_sink", True)
logs.set_level("WARNING")
logs._track_elevation(1, logging.DEBUG, datetime.now(timezone.utc) + timedelta(hours=1))
log = logging.getLogger("siftlode.sinktest.live")
try:
logsink.start()
log.info("GET /api/feed 200", extra={**logs.event("http.request"), "user_id": 2})
log.warning("a warning")
finally:
logsink.stop()
logs._base_level, logs._root_applied = saved[0], saved[2]
logs._elevations.clear()
logs._elevations.update(saved[1])
root.setLevel(saved[3])
assert [r.msg for r in db.execute(select(LogEvent)).scalars()] == ["a warning"]
+80
View File
@@ -0,0 +1,80 @@
"""The password unlock of a shared /watch link (routes/public.py), DB-lane.
Guesses are slowed per viewer (the link AND the client — an IPv6 client by its /64), under a per-link
ceiling on wrong passwords, so a client guessing wrong shuts out only itself until it spreads its guesses
over as many networks as the ceiling allows.
"""
import pytest
from fastapi.testclient import TestClient
from app.config import settings
from app.downloads import probe as probemod
from app.main import app
from app.ratelimit import RateLimiter
from app.routes import public
_PW = "open sesame"
@pytest.fixture
def unlock_url(as_user, make_user, tmp_path, monkeypatch):
monkeypatch.setattr(settings, "download_root", str(tmp_path))
monkeypatch.setattr(probemod, "probe_media", lambda _p: ("h264", "aac", 320, 240, 5))
owner = as_user(make_user())
job = owner.post(
"/api/downloads/upload", files={"file": ("clip.mp4", b"fake-video", "video/mp4")}
)
assert job.status_code == 200, job.text
link = owner.post(f"/api/downloads/{job.json()['id']}/links", json={"password": _PW})
assert link.status_code == 200, link.text
return f"/api/public/watch/{link.json()['token']}/unlock"
def _from(ip: str) -> TestClient:
# Its own socket peer: with no trusted proxy configured, that is the client's IP.
return TestClient(app, client=(ip, 50000))
def test_a_link_holder_without_the_password_cannot_lock_out_its_viewers(unlock_url):
# The limit used to be keyed on the link alone: ten wrong guesses every five minutes kept every
# viewer out, the ones with the password included.
guesser = _from("203.0.113.5")
for _ in range(10):
assert guesser.post(unlock_url, json={"password": "guess"}).status_code == 403
assert guesser.post(unlock_url, json={"password": "guess"}).status_code == 429
assert _from("198.51.100.7").post(unlock_url, json={"password": _PW}).status_code == 200
def test_an_ipv6_client_is_one_viewer_across_its_64(unlock_url):
# One subscriber gets a whole /64: keyed on the full address, it could rotate through endless viewers.
for _ in range(10):
wrong = _from("2001:db8:1:2::5").post(unlock_url, json={"password": "guess"})
assert wrong.status_code == 403
assert _from("2001:db8:1:2::6").post(unlock_url, json={"password": _PW}).status_code == 429
assert _from("2001:db8:1:3::5").post(unlock_url, json={"password": _PW}).status_code == 200
def test_a_token_naming_no_link_leaves_nothing_in_the_limiter(unlock_url):
# The limiter used to record the attempt before looking the link up: posting made-up tokens grew it by
# an entry each, kept for the whole window.
keys = len(public._unlock_limit._hits)
for n in range(5):
missing = _from("203.0.113.5").post(unlock_url.replace("/watch/", f"/watch/x{n}"), json={})
assert missing.status_code == 404
assert len(public._unlock_limit._hits) == keys
def test_wrong_passwords_from_many_clients_still_reach_the_links_ceiling(unlock_url, monkeypatch):
monkeypatch.setattr(public, "_unlock_failures", RateLimiter(max_events=3, window_seconds=300))
viewer = _from("198.51.100.7")
for _ in range(5): # unlocking with the password never adds to the ceiling
assert viewer.post(unlock_url, json={"password": _PW}).status_code == 200
for n in range(3):
wrong = _from(f"203.0.113.{n}").post(unlock_url, json={"password": "guess"})
assert wrong.status_code == 403
late = _from("203.0.113.9").post(unlock_url, json={"password": "guess"})
assert late.status_code == 429
+13
View File
@@ -57,6 +57,19 @@ def test_reset_clears_every_key():
assert lim.allow("b") is True
def test_exhausted_reports_a_full_window_without_counting_an_attempt():
# The asking half of a limit that counts only some outcomes (a share link's unlock counts only
# wrong passwords toward its ceiling): asking must never use up the window it asks about.
lim = RateLimiter(max_events=2, window_seconds=300)
for _ in range(5):
assert lim.exhausted("link") is False
assert lim.allow("link") is True
assert lim.exhausted("link") is False
assert lim.allow("link") is True
assert lim.exhausted("link") is True
def test_expired_attempts_fall_out_of_the_window():
# A zero-length window means every previous attempt is already outside it.
lim = RateLimiter(max_events=1, window_seconds=0)
+53 -3
View File
@@ -1,6 +1,8 @@
"""Guards that stop one bad YouTube answer from cascading: the subscription prune guard, the
playlist-push circuit breaker, and RSS telling a failure apart from an empty feed."""
import logging
import httpx
import pytest
@@ -120,7 +122,7 @@ class TestRssPollResult:
# One dead channel among healthy ones: counted, but the run itself is still a run.
def fetch(cid):
if cid == "UC_dead":
raise rss.RssError("404")
raise rss.RssError("404", "http_404")
return [{"id": "v"}]
out = self._run(
@@ -143,6 +145,54 @@ class TestRssPollResult:
)
assert out == {"new": 1, "failed": 1}, "an unstored channel must not read as ok"
def test_many_failures_write_one_warning_with_counts_not_one_per_channel(
self, monkeypatch, caplog
):
# YouTube's RSS outage of 2026-09-16: ~900 of 979 feeds failed on every 20-minute poll and
# each wrote its own WARNING — 11k lines in four hours, which the log_events table would
# have stored too. One summary per poll, per-channel detail at DEBUG.
channels = [f"UC{i}" for i in range(40)]
def fetch(cid):
n = int(cid[2:])
if n < 30:
raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404", "http_404")
if n < 35:
raise rss.RssError(f"RSS fetch failed for {cid}: timed out", "transport")
return [{"id": "v"}]
caplog.set_level(logging.DEBUG, logger="siftlode.sync")
out = self._run(monkeypatch, fetch=fetch, apply_=lambda db, ch, e: 1, channels=channels)
assert out == {"new": 5, "failed": 35}
warnings = [r for r in caplog.records if r.levelno >= logging.WARNING]
assert len(warnings) == 1
assert warnings[0].event == "sync.rss.failed"
assert warnings[0].attrs["reasons"] == {"http_404": 30, "transport": 5}
assert len(warnings[0].attrs["examples"]) == runner.RSS_FAILURE_EXAMPLES
assert sum(r.levelno == logging.DEBUG for r in caplog.records) == 35
def test_a_failure_on_our_side_gets_one_traceback_per_poll_not_one_per_channel(
self, monkeypatch, caplog
):
# A read-only database fails every channel's apply identically. The first gets a traceback
# at ERROR; repeating it for each of 979 channels would be the RSS flood all over again.
def apply_(db, ch, entries):
raise RuntimeError("read-only transaction")
caplog.set_level(logging.DEBUG, logger="siftlode.sync")
with pytest.raises(RuntimeError):
self._run(
monkeypatch,
fetch=lambda cid: [{"id": "v"}],
apply_=apply_,
channels=("A", "B", "C"),
)
errors = [r for r in caplog.records if r.levelno == logging.ERROR]
assert len(errors) == 1 and errors[0].exc_info is not None
assert (
sum(r.levelno == logging.DEBUG and r.exc_info is not None for r in caplog.records) == 2
)
def test_a_quiet_hour_is_distinguishable_from_an_outage(self, monkeypatch):
quiet = self._run(monkeypatch, fetch=lambda cid: [], apply_=lambda db, ch, e: 0)
assert quiet == {"new": 0, "failed": 0} # same "new", different "failed" — that's the point
@@ -152,7 +202,7 @@ class TestRssPollResult:
# here recorded the scheduler run as "ok", with the outage only visible inside the summary
# text — a green pill for a run that did nothing at all.
def boom(cid):
raise rss.RssError("Connection refused")
raise rss.RssError("Connection refused", "transport")
with pytest.raises(RuntimeError, match="all 2 channels"):
self._run(monkeypatch, fetch=boom, apply_=lambda db, ch, e: 1)
@@ -162,7 +212,7 @@ class TestRssPollResult:
# The card has to say which: `HTTP 404` = fix the subscription, `Connection refused` =
# the egress is down.
def dead(cid):
raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404")
raise rss.RssError(f"RSS fetch for {cid} returned HTTP 404", "http_404")
with pytest.raises(RuntimeError, match="HTTP 404"):
self._run(monkeypatch, fetch=dead, apply_=lambda db, ch, e: 1, channels=("UC_dead",))
+93
View File
@@ -0,0 +1,93 @@
"""The sysconfig registry: its order is the admin page's tab order, and a closed-set knob is a CHOICE.
The admin Configuration page used to keep its own hand-written group order, which had already drifted
(`explore` was missing, so its tab sorted last). The page now follows SPECS, so SPECS has to read in
tab order: one run of keys per group.
A closed-set admin knob is a CHOICE, not free text (E22 S2 follow-up).
`log_level`, `search_source`, `download_layout` and the transcode `hw`/`preset` used to be free-text
fields: the admin could type anything and every consumer carried its own `or "auto").lower()` guess
at what a typo meant. Now the value set is a `Literal` in config.py, pydantic holds the env to it at
boot, sysconfig holds a stored override to it, and the admin page renders a picker from the same list.
The DB lane (`db` fixture) covers the store; the rest is pure logic.
"""
import pytest
from pydantic import ValidationError
from app import sysconfig
from app.config import Settings, choices_of
CHOICE_KEYS = {s.key for s in sysconfig.SPECS if s.type == "choice"}
def test_each_group_is_one_run_of_keys_so_the_registry_reads_in_tab_order():
runs = [
s.group
for i, s in enumerate(sysconfig.SPECS)
if i == 0 or sysconfig.SPECS[i - 1].group != s.group
]
assert len(runs) == len(set(runs)), runs
assert runs[:2] == ["access", "email"]
def test_every_closed_set_knob_is_a_choice_whose_options_come_from_config():
assert CHOICE_KEYS == {
"search_source",
"download_layout",
"log_level",
"plex_transcode_hw",
"plex_transcode_preset",
}
for s in sysconfig.SPECS:
field = Settings.model_fields[s.default_attr]
if s.type == "choice":
assert s.choices, s.key
assert field.default in s.choices, s.key
else:
# A Literal-typed setting exposed as "str" would be a free-text field again.
assert s.choices == () and choices_of(field.annotation) == (), s.key
def test_the_env_accepts_any_letter_case_and_rejects_an_unknown_value():
assert Settings(log_level="debug").log_level == "DEBUG"
assert Settings(search_source=" API ").search_source == "api"
with pytest.raises(ValidationError):
Settings(log_level="verbose")
class _Row:
def __init__(self, value):
self.value, self.encrypted = value, False
class _FakeDb:
"""Just enough of a Session for sysconfig's reads: one stored override row per key."""
def __init__(self, **rows):
self.rows = {k: _Row(v) for k, v in rows.items()}
def get(self, _model, key):
return self.rows.get(key)
def test_a_stored_value_reads_back_canonical_and_a_stale_one_falls_back_to_the_default():
assert sysconfig.get_str(_FakeDb(plex_transcode_hw="OFF"), "plex_transcode_hw") == "off"
# A value that is no longer (or never was) a choice is not passed through to the consumer.
assert sysconfig.get_str(_FakeDb(plex_transcode_hw="maybe"), "plex_transcode_hw") == "auto"
def test_describe_hands_the_picker_its_options():
items = {i["key"]: i for g in sysconfig.describe(_FakeDb())["groups"].values() for i in g}
assert items["log_level"]["choices"] == ["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"]
assert items["log_level"]["value"] == "INFO"
assert items["quota_daily_budget"]["choices"] == []
def test_set_value_stores_the_canonical_spelling_and_refuses_anything_else(db):
sysconfig.set_value(db, "search_source", "API")
assert sysconfig.get_str(db, "search_source") == "api"
with pytest.raises(ValueError, match="scrape, api"):
sysconfig.set_value(db, "search_source", "bing")
db.rollback()
+34 -6
View File
@@ -424,10 +424,36 @@ class TestResolveMediaFacts:
],
},
)
v, a, w, h = probemod.resolve_media_facts(Path("x"), {"width": None, "height": None})
v, a, w, h, _d = probemod.resolve_media_facts(Path("x"), {"width": None, "height": None})
assert (v, a, w, h) == ("h264", "aac", 720, 1280)
assert mediakind.derive(v, a) == mediakind.VIDEO # NOT 'file'
def test_the_duration_comes_from_the_file_when_info_has_none(self, monkeypatch):
# Facebook reel 257 (E11.5): yt-dlp gave no duration, the file is 128.4 s. A NULL duration hid
# the Library row's trim action.
self._ffprobe(
monkeypatch,
{
"format": {"format_name": "mov,mp4,m4a", "duration": "128.400000"},
"streams": [{"codec_type": "video", "codec_name": "h264"}],
},
)
assert probemod.resolve_media_facts(Path("x"), {"duration": None})[4] == 128
# The file wins over a differing info duration, like its codecs do.
assert probemod.resolve_media_facts(Path("x"), {"duration": 300})[4] == 128
def test_info_fills_the_duration_the_probe_could_not_read(self, monkeypatch):
self._ffprobe(
monkeypatch,
{
"format": {"format_name": "mov,mp4,m4a"},
"streams": [{"codec_type": "video", "codec_name": "h264"}],
},
)
assert probemod.resolve_media_facts(Path("x"), {"duration": 61.7})[4] == 61
self._ffprobe(monkeypatch, None)
assert probemod.resolve_media_facts(Path("x"), {"duration": 61.7})[4] == 61
def test_probe_wins_over_a_DIFFERING_populated_info(self, monkeypatch):
# Both the probe AND info are populated but DISAGREE (info carries a pre-remux/pre-merge codec).
# The finished file on disk is ground truth, so the probe wins outright — pins the precedence a
@@ -442,7 +468,7 @@ class TestResolveMediaFacts:
],
},
)
v, a, w, h = probemod.resolve_media_facts(
v, a, w, h, _d = probemod.resolve_media_facts(
Path("x"), {"vcodec": "vp9", "acodec": "opus", "width": 640, "height": 480}
)
assert (v, a, w, h) == ("h264", "aac", 1920, 1080) # probe, not info
@@ -466,7 +492,7 @@ class TestResolveMediaFacts:
],
},
)
v, a, w, h = probemod.resolve_media_facts(
v, a, w, h, _d = probemod.resolve_media_facts(
Path("x"), {"vcodec": "avc1", "acodec": "mp4a", "width": 1280, "height": 720}
)
assert v is None and a == "mp3" # info's avc1 did NOT win
@@ -482,7 +508,7 @@ class TestResolveMediaFacts:
"streams": [{"codec_type": "video", "codec_name": "png", "width": 8, "height": 8}],
},
)
v, a, w, h = probemod.resolve_media_facts(
v, a, w, h, _d = probemod.resolve_media_facts(
Path("x"), {"vcodec": "png", "width": 8, "height": 8}
)
assert v is None and mediakind.derive(v, a) == mediakind.FILE
@@ -490,7 +516,7 @@ class TestResolveMediaFacts:
def test_info_fills_in_only_when_ffprobe_could_not_read_the_file(self, monkeypatch):
# ffprobe failed entirely (None) → fall back to info so we never regress the working path.
self._ffprobe(monkeypatch, None)
v, a, w, h = probemod.resolve_media_facts(
v, a, w, h, _d = probemod.resolve_media_facts(
Path("x"), {"vcodec": "vp9", "acodec": "opus", "width": 1920, "height": 1080}
)
assert (v, a, w, h) == ("vp9", "opus", 1920, 1080)
@@ -498,6 +524,8 @@ class TestResolveMediaFacts:
def test_info_none_codec_string_is_treated_as_absent(self, monkeypatch):
# yt-dlp writes the literal "none" for a missing stream — it must not become a codec name.
self._ffprobe(monkeypatch, None) # probe failed → info path, where the "none" scrub applies
v, a, w, h = probemod.resolve_media_facts(Path("x"), {"vcodec": "none", "acodec": "aac"})
v, a, w, h, _d = probemod.resolve_media_facts(
Path("x"), {"vcodec": "none", "acodec": "aac"}
)
assert (v, a) == (None, "aac")
assert mediakind.derive(v, a) == mediakind.AUDIO
+48 -15
View File
@@ -51,7 +51,7 @@ def rec(monkeypatch):
monkeypatch.setattr(worker, "_reset_asset_pending", record("reset_asset"))
monkeypatch.setattr(worker, "_requeue_if_running", record("requeue"))
monkeypatch.setattr(worker, "_set_job", record("set_job"))
monkeypatch.setattr(worker, "_fail_running", record("fail"))
monkeypatch.setattr(worker, "_cleanup_staging", record("cleanup"))
monkeypatch.setattr(worker, "_stop_child", record("stop_child"))
monkeypatch.setattr(worker._stop, "is_set", lambda: False)
@@ -84,9 +84,9 @@ def test_no_progress_stall_kills_and_fails_the_job(monkeypatch, rec):
worker._run_supervised(1, 99)
names = _names(rec)
assert "stop_child" in names and "reset_asset" in names
set_job = next(c for c in rec if c[0] == "set_job")
assert set_job[2]["status"] == "error" and set_job[2]["error_code"] == "network"
assert "stalled" in set_job[2]["error"].lower()
fail = next(c for c in rec if c[0] == "fail")
assert fail[2]["error_code"] == "network"
assert "stalled" in fail[2]["error"].lower()
def test_route_pause_kills_but_keeps_the_route_status(monkeypatch, rec):
@@ -97,7 +97,7 @@ def test_route_pause_kills_but_keeps_the_route_status(monkeypatch, rec):
worker._run_supervised(1, 99)
names = _names(rec)
assert "stop_child" in names and "reset_asset" in names
assert "set_job" not in names and "requeue" not in names
assert "fail" not in names and "requeue" not in names
def test_route_cancel_kills_but_keeps_the_route_status(monkeypatch, rec):
@@ -110,7 +110,7 @@ def test_route_cancel_kills_but_keeps_the_route_status(monkeypatch, rec):
worker._run_supervised(1, 99)
names = _names(rec)
assert "stop_child" in names and "reset_asset" in names
assert "set_job" not in names and "requeue" not in names
assert "fail" not in names and "requeue" not in names
def test_wall_clock_cap_kills_even_while_progressing(monkeypatch, rec):
@@ -118,9 +118,8 @@ def test_wall_clock_cap_kills_even_while_progressing(monkeypatch, rec):
_spawn(monkeypatch, _FakeProc(["timeout"]))
_liveness(monkeypatch, ("running", 1.0)) # fresh progress, but over the absolute cap
worker._run_supervised(1, 99)
set_job = next(c for c in rec if c[0] == "set_job")
assert set_job[2]["status"] == "error"
assert "time limit" in set_job[2]["error"].lower()
fail = next(c for c in rec if c[0] == "fail")
assert "time limit" in fail[2]["error"].lower()
def test_child_crash_without_finishing_is_failed(monkeypatch, rec):
@@ -130,8 +129,8 @@ def test_child_crash_without_finishing_is_failed(monkeypatch, rec):
worker._run_supervised(1, 99)
names = _names(rec)
assert "reset_asset" in names
set_job = next(c for c in rec if c[0] == "set_job")
assert set_job[2]["status"] == "error" and "unexpectedly" in set_job[2]["error"].lower()
fail = next(c for c in rec if c[0] == "fail")
assert "unexpectedly" in fail[2]["error"].lower()
def test_child_nonzero_exit_that_already_set_state_is_left_alone(monkeypatch, rec):
@@ -139,7 +138,7 @@ def test_child_nonzero_exit_that_already_set_state_is_left_alone(monkeypatch, re
_spawn(monkeypatch, _FakeProc([("exit", 1)]))
_liveness(monkeypatch, ("error", None))
worker._run_supervised(1, 99)
assert "set_job" not in _names(rec)
assert "fail" not in _names(rec)
def test_spawn_failure_frees_the_asset_and_requeues(monkeypatch, rec):
@@ -158,7 +157,7 @@ def test_shutdown_kills_and_requeues_for_next_boot(monkeypatch, rec):
_liveness(monkeypatch, ("running", 1.0))
worker._run_supervised(1, 99)
names = _names(rec)
assert "stop_child" in names and "requeue" in names and "set_job" not in names
assert "stop_child" in names and "requeue" in names and "fail" not in names
def test_a_long_download_survives_while_progress_stays_fresh(monkeypatch, rec):
@@ -191,7 +190,7 @@ def test_a_wedged_finished_child_is_killed_by_the_wall_clock(monkeypatch, rec):
_liveness(monkeypatch, ("done", 1.0)) # finished, but the process lingers
worker._run_supervised(1, 99)
names = _names(rec)
assert "stop_child" in names and "set_job" not in names # killed to free the slot, status kept
assert "stop_child" in names and "fail" not in names # killed to free the slot, status kept
def test_a_watchdog_error_still_stops_the_child(monkeypatch, rec):
@@ -206,7 +205,7 @@ def test_a_watchdog_error_still_stops_the_child(monkeypatch, rec):
worker._run_supervised(1, 99)
names = _names(rec)
assert "stop_child" in names and "reset_asset" in names
assert next(c for c in rec if c[0] == "set_job")[2]["status"] == "error"
assert "fail" in _names(rec)
# --- _stop_child escalation -----------------------------------------------------------------
@@ -237,3 +236,37 @@ def test_stop_child_noop_if_already_exited(killpg_rec):
proc.returncode = 0 # already gone
worker._stop_child(proc, grace_s=0.01)
assert killpg_rec == []
def test_a_crashed_jobs_crash_line_and_cleanup_carry_the_job_id(monkeypatch):
"""The job id is bound for the WHOLE job, the crash handler included: bound inside the `try`, it
was already reset when the one line that says the job crashed was logged."""
from app import logs
seen: dict[str, object] = {}
claims = iter([42])
def claim():
job = next(claims, None)
if job is None:
worker._stop.set()
return job
def crash(_job_id):
raise RuntimeError("boom")
monkeypatch.setattr(worker, "_claim_job", claim)
monkeypatch.setattr(worker, "_process_job", crash)
monkeypatch.setattr(
worker.log, "exception", lambda *a, **k: seen.setdefault("crash", logs._job_id.get())
)
monkeypatch.setattr(
worker, "_fail_running", lambda *a, **k: seen.setdefault("cleanup", logs._job_id.get())
)
monkeypatch.setattr(worker.settings, "download_poll_seconds", 0)
worker._stop.clear()
try:
worker._worker_loop(0)
finally:
worker._stop.clear()
assert seen == {"crash": 42, "cleanup": 42}
+13
View File
@@ -11,6 +11,15 @@
# applies local migrations to THIS local DB only. Seed the catalog once with a pg_dump from
# another instance if you want real data (see RUNBOOK).
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
# busy instance and a hard ceiling on the disk it can take.
x-logging: &logging
driver: json-file
options:
max-size: "50m"
max-file: "3"
services:
db:
image: postgres:16-alpine
@@ -27,6 +36,7 @@ services:
interval: 5s
timeout: 5s
retries: 12
logging: *logging
restart: unless-stopped
api:
@@ -63,6 +73,7 @@ services:
- ./downloads:/downloads
# Plex media, read-only (see the plex_media volume note below). Maps to plex_path_map.
- plex_media:/plex-media:ro
logging: *logging
restart: unless-stopped
# Dedicated download worker: same image, runs the yt-dlp job loop instead of the API.
@@ -96,6 +107,7 @@ services:
volumes:
- ./downloads:/downloads
- plex_media:/plex-media:ro
logging: *logging
restart: unless-stopped
# PO-token provider: mints YouTube Proof-of-Origin tokens on demand so the worker bypasses
@@ -106,6 +118,7 @@ services:
init: true
security_opt:
- apparmor:unconfined
logging: *logging
restart: unless-stopped
volumes:
+13
View File
@@ -10,6 +10,15 @@
# Everything else — the admin account, Google sign-in, SMTP — is set in the web wizard on
# first run. See docs/self-hosting.md.
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
# busy instance and a hard ceiling on the disk it can take.
x-logging: &logging
driver: json-file
options:
max-size: "50m"
max-file: "3"
services:
db:
image: postgres:16-alpine
@@ -27,6 +36,7 @@ services:
interval: 10s
timeout: 5s
retries: 12
logging: *logging
restart: unless-stopped
api:
@@ -71,6 +81,7 @@ services:
timeout: 5s
retries: 5
start_period: 30s
logging: *logging
restart: unless-stopped
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
@@ -109,6 +120,7 @@ services:
- ALL
tmpfs:
- /tmp
logging: *logging
restart: unless-stopped
# PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
@@ -119,6 +131,7 @@ services:
networks: [internal]
security_opt:
- no-new-privileges:true
logging: *logging
restart: unless-stopped
volumes:
+13
View File
@@ -1,3 +1,12 @@
# Bounded container logs. Docker's json-file driver keeps every line forever by default, and the API
# now writes one structured line per request (E22 S2) — 3×50 MB per service is days of history on a
# busy instance and a hard ceiling on the disk it can take.
x-logging: &logging
driver: json-file
options:
max-size: "50m"
max-file: "3"
services:
db:
image: postgres:16-alpine
@@ -12,6 +21,7 @@ services:
interval: 5s
timeout: 5s
retries: 12
logging: *logging
restart: unless-stopped
api:
@@ -49,6 +59,7 @@ services:
timeout: 5s
retries: 5
start_period: 25s
logging: *logging
restart: unless-stopped
# Download worker: same image, runs the yt-dlp / ffmpeg job loop instead of the API. Shares the
@@ -83,6 +94,7 @@ services:
- ${DOWNLOAD_HOST_PATH:-downloads}:/downloads
# Same read-only Plex library as the API — the worker resolves library paths too.
- ${PLEX_MEDIA_HOST_PATH:-plex_media}:/plex-media:ro
logging: *logging
restart: unless-stopped
# PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
@@ -90,6 +102,7 @@ services:
bgutil-pot:
image: brainicism/bgutil-ytdlp-pot-provider:1.3.1
init: true
logging: *logging
restart: unless-stopped
volumes:
+10
View File
@@ -17,6 +17,16 @@ siftlode e2e-seed # (re)seed the test account only, no tests
The localdev stack must be up (`siftlode dev`). `global.setup.ts` reseeds the account and logs in
once, sharing the session via `e2e/.auth/state.json` (git-ignored).
**Browsers** — every flow runs in three projects: `chromium` (Desktop Chrome), `webkit` (Desktop
Safari) and `ipad` (iPad gen 7 landscape: 1080×810, touch, mobile). Install the engines once per
machine with `npx playwright install chromium webkit`; run one project with
`npx playwright test --project=ipad`. WebKit catches what every iOS browser inherits from the engine
(JS, CSS/layout, event order). It does **not** reproduce iPad media refusals: Playwright's WebKit
decodes through the host OS (Media Foundation on Windows, GStreamer on Linux), and measured on
2026-09-14 it played an mp3-in-mp4 and an AV1 mp4 that a real iPad refuses. Check a media format
on a real iPad against the dev stack over the LAN (`http://<desktop LAN IP>:8080`, e.g. a
login-free `/watch/<token>` share link).
**Target** — `PW_BASE_URL` (default `http://127.0.0.1:8080`):
- **`:8080`** (baked SPA) is authoritative and same-origin. After a **frontend change you must
+6 -2
View File
@@ -67,13 +67,17 @@ setup("seed + authenticate", async ({ page }) => {
const me = await (await page.request.get("/api/me")).json();
expect(me.authenticated, "session not established after login").toBeTruthy();
// 4. Pin the UI locale and suppress the first-login onboarding wizard (a full-screen overlay
// 4. Pin the UI locale, suppress the first-login onboarding wizard (a full-screen overlay
// that would intercept every click — the test account has no YouTube connection, so it
// auto-opens). The dismissed flag is per-account: `siftlode.onboarding.dismissed.<id>`.
// auto-opens), and opt the account out of starting on Home. Since E24 a bare URL lands on
// Home, and the feed, channel and player flows this net locks all open "/" expecting the
// feed; the opt-out is the one Settings → Appearance → Startup writes. Both flags are
// per-account: `siftlode.onboarding.dismissed.<id>`, `siftlode.startOnHome.<id>`.
await page.goto("/");
await page.evaluate((id: number) => {
localStorage.setItem("siftlode.lang", "en");
localStorage.setItem(`siftlode.onboarding.dismissed.${id}`, "1");
localStorage.setItem(`siftlode.startOnHome.${id}`, "0");
}, me.id);
mkdirSync(dirname(STORAGE), { recursive: true });
await page.context().storageState({ path: STORAGE });
+69
View File
@@ -14,6 +14,36 @@ import globals from "globals";
//
// reportUnusedDisableDirectives: a `// eslint-disable` that no longer suppresses anything is itself
// an error — that is what keeps inert disable comments from re-accumulating (the S1b mandate).
// E22 S4: every way out of the browser goes through the one module that LOGS it — a request line
// with a request id (lib/http.ts), the upload XHR (lib/api/core.ts), the messages socket
// (lib/messagesSocket.ts). A bare `fetch` elsewhere is a request the client log never sees, which is
// exactly how the baseline survey found four. Each owner is exempt from its OWN global only.
const NETWORK_OWNERS = {
fetch: "src/lib/http.ts",
XMLHttpRequest: "src/lib/api/core.ts",
WebSocket: "src/lib/messagesSocket.ts",
EventSource: null,
};
const networkGlobals = (exempt) =>
Object.entries(NETWORK_OWNERS)
.filter(([name]) => name !== exempt)
.map(([name, owner]) => ({
name,
message: owner
? `Use the logged request layer: \`${name}\` belongs to ${owner} only (E22 S4 — a request outside it never reaches the client log).`
: `No \`${name}\` without a logged owner module (E22 S4).`,
}));
const networkProperties = [
{ object: "window", property: "fetch", message: "Use lib/http.ts `send` (E22 S4)." },
{ object: "globalThis", property: "fetch", message: "Use lib/http.ts `send` (E22 S4)." },
{
object: "navigator",
property: "sendBeacon",
message: "Use lib/api/core.ts `beacon` (E22 S4).",
},
];
export default tseslint.config(
{ ignores: ["dist", "node_modules", "playwright-report", "test-results", "e2e/.auth"] },
{
@@ -42,6 +72,11 @@ export default tseslint.config(
// types; i18n `t` props → TFunction), so the rule is now enforced to keep new ones from
// creeping back in. Prefer `unknown` + narrowing, or a typed boundary, over `any`.
"@typescript-eslint/no-explicit-any": "error",
// E22 S4: the client log (lib/log.ts) is the one place diagnostics go — redacted, levelled,
// bounded. A `console.*` call is none of those, and it is invisible to the user's own log.
"no-console": "error",
"no-restricted-globals": ["error", ...networkGlobals(null)],
"no-restricted-properties": ["error", ...networkProperties],
// Side-effect ternaries/short-circuits are an established idiom here (`cond ? a() : b()`,
// `v.paused ? v.play() : v.pause()`) — allow them rather than rewrite working code to if/else.
"@typescript-eslint/no-unused-expressions": [
@@ -109,5 +144,39 @@ export default tseslint.config(
],
},
},
{
// E11 S3: the same one-way rule one layer down. `lib/` is what components are built ON (hooks,
// the API slices, pure helpers), so it may not reach back UP into `components/` — not even for a
// type. That is how the `Column` contract came to be owned by a ui component that three lib
// helpers had to import from. A piece both layers need moves down into `lib/`.
files: ["src/lib/**/*.{ts,tsx}"],
rules: {
"no-restricted-imports": [
"error",
{
patterns: [
{
// Any depth (`lib/api/*` is one level deeper), and `(/|$)` so a bare barrel counts too.
regex: String.raw`^(\.\./)+components(/|$)`,
message:
"lib/ sits below components/: it may not import from components/, not even a type. Move the shared piece down into lib/.",
},
],
},
],
},
},
// The network owners (see NETWORK_OWNERS), each exempt from its own global only.
...Object.entries(NETWORK_OWNERS)
.filter(([, owner]) => owner)
.map(([name, owner]) => ({
files: [owner],
rules: { "no-restricted-globals": ["error", ...networkGlobals(name)] },
})),
{
// Tests stand doubles in for the network (`vi.stubGlobal("fetch", …)`) and may reach it directly.
files: ["src/**/*.test.{ts,tsx}"],
rules: { "no-restricted-globals": "off", "no-restricted-properties": "off" },
},
prettier,
);
+1 -1
View File
@@ -12,7 +12,7 @@
"build": "npm run typecheck:app && npm run typecheck:node && vite build",
"preview": "vite preview",
"knip": "knip",
"lint": "eslint .",
"lint": "eslint . --max-warnings 0",
"format": "prettier --write .",
"format:check": "prettier --check .",
"test": "vitest run",
+14
View File
@@ -47,5 +47,19 @@ export default defineConfig({
use: { ...devices["Desktop Chrome"], storageState: STORAGE_STATE },
dependencies: ["setup"],
},
// WebKit catches engine-level differences (JS, CSS/layout, event order) that every iOS browser
// inherits. It does NOT reproduce iPad MEDIA refusals: Playwright's WebKit decodes through the
// host OS (Media Foundation on Windows, GStreamer on Linux), and measured on 2026-09-14 it played
// an mp3-in-mp4 and an AV1 mp4 that iPad refuses. Media checks need a real iPad.
{
name: "webkit",
use: { ...devices["Desktop Safari"], storageState: STORAGE_STATE },
dependencies: ["setup"],
},
{
name: "ipad",
use: { ...devices["iPad (gen 7) landscape"], storageState: STORAGE_STATE },
dependencies: ["setup"],
},
],
});
-33
View File
@@ -1,33 +0,0 @@
// The global Shift+W wiring, pinned at its call site.
//
// What this is: a source contract, the same idiom as uiBoundary.contract.test.ts — a structural rule
// asserted directly rather than left to prose. What it is NOT: a behaviour test. The behaviour lives
// in history.dom.test.ts and history.burial.dom.test.ts, which drive `navigateOverOverlays` against
// real overlay entries; the end-to-end gesture (Shift+W over an open player, then Back) is a browser
// check, because jsdom reproduces history ordering only as bookkeeping.
//
// It exists because the composition line itself was unpinned: App.tsx is the one module no test
// renders (main.tsx lazy-imports it and nothing else does), globalKeymap.dom.test.tsx drives the
// hook with its OWN handler, and history.dom.test.ts drives the helper with its own navigate — so
// dropping `navigateOverOverlays` from App left all 818 tests green while the user-visible bug came
// straight back (round 2, verified by running that revert).
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { describe, expect, it } from "vitest";
const APP = readFileSync(fileURLToPath(new URL("./App.tsx", import.meta.url)), "utf8");
describe("App · global keymap wiring", () => {
it("routes the global home shortcut through navigateOverOverlays", () => {
const call = /useGlobalKeymap\(([\s\S]*?)\);/.exec(APP)?.[1];
// A rename or a move is a legitimate change — but then this test is the thing that has to be
// updated deliberately, which is the whole point of pinning it.
expect(
call,
"App.tsx no longer calls useGlobalKeymap in a shape this test can read",
).toBeTypeOf("string");
expect(call).toContain("navigateOverOverlays");
expect(call).toContain('setPage("home")');
});
});
+84 -99
View File
@@ -2,15 +2,15 @@ import { lazy, Suspense, useEffect, useState } from "react";
import { qk } from "./lib/queryKeys";
import { useTranslation } from "react-i18next";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { api, getActiveAccount, setActiveAccount, setUnauthorizedHandler } from "./lib/api";
import { setLanguage, isSupported, type LangCode } from "./i18n";
import { api, setUnauthorizedHandler, type Me } from "./lib/api";
import { notifyActionError } from "./lib/actionErrors";
import { setLanguage, type LangCode } from "./i18n";
import { clearParams } from "./lib/urlState";
import type { Page } from "./lib/urlState";
import {
loadLayout,
normalizeLayout,
PREF_KEY,
saveLayoutLocal,
seedLayout,
type PanelId,
type PanelLayout,
} from "./lib/panelLayout";
@@ -18,12 +18,14 @@ import { notify, removeByMetaKind } from "./lib/notifications";
import { LS, setAccountRaw } from "./lib/storage";
import { usePref } from "./lib/usePref";
import { useVerifyEntry } from "./lib/useVerifyEntry";
import { useNavigation, useNavigationActions } from "./components/NavigationProvider";
import { useFeedFilters, useFeedFiltersActions } from "./components/FeedFiltersProvider";
import { useChannelLayout } from "./components/ChannelsProvider";
import { useFeedView, useFeedViewActions } from "./components/FeedViewProvider";
import { moduleDef } from "./lib/modules";
import { PAGE_CONTENT, PAGE_RAIL } from "./components/moduleRegistry";
import { useNavigation, useNavigationActions } from "./components/navigationContext";
import { useFeedFilters, useFeedFiltersActions } from "./components/feedFiltersContext";
import { useChannelLayout } from "./components/channelsContext";
import { useFeedView, useFeedViewActions } from "./components/feedViewContext";
import { moduleDef, moduleOrder } from "./lib/modules";
import { PAGE_CONTENT, PAGE_RAIL, preloadPages } from "./components/moduleRegistry";
import { lazyWithPreload } from "./lib/lazyWithPreload";
import { LoadingState } from "./components/ui/QueryState";
// Persistent shell — always mounted, so eagerly imported.
import Welcome from "./components/Welcome";
import SetupWizard from "./components/SetupWizard";
@@ -37,8 +39,6 @@ import Toaster from "./components/ui/Toaster";
import KeyboardHelp from "./components/ui/KeyboardHelp";
import AppCommandPalette from "./components/AppCommandPalette";
import { useGlobalKeymap } from "./lib/keymap";
import { navigateOverOverlays } from "./lib/history";
import { StateMessage } from "./components/ui/QueryState";
import ErrorDialog from "./components/ErrorDialog";
import VersionBanner from "./components/VersionBanner";
import DemoBanner from "./components/DemoBanner";
@@ -48,11 +48,12 @@ import { focusAccessRequestsTab } from "./lib/adminUsersTab";
// pulls its whole ~900-line changelog into the eager App chunk and defeats the lazy ReleaseNotes
// split. FRONTEND_VERSION equals RELEASE_NOTES[0].version by release convention (both bump together).
import { FRONTEND_VERSION } from "./lib/version";
import { applyDiagnostics } from "./lib/diagnostics";
// The page modules (and their side rails) are code-split per route via the module registry
// (components/moduleRegistry). App keeps only the persistent-shell chunks + the channel page and the
// two modals, which sit outside the registry-driven content column.
const ChannelPage = lazy(() => import("./components/ChannelPage"));
const ChannelPage = lazyWithPreload(() => import("./components/ChannelPage"));
const About = lazy(() => import("./components/About"));
const ReleaseNotes = lazy(() => import("./components/ReleaseNotes"));
@@ -79,13 +80,20 @@ export default function App() {
// by the feed page and the channel page. App reads it only to hand to <Feed>/<ChannelPage>.
const view = useFeedView();
const { setView } = useFeedViewActions();
const queryClient = useQueryClient();
// Per-panel "customize" layouts (order/collapsed/hidden of each side panel's group cards),
// persisted per-account. One record so feed/plex/playlists share the same mechanism.
const [panelLayouts, setPanelLayouts] = useState<Record<PanelId, PanelLayout>>(() => ({
feed: loadLayout("feed"),
plex: loadLayout("plex"),
playlists: loadLayout("playlists"),
}));
// persisted per-account. One record so feed/plex/playlists share the same mechanism. Seeded from
// the account's server preferences — BootGate mounts App only once `me` is cached — so a layout
// changed on another device is right on the first paint; the local copy covers a key the server
// has never stored.
const [panelLayouts, setPanelLayouts] = useState<Record<PanelId, PanelLayout>>(() => {
const prefs = queryClient.getQueryData<Me | null>(qk.me())?.preferences;
return {
feed: seedLayout("feed", prefs),
plex: seedLayout("plex", prefs),
playlists: seedLayout("playlists", prefs),
};
});
const [aboutOpen, setAboutOpen] = useState(false);
const [notesOpen, setNotesOpen] = useState(false);
const [notesHighlight, setNotesHighlight] = useState<string | undefined>(undefined);
@@ -98,12 +106,14 @@ export default function App() {
function setPanelLayout(panel: PanelId, next: PanelLayout) {
setPanelLayouts((prev) => ({ ...prev, [panel]: next }));
saveLayoutLocal(panel, next);
api.savePrefs({ [PREF_KEY[panel]]: next }).catch(() => {});
api
.savePrefs({ [PREF_KEY[panel]]: next })
.catch((e: unknown) => notifyActionError(e, t("errors.writeFailed.prefs")));
}
// Collapse state for the two full-height panels (left nav + filter sidebar). Persisted to the
// user's preferences (via usePref) so it follows the account across devices; the localStorage mirror
// seeds the initial render synchronously so nothing flashes open before the server prefs arrive.
// user's preferences (via usePref) so it follows the account across devices, seeded from the `me`
// cache at mount; the localStorage mirror covers an account whose server prefs never stored it.
// Default unpinned (tab + hover-expand overlay); accounts that explicitly pin keep their choice.
const [filterCollapsed, setFilterCollapsed] = usePref<boolean>("filterCollapsed", true, {
localKey: LS.filterCollapsed,
@@ -117,47 +127,58 @@ export default function App() {
// (The feed filters + page/channel now LIVE in the address bar — FeedFiltersProvider and
// NavigationProvider read them on load and keep them mirrored — so there's no query to strip here.)
const queryClient = useQueryClient();
// First-run install gate: a fresh instance reports configured=false and runs in setup mode.
// We check this before anything else and render the wizard; `me` is held back until we know the
// instance is set up (otherwise it would 503 against the setup-mode lock).
// Passive observers (enabled:false) of the setup-status and `me` queries BootGate fetches, polls
// and waits for: App mounts only once both have answered, so their caches are always populated here.
const setupQuery = useQuery({
queryKey: qk.setupStatus(),
queryFn: api.setupStatus,
staleTime: Infinity,
enabled: false,
});
const needsSetup = setupQuery.data?.configured === false;
// Poll the session so an account suspended/deleted by an admin is noticed even with no
// interaction (option 1): the refetch 401s → the 401 handler below drops to the login page.
// Only poll while signed in (data present) — polling on the public login page is pointless and
// its periodic refetch caused a visible flicker there.
const meQuery = useQuery({
queryKey: qk.me(),
queryFn: api.me,
enabled: setupQuery.isSuccess && !needsSetup,
refetchInterval: (query) => (query.state.data ? 60_000 : false),
});
const meQuery = useQuery({ queryKey: qk.me(), queryFn: api.me, enabled: false });
// The `global` key scope: `h` opens the cheat sheet, Mod+K the jump-to palette, Shift+W jumps to
// Home ("mit nézzek?"). Bound only once signed in, because that is where the overlays are mounted
// (bottom of this tree) and where the modules exist at all — see useGlobalKeymap on why binding it
// above the signed-out branch is worse than not binding it.
// Through `navigateOverOverlays`, because a global key can fire while an overlay is open and does
// not know it: the page change unmounts the player (it lives inside the module), whose reconcile
// then reclaimed the entry this push had just made — Shift+W over a playing video closed the
// player and snapped straight back to the feed. `setPage`'s boolean is load-bearing here, not
// decoration: pressed while ALREADY on Home (the default landing page) it navigates nowhere, and
// burying the open player's entry under a push that never happened orphaned it — see `buryUnder`.
useGlobalKeymap(!!meQuery.data, () => navigateOverOverlays(() => setPage("home")));
// A global key can fire while an overlay is open; `setPage` closes every overlay before it pushes,
// so Shift+W over the player lands on Home with the player closed — also when Home is already the
// page underneath (owner decision 2026-09-09: the key always ends with the answer on screen).
// The `page` scope rides the same binding: the `g` chords jump to a module, but only to one this
// account can reach (the nav rail's own list, plus Settings, which is not in the rail's order).
useGlobalKeymap(
!!meQuery.data,
setPage,
meQuery.data ? [...moduleOrder(meQuery.data), "settings"] : [],
);
// Pin this tab to whatever account it first loaded as. Without this, a tab that never picked
// an account (it was riding the session's default) would silently swap identity when ANOTHER
// tab changes the default — e.g. that other tab adds/switches an account. Once pinned, this
// tab always sends its own account header, so cross-tab changes can't reach it. An explicit
// switch on THIS tab sets the override itself (so this no-ops then).
// The account's consented diagnostics level (E22 S5), applied to this tab's log: raised, the log is
// kept in more detail and streamed to the server until the level lapses. A lapse (its own timer, or
// the server saying it is off) refetches `me`, so Settings shows the base level again.
const diagnostics = meQuery.data?.diagnostics;
useEffect(() => {
if (meQuery.data && getActiveAccount() == null) setActiveAccount(meQuery.data.id);
}, [meQuery.data?.id]);
applyDiagnostics(diagnostics, () => void queryClient.invalidateQueries({ queryKey: qk.me() }));
}, [diagnostics?.level, diagnostics?.until]); // eslint-disable-line react-hooks/exhaustive-deps
// Fetch the code of every module this account can open — and the channel page — once the first page
// is on screen, so a module's first visit renders at once instead of flashing the chunk fallback.
// Idle-scheduled so it never competes with the first page's own requests (Safari has no
// requestIdleCallback, hence the timer).
useEffect(() => {
const me = meQuery.data;
if (!me) return;
const run = () => {
preloadPages([...moduleOrder(me), "settings"]);
ChannelPage.preload().catch(() => {});
};
const idle = window.requestIdleCallback as typeof window.requestIdleCallback | undefined;
if (idle) {
const id = idle(run, { timeout: 3000 });
return () => window.cancelIdleCallback(id);
}
const id = window.setTimeout(run, 1000);
return () => window.clearTimeout(id);
}, [meQuery.data?.id]); // eslint-disable-line react-hooks/exhaustive-deps
// Measure the scrollbar's gutter width once and publish it as --sbw, so the floating header and the
// toolbar band can pad their right edge to line up with the scrolled cards (the app scroller reserves
@@ -233,28 +254,6 @@ export default function App() {
clearParams("admin");
}, [adminDeepLink, meQuery.data?.id]); // eslint-disable-line react-hooks/exhaustive-deps
// On login, adopt the server-stored SHELL preferences (panel layouts + language). The rail-collapse
// flags (filterCollapsed / playlistsCollapsed) are adopted by their own usePref hooks, the
// Settings-page prefs (theme/perf/hints/notifications) by PrefsProvider, and the feed view by
// FeedViewProvider — each via its own passive qk.me() observer.
useEffect(() => {
const prefs = meQuery.data?.preferences;
if (!prefs) return;
(["feed", "plex", "playlists"] as PanelId[]).forEach((p) => {
const raw = prefs[PREF_KEY[p]];
if (raw) {
const l = normalizeLayout(p, raw);
setPanelLayouts((prev) => ({ ...prev, [p]: l }));
saveLayoutLocal(p, l);
}
});
const lang = prefs.language;
if (typeof lang === "string" && isSupported(lang)) setLanguage(lang);
// (Per-account filters — incl. the demo account's whole-library default — are loaded by the
// dedicated effect above, which also covers accounts that have no stored preferences.)
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [meQuery.data?.id]);
// Nudge admins when access requests are waiting (in lieu of a server-push bell). Keyed on the
// pending COUNT so it re-resolves the moment an approval/denial changes it — not only on a
// fresh load (which is why the notice used to linger until F5 after an approval).
@@ -279,8 +278,8 @@ export default function App() {
label: t("common.accessRequestsReview"),
onClick: () => {
focusAccessRequestsTab();
// Over whatever overlay the toast was clicked in — as navigateTo does (see lib/nav).
navigateOverOverlays(() => setPage("users"));
// Clicked inside whatever overlay hosts the toast; setPage closes it before pushing.
setPage("users");
},
},
});
@@ -291,36 +290,22 @@ export default function App() {
// Language is edited outside the Settings page (sidebar/login), so it stays instant.
function changeLanguage(code: LangCode) {
setLanguage(code);
api.savePrefs({ language: code }).catch(() => {});
// Also from the signed-out login page, where the save 401s — `req` marks that as handled, so
// the reporter stays quiet there.
api
.savePrefs({ language: code })
.catch((e: unknown) => notifyActionError(e, t("errors.writeFailed.prefs")));
}
// First-run: a fresh instance is in setup mode (all other routes are locked) → show the wizard.
if (setupQuery.isLoading)
return (
<div className="min-h-screen grid place-items-center text-muted">{t("common.loading")}</div>
);
if (needsSetup) return <SetupWizard />;
if (meQuery.isLoading)
return (
<div className="min-h-screen grid place-items-center text-muted">{t("common.loading")}</div>
);
// /api/me answers 200 always now (null body = logged out), so a thrown error here is a real
// network/5xx failure, and no data means "not signed in" → the public landing.
if (meQuery.error)
return (
<div className="min-h-screen grid place-items-center">
<StateMessage tone="error" onRetry={() => meQuery.refetch()}>
{t("common.somethingWrong")}
</StateMessage>
</div>
);
// /api/me answers 200 always (null body = logged out), and BootGate has already handled a failed
// probe — so no data here means "not signed in" → the public landing.
if (!meQuery.data) return <Welcome verify={verifyEntry} />;
// Shown briefly while a lazy page/module chunk loads (first visit to that page only).
const pageFallback = (
<div className="flex-1 grid place-items-center text-muted">{t("common.loading")}</div>
);
// Shown while a page's chunk is still on its way: only if its visit beats the preload above (or the
// preload failed), and like every loading state it stays invisible for a load that is quick anyway.
const pageFallback = <LoadingState className="flex-1 justify-center" />;
// Scroll restore/reset key for the shared normal-page scroller (see PageScroller): the same
// page/feed-filters returns you to where you were (bug 1); a changed feed filter is a new key, so
+2 -1
View File
@@ -4,13 +4,14 @@ import { useTranslation } from "react-i18next";
import { Info, Sparkles } from "lucide-react";
import { api } from "../lib/api";
import { FRONTEND_VERSION, FRONTEND_SHA, FRONTEND_BUILD_DATE } from "../lib/version";
import { formatDateTime } from "../lib/format";
import { Button } from "./ui/Button";
import Modal from "./ui/Modal";
function fmtDate(d?: string | null): string {
if (!d) return "—";
const t = new Date(d);
return isNaN(t.getTime()) ? d : t.toLocaleString();
return isNaN(t.getTime()) ? d : formatDateTime(d);
}
// About dialog: app + build + schema versions (frontend/backend/database), plus a
+3 -2
View File
@@ -3,7 +3,8 @@ import { useQuery } from "@tanstack/react-query";
import { qk } from "../lib/queryKeys";
import { api } from "../lib/api";
import ChannelAboutContent from "./ChannelAbout";
import { PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
import { PopoverSurface } from "./ui/AnchoredPopover";
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
/**
* A channel-description cell: one truncated line in the table, revealing the FULL channel "About"
@@ -49,7 +50,7 @@ export default function AboutCell({ text, channelId }: { text: string | null; ch
ref={pop.refs.setReference}
{...pop.getReferenceProps()}
tabIndex={0}
className="block max-w-88 truncate text-muted cursor-help rounded-sm outline-hidden focus-visible:outline-2 focus-visible:outline-accent/60 focus-visible:outline-offset-2"
className="block max-w-88 truncate text-muted cursor-help rounded-sm outline-hidden focus-visible:outline-2 focus-visible:outline-accent focus-visible:outline-offset-2"
>
{preview}
</span>
+24 -12
View File
@@ -1,14 +1,16 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Check, Clock, Copy, Globe, KeyRound, Mail, Send, ShieldAlert } from "lucide-react";
import { Check, Clock, Copy, Globe, KeyRound, LogIn, Mail, Send, ShieldAlert } from "lucide-react";
import Modal from "./ui/Modal";
import { LoadingState, StateMessage } from "./ui/QueryState";
import { api } from "../lib/api";
import { qk } from "../lib/queryKeys";
import { notify } from "../lib/notifications";
import { notifyActionError } from "../lib/actionErrors";
import { copyText } from "../lib/clipboard";
import { invalidateUserAccess, isEmailOff } from "../lib/adminUsers";
import { formatDateTime } from "../lib/format";
import { invalidateUserAccess } from "../lib/adminUsers";
import type { AdminUserAccess, AdminUserRow } from "../lib/api/admin";
// The admin's answer to "why can this person not get in?", and the place every recovery action for
@@ -66,25 +68,19 @@ export default function AccountPanel({
const issue = useMutation({
mutationFn: () => api.createSetPasswordLink(user.id),
onSuccess: (r) => setLink({ url: r.url, expires_at: r.expires_at }),
onError: () => notify({ level: "error", message: t("users.account.linkFailed") }),
onError: (err) => notifyActionError(err, t("users.account.linkFailed")),
});
const resend = useMutation({
mutationFn: () => api.resendUserVerification(user.id),
onSuccess: () => notify({ level: "success", message: t("users.account.resendSent") }),
onError: (err) =>
notify({
level: "error",
message: isEmailOff(err)
? t("users.pendingAccess.resendEmailOff")
: t("users.pendingAccess.resendFailed"),
}),
onError: (err) => notifyActionError(err, t("users.pendingAccess.resendFailed")),
});
const activate = useMutation({
mutationFn: () => api.activateUser(user.id),
onSuccess: () => invalidateUserAccess(qc, user.id),
onError: () => notify({ level: "error", message: t("users.pendingAccess.activateFailed") }),
onError: (err) => notifyActionError(err, t("users.pendingAccess.activateFailed")),
});
const copy = async () => {
@@ -173,6 +169,20 @@ export default function AccountPanel({
}
tone={a.has_google ? "ok" : "muted"}
/>
{/* The exact moment; the list row beside it only says how long ago. Null has two different
meanings, so it says which: nothing recorded yet, or the shared demo account, which is
deliberately never stamped. */}
<AccessRow
icon={LogIn}
label={t("users.account.lastLogin")}
value={
a.last_login_at
? formatDateTime(a.last_login_at)
: a.is_demo
? t("users.account.lastLoginDemo")
: t("users.account.lastLoginNone")
}
/>
{/* The gate that explains a Google sign-in silently becoming a fresh access request. */}
{!a.is_allowed && (
<AccessRow
@@ -201,7 +211,9 @@ export default function AccountPanel({
<button
onClick={() => resend.mutate()}
disabled={!a.email_configured || a.email_verified || resend.isPending}
// The demo account's address is never confirmed, and the server refuses to mail it — as
// it refuses it a set-password link below (deep-review E11 round 3).
disabled={user.is_demo || !a.email_configured || a.email_verified || resend.isPending}
className="w-full glass-card glass-hover flex items-center gap-2 px-3 py-2.5 rounded-lg text-xs disabled:opacity-45 transition"
>
<Send className="w-4 h-4 shrink-0" />
@@ -4,7 +4,8 @@ import { Filter, X } from "lucide-react";
import type { ActiveFilter } from "../lib/useActiveFilters";
import { decideChipLayout } from "../lib/chipOverflow";
import { chipLaneClass, useChipOverflowFit } from "../lib/useChipOverflowFit";
import { PopoverItem, PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
import { PopoverItem, PopoverSurface } from "./ui/AnchoredPopover";
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
// The applied-filter row above the feed: every narrowing currently in force, each removable on its
// own. It exists because the filter panel now sits collapsed as a tab — without this you'd have to
+2 -1
View File
@@ -7,7 +7,8 @@ import { api, type Playlist } from "../lib/api";
import { notifyActionError } from "../lib/actionErrors";
import { useScrollFade } from "../lib/useScrollFade";
import { playlistName } from "../lib/playlistName";
import { PopoverItem, PopoverSurface, useAnchoredPopover } from "./ui/AnchoredPopover";
import { PopoverItem, PopoverSurface } from "./ui/AnchoredPopover";
import { useAnchoredPopover } from "./ui/anchoredPopoverHooks";
// A small anchored menu for toggling a video's membership in the user's playlists, with inline
// "new playlist" creation. Same shape as the channel tag editor (R8 S1's pilot): radio-style
@@ -24,7 +24,7 @@ vi.mock("../lib/useMe", () => ({
pending_activations: 0,
}),
}));
vi.mock("./ConfirmProvider", () => ({ useConfirm: () => async () => true }));
vi.mock("./confirmContext", () => ({ useConfirm: () => async () => true }));
const { notify } = vi.hoisted(() => ({
notify: vi.fn((_n: { level: string; message: string }) => 0),
}));
@@ -36,6 +36,7 @@ vi.mock("../lib/errorDialog", () => ({ reportError: vi.fn() }));
import AdminUsers from "./AdminUsers";
import { api, type AdminUserRow } from "../lib/api";
import { relativeTime } from "../lib/format";
import i18n from "../i18n";
const row = {
@@ -76,6 +77,27 @@ function mount() {
return users;
}
describe("the last sign-in column (E22 S1)", () => {
it("says how long ago each account signed in, and a dash where nothing is recorded", async () => {
const at = new Date(Date.now() - 3 * 86_400_000).toISOString();
vi.spyOn(api, "adminUsers").mockResolvedValue([
{ ...row, id: 3, email: "seen@x", last_login_at: at },
{ ...row, id: 4, email: "never@x", last_login_at: null },
{ ...row, id: 5, email: "demo@x", is_demo: true, last_login_at: null },
] as never);
render(
<QueryClientProvider
client={new QueryClient({ defaultOptions: { queries: { retry: false } } })}
>
<AdminUsers />
</QueryClientProvider>,
);
expect(await screen.findByText(relativeTime(at))).toBeTruthy();
expect(screen.getAllByText("—")).toHaveLength(2);
});
});
describe("a user action that fails", () => {
it("says a user who is gone, and refetches the list", async () => {
const users = mount();
@@ -86,7 +108,8 @@ describe("a user action that fails", () => {
await waitFor(() =>
expect(notify).toHaveBeenCalledWith({
level: "error",
message: i18n.t("errors.notFound.user"), // the request layer localizes a structured detail
title: "users.roles.failed", // what failed…
message: i18n.t("errors.notFound.user"), // …and why: the request layer localizes a structured detail
}),
);
await waitFor(() => expect(users).toHaveBeenCalledTimes(2));
+38 -39
View File
@@ -24,7 +24,8 @@ import {
type PendingAccessRow,
} from "../lib/api";
import { notify } from "../lib/notifications";
import { invalidateUserAccess, isEmailOff } from "../lib/adminUsers";
import { formatDateTime, relativeTime } from "../lib/format";
import { invalidateUserAccess } from "../lib/adminUsers";
import { ADMIN_USERS_TAB_KEY } from "../lib/adminUsersTab";
import { Button } from "./ui/Button";
import Tooltip from "./ui/Tooltip";
@@ -34,11 +35,12 @@ import SegmentedToggle from "./ui/SegmentedToggle";
import { LANGUAGES, type LangCode } from "../i18n";
import { LoadingState, StateMessage } from "./ui/QueryState";
import AccountPanel from "./AccountPanel";
import { useConfirm } from "./ConfirmProvider";
import Tabs, { usePersistedTab } from "./ui/Tabs";
import { useConfirm } from "./confirmContext";
import Tabs from "./ui/Tabs";
import { useAccountPersistedState } from "../lib/storage";
import { PageToolbar } from "./ui/PageShell";
import { useMe } from "../lib/useMe";
import { notifyActionError } from "../lib/actionErrors";
import { notifyActionError, notifyKnownFailure } from "../lib/actionErrors";
// Admin-only user & access management: roles, access requests (the Invite whitelist), and the
// demo whitelist + reset. Each section is its own tab (persisted across reloads); the Access tab
@@ -49,7 +51,7 @@ import { notifyActionError } from "../lib/actionErrors";
export default function AdminUsers() {
const me = useMe();
const { t } = useTranslation();
const [tab, setTab] = usePersistedTab(ADMIN_USERS_TAB_KEY, "roles");
const [tab, setTab] = useAccountPersistedState(ADMIN_USERS_TAB_KEY, "roles");
const tabs = [
{ id: "roles", label: t("users.tabs.roles") },
// Both kinds of "waiting on the admin": a request needing a decision, and a sign-up whose
@@ -310,6 +312,23 @@ function UsersRoles({ me }: { me: Me }) {
icon={<Trash2 className="w-4 h-4" />}
/>
</Tooltip>
{/* How long ago this account last signed in. LAST in the row on purpose: the badges and
the role button before it vary in width row to row, so only the row's own right
edge lines the values up into one column. The exact moment is the hint here and a
line in the account panel; below `sm` the row has no room, and the panel still says it. */}
<Tooltip
hint={
u.last_login_at
? t("users.roles.lastLogin", { time: formatDateTime(u.last_login_at) })
: u.is_demo
? t("users.roles.lastLoginDemo")
: t("users.roles.lastLoginNone")
}
>
<span className="hidden sm:inline-block w-20 shrink-0 text-right text-2xs text-muted tabular-nums">
{u.last_login_at ? relativeTime(u.last_login_at) : "—"}
</span>
</Tooltip>
</div>
);
})}
@@ -319,12 +338,6 @@ function UsersRoles({ me }: { me: Me }) {
);
}
/** Did the invitation fail because this instance has no working SMTP? */
function inviteEmailOff(err: HttpError): boolean {
const d = err.detailData as { code?: string; reason?: string } | null | undefined;
return d?.code === "invite_send" && d.reason === "email_off";
}
// --- Sign-ups that are still outside (roadmap UX-25) --------------------------------------------
//
// The gap this closes: approve an address BEFORE the person registers and their sign-up produces no
@@ -351,7 +364,7 @@ function PendingAccess() {
refresh();
notify({ level: "success", message: t("users.pendingAccess.activated", { email: u.email }) });
},
onError: () => notify({ level: "error", message: t("users.pendingAccess.activateFailed") }),
onError: (err) => notifyActionError(err, t("users.pendingAccess.activateFailed")),
});
const resend = useMutation({
@@ -360,15 +373,9 @@ function PendingAccess() {
refresh();
notify({ level: "success", message: t("users.pendingAccess.resent", { email: u.email }) });
},
// The public resend is deliberately silent; this one is not, so say WHY it failed — an instance
// with no working SMTP would otherwise show a success toast for a mail that was never sent.
onError: (err) =>
notify({
level: "error",
message: isEmailOff(err)
? t("users.pendingAccess.resendEmailOff")
: t("users.pendingAccess.resendFailed"),
}),
// The public resend is deliberately silent; this one is not, and the server's reason says WHY
// (no working SMTP, a refused send) — a success toast for a mail never sent would be the lie.
onError: (err) => notifyActionError(err, t("users.pendingAccess.resendFailed")),
});
const list = rows.data ?? [];
@@ -425,7 +432,7 @@ function PendingAccessRowView({
{row.created_at && (
<span>
{t("users.pendingAccess.signedUp", {
time: new Date(row.created_at).toLocaleString(),
time: formatDateTime(row.created_at),
})}
</span>
)}
@@ -563,12 +570,8 @@ function AdminInvites() {
// A 409 means the list was stale — someone registered on that address since it loaded. Say
// what to do instead of a generic failure, and refetch so the row re-renders as read-only.
const conflict = e instanceof HttpError && e.status === 409;
notify({
level: "error",
message: t(
conflict ? "settings.invites.deleteHasAccount" : "settings.invites.deleteFailed",
),
});
if (conflict) notifyKnownFailure(e, t("settings.invites.deleteHasAccount"));
else notifyActionError(e, t("settings.invites.deleteFailed"));
if (conflict) refresh();
},
});
@@ -598,14 +601,7 @@ function AdminInvites() {
refresh();
notify({ level: "success", message: t("settings.invites.invited", { email: v.inv.email }) });
},
onError: (err) =>
notify({
level: "error",
message:
err instanceof HttpError && inviteEmailOff(err)
? t("settings.invites.inviteEmailOff")
: t("settings.invites.inviteFailed"),
}),
onError: (err) => notifyActionError(err, t("settings.invites.inviteFailed")),
});
// The one recipient the app has no language signal about: the admin typed this address in, so
// nobody ever told us how to write to its owner. The dialog is where that gets decided — and it
@@ -624,7 +620,10 @@ function AdminInvites() {
inv.invited_at
? "settings.invites.inviteConfirmAgain"
: "settings.invites.inviteConfirm",
{ email: inv.email, time: new Date(inv.invited_at ?? 0).toLocaleString() },
{
email: inv.email,
time: formatDateTime(inv.invited_at ?? new Date(0).toISOString()),
},
)}
</p>
<InviteLangPicker initial={lang} onPick={(v) => (lang = v)} />
@@ -723,7 +722,7 @@ function AdminInvites() {
hint={
i.invited_at
? t("settings.invites.invitedAt", {
time: new Date(i.invited_at).toLocaleString(),
time: formatDateTime(i.invited_at),
})
: t("settings.invites.inviteHint")
}
@@ -921,7 +920,7 @@ function InviteRow({
{inv.requested_at && (
<div className="text-2xs text-muted">
{t("settings.invites.requested", {
time: new Date(inv.requested_at).toLocaleString(),
time: formatDateTime(inv.requested_at),
})}
</div>
)}
@@ -10,10 +10,10 @@ import { moduleDef, moduleOrder } from "../lib/modules";
import { SCHEMES, type Scheme } from "../lib/theme";
import { closePalette, toggleKeyboardHelp, usePaletteOpen } from "../lib/keymap";
import { requestSurprise } from "../lib/homeIntent";
import { useNavigationActions } from "./NavigationProvider";
import { usePlaylistsActions } from "./PlaylistsProvider";
import { usePrefs } from "./PrefsProvider";
import { useSavedViews } from "./SavedViewsProvider";
import { useNavigationActions } from "./navigationContext";
import { usePlaylistsActions } from "./playlistsContext";
import { usePrefs } from "./prefsContext";
import { useSavedViews } from "./savedViewsContext";
// What Ctrl/Cmd+K can reach. The generic list/filter/keyboard shell is `ui/CommandPalette`; this is
// the domain half — the layer boundary says a primitive may not import `lib/api` or a module.
+1 -1
View File
@@ -8,7 +8,7 @@ import { notify } from "../lib/notifications";
import { LS } from "../lib/storage";
import { Button } from "./ui/Button";
import { Section } from "./ui/form";
import { useConfirm } from "./ConfirmProvider";
import { useConfirm } from "./confirmContext";
import DataTable from "./ui/DataTable";
import { LoadingState, StateMessage } from "./ui/QueryState";
import { auditColumns } from "./auditColumns";
@@ -0,0 +1,209 @@
// @vitest-environment jsdom
//
// BootGate settles the tab's identity BEFORE the app tree mounts. The bug it replaced: a new tab (no
// pinned account in sessionStorage) built the app for "no account" first — so with Settings →
// Appearance → Startup turned off, NavigationProvider's initializer could not read the opt-out and
// landed on Home, mirrored ?page=home into the address bar, and then re-read its own mirror when the
// account was pinned, staying on Home. Only a same-tab reload (account already pinned) was right.
//
// Real NavigationProvider, real storage and i18n; only the two boot probes are stubbed. The probe
// records EVERY render, so "never showed the wrong page, not even for one render" is what is asserted.
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import { render, screen, cleanup, act, fireEvent, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { useEffect } from "react";
import { useTranslation } from "react-i18next";
import BootGate from "./BootGate";
import { NavigationProvider } from "./NavigationProvider";
import { useNavigation } from "./navigationContext";
import { api, type Me } from "../lib/api";
import { ACTIVE_ACCOUNT_KEY } from "../lib/activeAccount";
import { qk } from "../lib/queryKeys";
import i18n from "../i18n";
const renders: { page: string; lang: string }[] = [];
let mounts = 0;
function Probe() {
const { page } = useNavigation();
const { i18n: inst } = useTranslation();
renders.push({ page, lang: inst.language });
useEffect(() => {
mounts++;
}, []);
return <div data-testid="app">{page}</div>;
}
function mount() {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
render(
<QueryClientProvider client={qc}>
<BootGate>
<NavigationProvider>
<Probe />
</NavigationProvider>
</BootGate>
</QueryClientProvider>,
);
return qc;
}
const account = (over: Partial<Me> = {}) => ({ id: 5, preferences: {}, ...over }) as Me;
const retryButton = () => screen.findByRole("button", { name: i18n.t("common.retry") });
beforeEach(async () => {
localStorage.clear();
sessionStorage.clear(); // a NEW tab: nothing pinned yet
window.history.replaceState(null, "", "/");
renders.length = 0;
mounts = 0;
await i18n.changeLanguage("en");
vi.spyOn(api, "setupStatus").mockResolvedValue({ configured: true });
});
afterEach(() => {
cleanup();
vi.restoreAllMocks();
vi.unstubAllGlobals();
});
describe("BootGate", () => {
it("a new tab with Startup off opens on the page the account left — never on Home first", async () => {
localStorage.setItem("siftlode.startOnHome.5", "0");
localStorage.setItem("siftlode.page.5", "downloads");
vi.spyOn(api, "me").mockResolvedValue(account());
mount();
expect((await screen.findByTestId("app")).textContent).toBe("downloads");
expect(renders.map((r) => r.page)).toEqual(renders.map(() => "downloads"));
expect(new URLSearchParams(window.location.search).get("page")).toBe("downloads");
expect(sessionStorage.getItem(ACTIVE_ACCOUNT_KEY)).toBe("5"); // the tab is pinned
});
it("renders the app in the account's language from its first render", async () => {
vi.spyOn(api, "me").mockResolvedValue(account({ preferences: { language: "hu" } }));
mount();
await screen.findByTestId("app");
expect(renders.map((r) => r.lang)).toEqual(renders.map(() => "hu"));
});
it("mounts nothing of the app while the identity is still unknown", async () => {
vi.spyOn(api, "me").mockReturnValue(new Promise<Me | null>(() => {}));
mount();
await act(async () => {}); // the setup probe resolves; `me` never does
expect(screen.queryByTestId("app")).toBeNull();
expect(renders).toHaveLength(0);
});
it("signed out: mounts the app (App shows the landing) without pinning an account", async () => {
vi.spyOn(api, "me").mockResolvedValue(null);
mount();
await screen.findByTestId("app");
expect(sessionStorage.getItem(ACTIVE_ACCOUNT_KEY)).toBeNull();
});
it("setup mode: mounts the app (App shows the wizard) without asking for the account", async () => {
vi.mocked(api.setupStatus).mockResolvedValue({ configured: false });
const me = vi.spyOn(api, "me").mockResolvedValue(account());
mount();
await screen.findByTestId("app");
expect(me).not.toHaveBeenCalled();
expect(sessionStorage.getItem(ACTIVE_ACCOUNT_KEY)).toBeNull();
});
it("an unanswered setup probe is an error with a retry — never the signed-out landing", async () => {
vi.mocked(api.setupStatus)
.mockRejectedValueOnce(new Error("502"))
.mockResolvedValue({ configured: true });
vi.spyOn(api, "me").mockResolvedValue(account());
mount();
fireEvent.click(await retryButton());
await screen.findByTestId("app");
expect(renders.length).toBeGreaterThan(0);
expect(sessionStorage.getItem(ACTIVE_ACCOUNT_KEY)).toBe("5"); // it mounted for the account
});
it("an unanswered first account probe is an error with a retry", async () => {
vi.spyOn(api, "me").mockRejectedValueOnce(new Error("502")).mockResolvedValue(account());
mount();
fireEvent.click(await retryButton());
await screen.findByTestId("app");
expect(sessionStorage.getItem(ACTIVE_ACCOUNT_KEY)).toBe("5");
});
it("a background refetch that fails keeps the app mounted, as it was", async () => {
const me = vi.spyOn(api, "me").mockResolvedValue(account());
const qc = mount();
await screen.findByTestId("app");
me.mockRejectedValue(new Error("502"));
await act(() => qc.refetchQueries({ queryKey: qk.me() }));
// react-query hands the error to its observers on a timer: let that render land before asserting.
await act(() => new Promise((r) => setTimeout(r, 20)));
expect(qc.getQueryState(qk.me())?.status).toBe("error");
expect(screen.getByTestId("app")).toBeTruthy();
expect(mounts).toBe(1);
});
it("a pin naming another account than the server answered as is corrected before the app reads it", async () => {
// Pinned to 9, but 9 was signed out in another tab: the server answers as the browser's default, 5.
sessionStorage.setItem(ACTIVE_ACCOUNT_KEY, "9");
localStorage.setItem("siftlode.startOnHome.5", "0");
localStorage.setItem("siftlode.page.5", "downloads");
vi.spyOn(api, "me").mockResolvedValue(account());
mount();
expect((await screen.findByTestId("app")).textContent).toBe("downloads");
expect(renders.map((r) => r.page)).toEqual(renders.map(() => "downloads"));
expect(sessionStorage.getItem(ACTIVE_ACCOUNT_KEY)).toBe("5");
});
it("a mounted tab whose identity changes reloads rather than keep the previous account's tree", async () => {
const me = vi.spyOn(api, "me").mockResolvedValue(account());
const qc = mount();
await screen.findByTestId("app");
const reload = vi.fn();
vi.stubGlobal("location", { ...window.location, reload });
me.mockResolvedValue(null); // the session ended
await act(() => qc.refetchQueries({ queryKey: qk.me() }));
await waitFor(() => expect(reload).toHaveBeenCalled());
});
it("an identity change reloads without moving the pin first", async () => {
// The tree being torn down still runs its unmount saves before the page goes; with the pin already
// moved they would land under the NEW account's local keys (a player volume still pending, say).
const me = vi.spyOn(api, "me").mockResolvedValue(account());
const qc = mount();
await screen.findByTestId("app");
const reload = vi.fn();
vi.stubGlobal("location", { ...window.location, reload });
me.mockResolvedValue(account({ id: 6 })); // 5 was signed out elsewhere: the server answers as 6
await act(() => qc.refetchQueries({ queryKey: qk.me() }));
await waitFor(() => expect(reload).toHaveBeenCalled());
expect(sessionStorage.getItem(ACTIVE_ACCOUNT_KEY)).toBe("5");
});
it("a change to the account's data, not its identity, leaves the app as it is", async () => {
vi.spyOn(api, "me").mockResolvedValue(account());
const qc = mount();
await screen.findByTestId("app");
const reload = vi.fn();
vi.stubGlobal("location", { ...window.location, reload });
// What every pref save does to the cache (usePref's mergePrefsCache): a new `me` for the same account.
act(() => qc.setQueryData(qk.me(), account({ preferences: { language: "hu" } })));
await act(() => new Promise((r) => setTimeout(r, 20)));
expect(reload).not.toHaveBeenCalled();
expect(mounts).toBe(1);
expect(i18n.language).toBe("en"); // the language is adopted once, never over the user's own choice
});
});
+124
View File
@@ -0,0 +1,124 @@
import { useLayoutEffect, useState, type ReactNode } from "react";
import { useQuery } from "@tanstack/react-query";
import { useTranslation } from "react-i18next";
import { api, getActiveAccount, setActiveAccount } from "../lib/api";
import { qk } from "../lib/queryKeys";
import i18n, { isSupported, setLanguage } from "../i18n";
import { LoadingState, StateMessage } from "./ui/QueryState";
// The boot sequence, in ONE place: nothing of the app renders until this tab knows WHO it is.
//
// Per-account state is everywhere — the landing page, the feed's filters, the Plex scope, the manager's
// tab, the theme, the language — and all of it is read synchronously when its provider mounts. Those
// providers used to mount before /api/me answered, so a new tab first built the app for "no account"
// (defaults: Home instead of the page you left, the default theme, the browser language) and then
// patched each piece once <App> pinned the account. Every patch was a visible jump at best, and at
// worst it read a value the first pass had already written (the landing page mirrored ?page=home into
// the address bar, then re-read its own mirror and stayed on Home). So the order is inverted instead:
// settle the identity, adopt what must be adopted before a single pixel of the app is drawn, THEN mount
// the provider tree — whose initializers are then simply right, with nothing to re-read.
//
// The per-account initializers read the tab's PIN (sessionStorage), not `me`, so the pin is made to
// match the account the server answered as before they run. And the tree is only ever built for one
// identity: if a mounted tab's identity changes without a reload (its session ended, or the account it
// was pinned to was signed out in another tab and the server now answers as the browser's default),
// the tab reloads, like every account switch does (NavSidebar) — which also drops the previous
// account's query cache, whose keys are not per account.
/** The quiet screen shown while the tab's identity (or the app chunk) is still on its way — the shared
* loading visual, full screen. Its reveal counts from navigation start, so the gate's screen, the
* Suspense fallback after it and the first page's own loading state keep one clock (see LoadingState). */
export function BootScreen() {
return (
<div className="min-h-screen grid place-items-center">
<LoadingState since={0} delayMs={400} />
</div>
);
}
export default function BootGate({ children }: { children: ReactNode }) {
const { t } = useTranslation();
// First-run install gate: a fresh instance reports configured=false and runs in setup mode, where
// every other route is locked — so `me` is held back until the instance is known to be set up
// (otherwise it would 503 against the setup-mode lock).
const setupQuery = useQuery({
queryKey: qk.setupStatus(),
queryFn: api.setupStatus,
staleTime: Infinity,
});
const needsSetup = setupQuery.data?.configured === false;
// THE fetching `me` query; every other reader is a passive (enabled:false) observer of its cache.
// Polled while signed in so an ended session is noticed with no interaction: /api/me answers a
// signed-out (or suspended) caller with 200 and no account, the identity drops to "nobody", and the
// tab reloads to the login page. Not polled signed out — that refetch is pointless and used to
// flicker the public landing.
const meQuery = useQuery({
queryKey: qk.me(),
queryFn: api.me,
enabled: setupQuery.isSuccess && !needsSetup,
refetchInterval: (query) => (query.state.data ? 60_000 : false),
});
const me = meQuery.data;
// Who this tab is, once known: an account id, `null` for "nobody" (signed out, or setup mode — App
// renders the landing / the wizard for those), `undefined` while unknown or unanswerable.
// Keyed on the DATA, not the status: a background refetch that fails keeps the last answer, so a
// server blip never tears the whole app down to this screen.
const identity: number | null | undefined = !setupQuery.data
? undefined
: needsSetup
? null
: me === undefined
? undefined
: (me?.id ?? null);
// Adopt the identity before the tree mounts. A layout effect, so the re-render it triggers happens
// before the browser paints: the frame that first shows the app already has the account pinned.
// • Pin this tab to the account the server answered as. Without the pin, a tab riding the
// session's default account would silently swap identity when ANOTHER tab changes that default
// (adds or switches an account); once pinned it sends its own account header. The server honours
// that header only for an account still signed in to this browser, so a pin naming one that is
// not is corrected here — otherwise the tab would run as the default account on the server while
// reading and writing the gone account's local state.
// • Adopt the account's language, which follows the account across devices — the local copy only
// knows what this browser last showed.
const [readyFor, setReadyFor] = useState<number | null | undefined>(undefined);
useLayoutEffect(() => {
if (identity === undefined) return;
if (readyFor !== undefined) {
// The tree was built for another identity — see the header. The pin stays as it is: the tree
// being torn down still runs its unmount saves before the page goes, and with the pin moved they
// would land under the new account's local keys. The reloaded page's first probe corrects it.
location.reload();
return;
}
if (me && getActiveAccount() !== me.id) setActiveAccount(me.id);
const lang = me?.preferences?.language;
if (typeof lang === "string" && isSupported(lang) && lang !== i18n.language) setLanguage(lang);
setReadyFor(identity);
// `me` is read for the identity it produced; a background refetch of the same account must not
// re-run the adoption over a language the user has just changed.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [identity]);
// No answer at all (a network or 5xx failure on the very first setup or account probe — past the
// one retry `req` already makes): say so, with a retry. An unanswered setup probe is NOT "signed
// out": rendering the login page to a signed-in user would be a lie. After a retry the identity
// lands and the tree mounts from scratch, like any first load.
const failed =
!setupQuery.data && setupQuery.isError
? setupQuery
: me === undefined && meQuery.isError
? meQuery
: null;
if (failed)
return (
<div className="min-h-screen grid place-items-center">
<StateMessage tone="error" onRetry={() => failed.refetch()}>
{t("common.somethingWrong")}
</StateMessage>
</div>
);
if (identity === undefined || readyFor !== identity) return <BootScreen />;
return <>{children}</>;
}
+10 -5
View File
@@ -21,8 +21,8 @@ import { LoadingState, StateMessage } from "./ui/QueryState";
import ColumnSortControl from "./ui/ColumnSortControl";
import Pager from "./ui/Pager";
import { PageToolbar } from "./ui/PageShell";
import { useConfirm } from "./ConfirmProvider";
import { useChannelLayout } from "./ChannelsProvider";
import { useConfirm } from "./confirmContext";
import { useChannelLayout } from "./channelsContext";
// The Channel manager's "Discovery" tab: channels that turn up in the user's playlists but
// that they don't subscribe to. Subscribing here only creates the subscription + enriches
@@ -108,9 +108,13 @@ export default function ChannelDiscovery({
}
>
<button
onClick={() => onSubscribe(c)}
disabled={!canWrite || subscribe.isPending}
className="glass-card glass-hover inline-flex items-center gap-1.5 px-2.5 py-1.5 rounded-lg text-xs disabled:opacity-50 transition"
// Busy blocks the click, not the focus: a focused button that turns natively `disabled`
// drops the keyboard to <body> (the rule and its measurement: ui/Button).
onClick={() => !subscribe.isPending && onSubscribe(c)}
disabled={!canWrite}
aria-disabled={subscribe.isPending || undefined}
aria-busy={subscribe.isPending || undefined}
className="glass-card glass-hover inline-flex items-center gap-1.5 px-2.5 py-1.5 rounded-lg text-xs disabled:opacity-50 aria-disabled:opacity-50 transition"
>
<UserPlus className="w-3.5 h-3.5" />
{t("channels.discovery.subscribe")}
@@ -238,6 +242,7 @@ export default function ChannelDiscovery({
sort={sort}
onSortChange={setSort}
emptyText={t("channels.discovery.empty")}
keyboardNav
/>
) : (
<ChannelLayoutGrid
@@ -0,0 +1,87 @@
// @vitest-environment jsdom
//
// The channel manager's card layout as a keyboard list (E11 S4c): each card is an item, and Enter opens
// the channel through the name ChannelLink renders. VirtualGrid is replaced by a stand-in that mounts
// every item — jsdom has no layout for the virtualizer to measure — and registers with listNav exactly
// when the real one would: when it is asked to (`keyboardNav`).
import { Fragment, useRef } from "react";
import { afterEach, describe, expect, it, vi } from "vitest";
import { act, cleanup, render } from "@testing-library/react";
import type { VirtualGridProps } from "./ui/VirtualGrid";
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
return { ...actual, useTranslation: () => ({ t: (key: string) => key }) };
});
vi.mock("./ui/VirtualGrid", async () => {
const { useListNav } = await import("../lib/listNav");
return {
default: function GridStandIn({
items,
getKey,
renderItem,
keyboardNav,
}: VirtualGridProps<unknown>) {
const ref = useRef<HTMLDivElement>(null);
useListNav(ref, { enabled: !!keyboardNav });
return (
<div ref={ref}>
{items.map((item, i) => (
<Fragment key={getKey(item)}>{renderItem(item, 0, i)}</Fragment>
))}
</div>
);
},
};
});
import ChannelLayoutGrid from "./ChannelLayoutGrid";
import ChannelLink from "./ChannelLink";
import { type Column } from "./ui/DataTable";
afterEach(cleanup);
type Ch = { id: string; title: string };
const rows: Ch[] = [
{ id: "UCa", title: "Alpha" },
{ id: "UCb", title: "Beta" },
];
function press(key: string) {
act(() => {
(document.activeElement ?? document.body).dispatchEvent(
new KeyboardEvent("keydown", { key, bubbles: true, cancelable: true }),
);
});
}
const focusedTitle = () =>
(document.activeElement as HTMLElement).querySelector("[data-nav-open]")?.textContent;
describe("ChannelLayoutGrid · keyboard", () => {
it("walks the cards with j/k and opens the channel on Enter", () => {
const opened: string[] = [];
const columns: Column<Ch>[] = [
{
key: "channel",
header: "Channel",
cardPrimary: true,
render: (c) => (
<ChannelLink
id={c.id}
title={c.title}
thumbnailUrl={null}
onView={() => opened.push(c.id)}
/>
),
},
];
render(<ChannelLayoutGrid rows={rows} columns={columns} rowKey={(c) => c.id} />);
press("j");
expect(focusedTitle()).toBe("Alpha");
press("j");
expect(focusedTitle()).toBe("Beta");
press("Enter");
expect(opened).toEqual(["UCb"]);
});
});
@@ -3,6 +3,7 @@ import { useTranslation } from "react-i18next";
import { type Column } from "./ui/DataTable";
import VirtualGrid from "./ui/VirtualGrid";
import { CHANNEL_LAYOUT_SPEC } from "../lib/channelLayout";
import { navItemAttrs } from "../lib/listNav";
// The channel manager's `cards` layout (E4 S3), built by reusing the SAME `Column<T>` render closures
// the DataTable draws — so the subscribed and discovery tables (which have different columns) both get
@@ -12,6 +13,9 @@ import { CHANNEL_LAYOUT_SPEC } from "../lib/channelLayout";
// Named slots place the columns that need bespoke positions; everything else flows in the meta line.
// Unlike DataTable's cramped narrow-screen card fallback, the desktop card is roomy, so it shows the
// `hideInCard` columns too (About, priority) — the user wants those affordances in every layout.
//
// Each card is a keyboard item (lib/listNav): `j`/`k` and the arrows walk the cards, and Enter opens
// the channel — the name ChannelLink renders is the card's open element.
interface ChannelLayoutGridProps<T> {
rows: T[];
@@ -52,9 +56,10 @@ export default function ChannelLayoutGrid<T>({
// Everything else flows in the meta line: not the heading, not a slotted column.
const meta = columns.filter((c) => !c.cardPrimary && !slotted.has(c.key));
const renderCard = (row: T) => (
const renderCard = (row: T, index: number) => (
<div
{...rowProps?.(row)}
{...navItemAttrs(index)}
className={`glass-card rounded-xl p-3 h-full flex flex-col gap-2 ${rowClassName?.(row) ?? ""}`}
>
<div className="flex items-start justify-between gap-2">
@@ -91,7 +96,8 @@ export default function ChannelLayoutGrid<T>({
minCol={spec.minCol}
colMaxWidth={null}
rowEst={spec.rowEst}
renderItem={renderCard}
keyboardNav
renderItem={(row, _itemWidth, index) => renderCard(row, index)}
/>
);
}
+3
View File
@@ -30,6 +30,9 @@ export default function ChannelLink({
{onView ? (
<button
onClick={onView}
// The channel's "open" element: in a keyboard-walkable list (lib/listNav) Enter on the row or
// card clicks this.
data-nav-open=""
onAuxClick={(e) => {
// Middle-click opens the channel's YouTube page in a new tab; left-click keeps
// opening the in-app channel detail.
+1 -1
View File
@@ -11,7 +11,7 @@ import { urlWithParams } from "../lib/urlState";
// The href is the same URL `openChannel` pushes — `?chan=<id>` on the current page — so a new tab
// lands where the in-app navigation would have. A plain left click is still handled in-app, because
// the SPA path does more than navigate (the player has to close first and reclaim its own
// Back-to-close history entry; see closeThenNavigate).
// Back-to-close history entry; see lib/history.closeOverlaysThenNavigate).
type Props = Omit<AnchorHTMLAttributes<HTMLAnchorElement>, "href" | "onClick" | "title"> & {
channelId: string;
title: string | null | undefined;
@@ -16,11 +16,11 @@ vi.mock("react-i18next", async (importOriginal) => ({
...(await importOriginal<typeof import("react-i18next")>()),
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
}));
vi.mock("./NavigationProvider", () => ({
vi.mock("./navigationContext", () => ({
useNavigationActions: () => ({ closeChannel: vi.fn(), openChannel: vi.fn() }),
}));
vi.mock("./ConfirmProvider", () => ({ useConfirm: () => async () => true }));
vi.mock("./FeedFiltersProvider", () => ({
vi.mock("./confirmContext", () => ({ useConfirm: () => async () => true }));
vi.mock("./feedFiltersContext", () => ({
useFeedFilters: () => ({ includeNormal: true, includeShorts: false, includeLive: false }),
}));
vi.mock("./Feed", () => ({ default: () => null }));

Some files were not shown because too many files have changed in this diff Show More