A person's TMDB bio was fetched only when their page was first opened, off the request path, so the first
open showed no bio and the page never asked again (2 of 18,202 linked people were ever enriched).
- people_enrich scheduler job (every 10 min, 300 per run): linked people's bios from TMDB and their
portraits into the image cache, the most-credited people first. A failed TMDB lookup ends the batch.
- Portraits are resized through Plex's photo transcoder to a 320 box (~15 KB instead of ~170 KB; the
person-image proxy uses the same cache key, and a full-size original is removed as its small twin lands).
- The person response says `enriching` while a bio is on its way (a person the job hasn't reached yet).
- "Known for" is ranked by vote count, without guest visits as themselves (talk shows had crowded out the
films: Tom Cruise had 14 of 15), with directed works, from a 40-deep pool; the page shows up to 15.
Migration 0093 un-stamps enriched people so the job refetches them.
- A phased job result (backfill, people_enrich) now counts as a change for the audit trail; before, an
automatic backfill run that added videos was never audited.
- HU: the watch reconcile job reads "egyeztetés" (a leftover from the S5 sweep).
A second review round over the first round's fixes, which is where the remaining
real bugs usually are. All four are the same failure as the round it reviewed: a
claim stated in a comment that the code does not hold.
The middleware moved inside CORS, but the setup gate is registered LATER and is
therefore OUTSIDE it — so the comment's "the setup gate never runs for a body that
is already too large" was false the moment it was written. On an unconfigured
instance that gate opens a DB session per request, which is exactly the anonymous
pre-auth work SEC-20 exists to remove. The cap now sits between them, and the
order is asserted against `app.user_middleware` rather than described: prose about
a stack that `add_middleware` builds inside-out is how this got backwards.
The edge ceiling and the route's own cap were pinned EQUAL, but they measure
different things — the edge counts raw body bytes, the route counts the decoded
file. A file of exactly the advertised maximum was refused one layer before the
route that would have taken it. The edge now allows for the multipart envelope,
and the test asserts the relation and the gap instead of equality.
`sysconfig`'s new `max` was enforced on write only. That bound did not exist
before this epic, so a value that was legal when it was written can be sitting in
the database — served unclamped it produces exactly the drift the comment says is
impossible, with the admin page showing a number outside its own range. Int
overrides are clamped on read now, which also survives an env default being
lowered later.
And `DOWNLOAD_UPLOAD_MAX_BYTES=0` meant "no per-file cap" to the route but a hard
zero to the edge: every upload refused, including the admin page's own way back,
recoverable only by editing env and restarting. The setting takes `ge=1`.
Two test gaps from the same round: the header assertion that is the ONLY check of
the middleware's position was DB-gated and skipped on a host without Postgres, and
the headroom "ratio" the docblock calls load-bearing was held as `<=`, i.e. no
ratio at all. Both fixed, plus a sweep that every `capped_body` call site in the
routers is known and fits under its ceiling — the point of that list is the third
entry, not the two that exist today.
- admin config group "Bug reports": Gitea address, repository, token (secret),
and "Check the connection" — the repository must answer, take issues and
be PRIVATE (a report carries its reporter's log); checked on every send too
- POST /api/debug-sessions/{id}/issue sends the SAVED draft: create the issue,
upload each file on its own request (the measured API), then show the
screenshots inline; progress is kept in the draft, so a send that stops
half-way keeps its issue and the retry sends only what is missing; 5 new
issues an hour per user; audited as issue_create
- the composer: Create issue (only where a tracker is set up, naming the
repository), a partial-send banner with retry, and a Preview — the tracker's
own rendering in a sandboxed frame, or the zip's markdown without one
- the recorder's refusals answer in the localized family (debug_session_*,
screenshot_*, issue(s)_*), not bare codes the UI could only call generic
- a tracker refusal logs the status only: its message can quote the title back
HLS sessions were keyed by the item, so a second account opening the same
film killed the first one's session; the displaced player reclaimed at its
playhead and evicted the other in turn. The registry is now keyed by
(viewer, rating_key), segment requests resolve by both, and the session
directory carries the viewer (the orphan sweep still recognises the older
shapes).
Only transcodes counted toward a cap, so remuxes — each a whole segmented
copy of the film on the scratch volume — were bounded by the library size.
Three bounds now apply after every start, each dropping the least recently
used: two sessions per account, the transcode cap, and a new admin knob,
plex_max_sessions (default 8), over every session. A superseded session
request returns at once, since it waits on the killed ffmpeg, so an account
cannot park more request workers than it may hold sessions.
The subtitle extractor's full-pass slots were global and minutes long each;
one account opening subtitles on film after film could hold them all. An
account may now hold three of the six, and still gets windows past that.
YouTube's RSS goes dark per channel for hours (prod 2026-09-18/19: 393 of
396 feeds answered 404 for four hours, sticky per channel, so a retry does
not help), and nothing else read a followed channel's new uploads. A
channel whose feed has not answered for `rss_fallback_hours` (admin knob,
default 2, 0 = off) now gets one page of its uploads playlist through the
Data API (1 unit), at most once per window, yielding to the backfill
reserve. Spend is attributed to its own quota action.
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.
What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
degraded to a software transcode with nothing saying why; a failed audio probe
collapsed a multi-audio file to one track and the player showed no language
picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
own root, leaving every crashed session's segments on disk forever.
Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.
ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.
logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.
tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
The explore purge rebuilt followed_channel's two EXISTS clauses inline, so what is polled and what may be deleted could drift; it now uses the predicate, which moved to sync/explore.py. The env validator and sysconfig each folded a choice value on their own; both call config.canonical_choice. The deep-backfill comment no longer calls recent backfill unconditional.
The log now has a stored, queryable copy for the diagnostics console:
every WARNING and above plus the request line of every request, kept
for log_retention_days (default 7, no "forever") and pruned hourly by
the new log_gc job.
Rows are built on the logging thread from the same redacted fields as
the stdout line, queued without blocking and written in batches by one
background thread on its own connection. A full queue or a failed write
drops and reports instead of retrying. Flood control keeps 30 of any
one message per minute plus a count of the rest; anonymous request
lines are capped at 60 a minute per client, and the client address is
never stored. Deleting an account deletes its stored events.
Measured on prod first: a YouTube RSS outage wrote 11k warnings in four
hours, one per channel per poll. The poll now writes one warning with
per-reason counts and a few examples, and a failure on our side gets
one traceback per poll instead of one per channel.
log_level, search_source, download_layout and the transcode hw/preset
were typed by hand, and every consumer carried its own guess at what a
typo meant. The value set is now a Literal in config.py: pydantic
rejects a bad env value at boot (any letter case accepted), sysconfig
refuses a bad override and derives the admin picker's options from the
same Literal, and the consumers read the value as-is.
The Configuration page renders a choice key as a segmented bar, or a
dropdown past six options, with translated option names. Its tabs now
follow the registry's order instead of a hand-kept copy that had
already dropped the explore group.
A single-choice FilterSelect now opens on the option in use: in the
browser it could land on the first row instead.
A YouTube link can fail with "Sign in to confirm you're not a bot" while every
other YouTube download works — YouTube gates INDIVIDUAL videos behind a
signed-in session. Measured on prod job 273: all twelve of yt-dlp's player
clients fail on that one video while its neighbours download fine, the bgutil
PO-token sidecar is live and answering, and the newest yt-dlp fails identically.
So the download is not broken by our stack or a stale yt-dlp, and no retry,
client switch or upgrade gets past it. A cookie is the only way in.
The failure copy already sends the user to Settings -> Account -> Download
authentication, but YouTube was not in the cookie host registry, so that
dropdown could not help them: the app asked for something it did not offer.
credentials: YouTube joins SUPPORTED_HOSTS (youtube.com, youtu.be,
youtube-nocookie.com). The Settings dropdown, the SEC-12 host scope and the
private-asset cache salt all derive from the row, so that is the whole wiring.
adapters: a signed-in YouTube download runs only cookie-capable player clients.
Measured against yt-dlp's own INNERTUBE_CLIENTS table: with account cookies
attached yt-dlp REMOVES every client whose SUPPORTS_COOKIES is False, and a set
it empties that way raises "No player clients have been requested" — which is
not a per-client flake it retries past. Our primary set is android_vr, one of
those, so without this the cookie would have failed the download on attempt 1
and never reached the POT-backed web fallback it exists for. A test pins the
client list against the installed yt-dlp so a version bump turns the gate red
instead of silently un-authing every YouTube download, and the empty-set message
joins the retriable markers as the second line of defence.
errors: catalog entries for the wall with and without a cookie, so a user who
has already added cookies is told they were rejected (YouTube rotates exported
ones) rather than to add some. The marker is quote-agnostic: YouTube writes the
apostrophe as U+2019, so "you're not a bot" with a straight quote never matches
the real text. EN + HU copy.
A deployment printed four different log formats and none of them could be
joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in
main.py for anything not started by uvicorn, logging.basicConfig() in the worker
and in the download child (no timestamp at all), and alembic's fileConfig.
app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log
data model, no SDK): ts, level, severity, source, logger, event code, message,
attributes, plus the correlation the context carries — request id, account,
download job.
Redaction is applied at the HANDLER, not at the call sites: every message, every
attribute and every traceback is scrubbed on the way out, so a secret cannot
leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies,
secret config values and email addresses never reach a line.
The access log is ours now (`http.request`): the route template instead of the
concrete path, the status, the duration and the account — and never the query
string, which uvicorn's access log printed on every single request (`?account=`,
`?source=`, the `?token=` of a share link). A well-formed client request id is
accepted and echoed in X-Request-Id, so a browser line and a server line join up.
The account comes from `request.state`, filled by the auth dependency: the
middleware is the outermost one, so it runs outside SessionMiddleware and in a
different task context.
`log_level` becomes an admin knob (sysconfig), applied without a restart and
re-applied at startup — best-effort on purpose, since a logging knob must never
be what stops the app from booting. Every compose service gets bounded json-file
logs (3×50 MB): Docker keeps every line forever by default, and the API now
writes one line per request.
Verified on the localdev stack: JSON everywhere including alembic and the
worker, the query strings gone, a supplied X-Request-Id echoed and logged, and
three /healthz hits of which only the one made while the level was DEBUG
produced a line.
Deep-review round 4, targeted at round 3's changes (f569b19c..e7f38396).
Accounts
- Switching, adding or signing out an account moved the tab's pin before
the page unloaded, so the players' resume beacons fired on the way out
under the NEW account: one account's playback position saved into the
other. The pin is now handed to the next page (pinOnNextPage) and taken
over at import; clearActiveAccount is gone.
- Switching to a suspended account was refused but left it in the
switcher, refused on every click; a deactivated one was not refused at
all, and made the default it signed every account in the browser out on
the next request. Every account _session_user would refuse — gone,
deactivated, suspended, revoked — is refused and dropped from the list.
Share page (/watch)
- An IPv6 client is limited by its /64 (client_ip becomes client_key, for
every limiter): keyed on the full address, one subscriber could rotate
through endless identities. The comments now state what the per-link
ceiling costs: guessing from five networks can keep a link shut for its
window, and guesses already running when it fills are still checked.
- The link is looked up before the limiter records the attempt, so made-up
tokens no longer grow it by an entry each.
Forms
- Failing the same way twice is announced again: the sign-in, passphrase
and setup errors are keyed on the attempt (a clear and a set in one
handler left the alert node untouched).
- The install wizard's own validation showed the generic error — it threw
plain objects its catch did not read. It sets the step's message.
Tests: a re-pin leaving the previous account's notices behind; the pin
hand-over; switching to an ended account; the /64 viewer; made-up tokens;
the repeated alerts. Each mutation-checked RED for its own reason.
Deep-review round 3 over main..improvement/watch-background-audio.
Account boot
- The notification history and settings, and the hints toggle, are read
when their modules load, which in a new tab (bookmark, typed URL) is
before BootGate pins the account: they read as nobody's, and the tab's
first notice replaced the account's stored history (up to 100 entries)
with a one-item list. They now read again when the tab is pinned
(onAccountPinned, run by setActiveAccount); a notice raised before the
pin stays, after the history.
- A tab pinned to an account still in this browser's wallet but whose
session ended (suspended, deleted, or revoked by "log out everywhere")
got "signed out" from /api/me while the default account was still
signed in, and signing in again as anyone else left it there. A refused
per-tab account now leaves the wallet, and the probe answers as the
default, so the tab reloads onto it.
Share page (/watch)
- The password unlock is limited per viewer (link + client IP), under a
per-link ceiling that counts only wrong passwords. Keyed on the link
alone, ten wrong guesses every five minutes locked out every viewer,
the ones with the password included. client_ip moves to app.ratelimit
so the public routes can key on it.
- Its failure messages and the load-failure screen are announced
(role=alert, the password input described by its error). The sign-in,
passphrase, password-change and setup form errors get role=alert too.
Comments that still described removed machinery are corrected: App's
collapse flags, SettingsPanel's Save bar, ConfigPanel, authed_format_sig.
Tests: the account's history and hints across the pin; a revoked and a
suspended pinned account; the per-viewer limit and the ceiling; the
limiter's exhausted(); the share page's announced failures; a share
link's feed filters persisted. Each mutation-checked RED for its own
reason.
Bug hunt 2026-09, item 14 — "how does the multichannel downmix work, and is it
normalised?" It isn't: `plex/stream.py` has always encoded `-c:a aac -ac 2 -b:a
192k`, i.e. fixed stereo with ffmpeg's default downmix matrix and no loudness
handling at all, which is why a film mastered for a cinema arrives with whispered
dialogue and deafening action.
New admin knob `plex_audio_loudnorm` (Configuration → Plex), DEFAULT OFF: it makes
a film markedly louder and narrows its dynamic range, which is a taste call rather
than a fix. It sits at the end of the Plex group, after the software-transcode
knobs — it is an audio concern, not one of the libx264 ones.
-af aformat=channel_layouts=stereo,loudnorm=I=-16:TP=-1.5:LRA=11,aresample=48000
Three things about that chain are load-bearing and each has a test:
* the downmix is FORCED FIRST rather than left to `-ac 2` — measuring 5.1 and
folding it down afterwards would normalise something other than what is heard;
* the trailing resample is not decoration: `loudnorm` runs its own resampler at
192 kHz internally, which the AAC encoder will not take;
* one `-af`, no stream index, so EVERY audio rendition of a multi-audio file is
normalised — not just the first.
The knob is also part of the session directory name (`…_multi-ln_+0.00`), because
it changes the audio the segments contain, and a session left over from before the
admin flipped it must not be mistaken for one that honours it. The suffix is
appended with a DASH: `_SESSION_DIR_RE` treats the tag as one underscore-free
field, and that regex is the only thing keeping the orphan sweep off what may be a
shared scratch path. A test pins both halves — the name changes, and the sweep
still recognises it.
Measured end-to-end on the app's own HLS segments (a real 5.1 AC3 file, same item,
same 160 s offset, same ten segments): OFF -35.2 LUFS, ON -17.2 LUFS. On a 60 s
bench: I -27.6 → -16.6 LUFS with LRA 13.3 → 9.8 LU. A/V sync is untouched — under
this pipeline's own `-copyts` the audio start PTS is identical with and without the
filter, so the 3 s lookahead desyncs nothing.
Direct-played files are served untouched and stay so; a 5.1 mix is always `remux`,
so the reported case is fully covered.
Deep-review round 2 on the round-1 fix:
- Race (medium): the round-1 identity guard did the supersede check and the
respawn in TWO separate _lock sections. A newer same-key session installed in
the gap was still torn down by _spawn's unconditional pop+kill (the round-1
comment even had the direction backwards). Fold the check into _spawn via a
new replace_only= param: the identity test and the pop+replace now share ONE
lock, so the respawn is a true no-op when superseded/culled.
- Honest states (low): the superseded branch returned current_session(key),
which is None when the key was cap-culled during our out-of-lock wait; the
route turns None into a dishonest 'Media file not found on disk' 404 for a
file that exists. Yield 'current_session(key) or s' so a culled session
degrades to the route's retryable 'not available yet' path instead.
- Hygiene: _want_hw docstring corrected (it is a test-only shim now; production
decides via _select_encode) and the two stale _want_hw references refreshed
(stream.py header, config.py plex_transcode_hw).
- tests: the cap-cull no-respawn/no-None sub-case; the _opencl_present except
OSError branch (glob raising, not just empty); tightened a tautological -vf
assertion. Backend gate green (ruff + full pytest).
The software-fallback re-encode now runs deprioritized and admin-configurable.
The SW transcode ffmpeg child is niced (preexec_fn → os.nice, applied per-child
because the Popen lives in the long-lived API process, which must NOT itself be
niced — unlike the download worker's whole-process _deprioritize) so a CPU-bound
re-encode can't starve request handling or playback; the VAAPI path needs none of
it.
New sysconfig knobs (plex group, rendered on the admin Configuration page, i18n
HU/EN): plex_transcode_hw (auto/on/off, layered onto _want_hw), plex_transcode_
preset, plex_transcode_crf, plex_transcode_max_height (software-only downscale-to
cap; the GPU path encodes at source resolution). plex_max_transcodes stays the
concurrency cap (comment corrected — the fallback is shipped, not "later").
pytest extends the command-builder + _want_hw coverage (preset/crf/max_height
plumbing, the on/off/auto policy). Browser-UAT'd: max_height=240 drops the live
resolution to 330x240; the SW ffmpeg child runs at nice 10 (ppid=api process).
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.
- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
gates library actions. Set at every finalize (download probe, editor carries the
source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
places it in a per-user .uploads tree, and creates an already-ready asset + done job
(the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
(download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
application/octet-stream) on both the in-app and public endpoints, so uploaded
HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API,
Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and
starved web requests + playback until it finished. Full-scope S4 mitigations:
- Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp
descendant): CPU nice increment + best-effort idle ionice class + an
OMP_NUM_THREADS budget, all config-gated and independently disable-able.
- Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode
reserves a core for foreground work instead of claiming all of them.
- Throttle the retention GC's whole-tree orphan sweep (yield IO after each
batch of files) so its rglob can't monopolize disk IO in the API process.
- Idle-back-off the Download Center (2s→15s when nothing queued/running) and
UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the
next fetch after an enqueue (invalidateDownloads refetches immediately).
Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence,
_deprioritize (nice/ionice/OMP branches), GC sweep throttle batching.
Make Plex the source of truth for watch state:
- Write-back during playback: item_progress now pushes the resume position to
Plex on the periodic checkpoint too (not only the final flush), throttled to
one push per item every ~10s (in-memory, single-process rate limiter). The FE
now also fires an immediate (debounced) push on seek.
- Faster incremental pull: 30 → 15 min (plex_watch_sync_interval_min).
- Plex-authoritative pull: _pull_apply resolves a genuine divergence to Plex's
state instead of last-write-wins; the _same_state + synced_to_plex short-circuit
still prevents a freshly-pushed row from bouncing. Removed the now-unused
_LWW_TOLERANCE_S / _sift_ts.
Tests: 5 new in test_plex_watch_sync.py (Plex-wins resume/watched, no-bounce,
throttle final vs periodic).
Matches each mirrored Plex movie/show to TMDB by its stored canonical guid,
fetches once (append_to_response), and PERSISTS the enrichment onto the row —
served from the DB, never a live TMDB call at serve time. Additive: the
Plex-sourced imdb_rating/rating are untouched; tmdb_rating is the distinct
TMDb community score (ratings scope = TMDb + Plex's IMDb/audience, no OMDb/RT).
- config.py: tmdb_api_key + api_base/language/interval/batch defaults.
- sysconfig SPECS: tmdb_api_key (secret, Fernet) + tmdb_language, auto-rendered
in the admin Config page; read like the Plex token.
- migration 0071 (ADD-only nullable, reversible): plex_items + plex_shows gain
tmdb_id / tmdb_enriched_at (watermark) / tmdb_rating / tmdb_vote_count /
budget / revenue / keywords / crew / studios / production_countries /
recommendations / backdrops; plex_people gains tmdb_person_id (the S2
foundation for S6 person bios). No watermark reset needed.
- app/plex/tmdb.py: TmdbClient (mirrors PlexClient; throttled; 404->None,
transient->TmdbError), parse_tmdb / apply_tmdb_detail (one-parser rule),
guid_to_tmdb_id (tmdb: direct, imdb:/tvdb: via /find), enrich_batch (movies
then shows, transient error leaves the watermark NULL to retry, definitive
no-match still stamps so it isn't re-fetched), _link_people (upserts
tmdb_person_id by name), serve_block (recommendations resolved to library
rows by guid), run_enrich (no-op without a key).
- scheduler: tmdb_enrich nightly job (default 120 min, no YouTube quota),
runtime-tunable + "run now" from the Scheduler dashboard.
- detail endpoints emit `tmdb` (movies + shows) so it's curl-verifiable now;
the FE render lands in S5.
Note: _link_people inserts plex_people rows for TMDB cast/crew; unreferenced
crew (composers etc., not in cast_names/directors) get pruned by the orphan
sweep — acceptable, the person page is cast-driven and cast rows survive.
Gate: ruff + py_compile green; backend test lane FULL suite 387 passed
(incl. migration drift downgrade->re-upgrade + test_plex_tmdb 9 tests).
Frontend unchanged since S1. Not merged (epic branch, stacks on S1).
This codebase leans on "why" comments as its architecture doc, so a header that
claims the opposite of the code below it is worse than none. Fix the seven the
review flagged: channels.py (subscribe/unsubscribe shipped), playlists.py +
sync/playlists.py (YouTube two-way sync + write-back shipped), setup.py (drop the
"epic 6c" ref — the wizard steps exist), config.py ("auth._client_ip" ->
"routes.auth._client_ip"), db.py ("routes/plex.py" -> "routes/plex/" after the
package split), sysconfig.py (the "Phase C" note — the watch-sync job that reads
plex_watch_sync_interval_min has landed; clarified it's a boot default, not a knob).
Docs only.
Adds the storage + sync for the Hungarian-metadata overlay (the browsable EN sections gain HU
twin fields written from a paired HU section, matched by guid; no browsable HU rows).
Schema (2 migrations, ADD-only; run before the first HU sync while *_hu is NULL so the generated
column build is trivial):
- 0063: plex_libraries.metadata_lang ('en' default) + self-FK overlay_of.
- 0064: plex_items/plex_shows.guid (indexed) + title_hu/summary_hu/genres_hu (+ GIN) + a STORED
generated search_vector_hu mirroring the EN vector (title_hu A / people_text B / summary_hu C) + GIN.
Sync:
- includeGuids=1 on section_items so the canonical guid rides the cheap listing.
- _canonical_guid(meta) (tmdb > imdb > tvdb) captured onto every EN row in _apply_item/_sync_shows.
- Two-pass sync(): pass 1 mirrors browsable (non-overlay) sections; pass 2 reads plex_overlay_map
('huKey:enKey', new sysconfig), designates each HU section (metadata_lang='hu' + overlay_of), and
_sync_overlay writes title_hu/summary_hu/genres_hu onto the guid-matched EN rows.
Browse: _content_libs now excludes overlay libraries (overlay_of is None); the unified_library and
facets inline library partitions fold into it, and list_libraries/collections exclude overlays too.
Tests: DB-lane coverage of the overlay write (matched vs unmatched rows, no browsable HU rows, the
generated search_vector_hu populates), the guid parse, and the browse exclusion. Full suite 272
passed; migrations verified reversible (upgrade→downgrade→upgrade). S5b (persist the live detail
fields + serve item/show detail from DB) is the follow-up.
Defense-in-depth after the ReDoS incident. The heavy work (yt-dlp download / ffmpeg edit)
now runs in a child process (app.downloads.runner) supervised by the worker thread, not in
the thread itself. A thread can't be killed and a C-level spin can't even be cooperatively
cancelled, so one wedged download used to pin a core, hold the GIL and freeze the whole
queue; a child process can be SIGTERM/SIGKILLed.
The supervisor kills the child (and its process group, so ffmpeg grandchildren die too)
when: a route pauses/cancels the job, it makes no progress for download_stall_timeout_seconds
(150s), it runs past download_wall_clock_max_seconds (2h), the worker is shutting down, or
the child crashes without setting state. On a kill it frees the asset and fails the job with
a retryable network error (stalls are not auto-requeued). A clean exit is left alone — the
child already wrote the terminal state, including a cooperative pause via the progress hook.
- config.py: stall / wall-clock / supervisor-poll / kill-grace settings.
- worker.py: _run_supervised + _spawn_download_child / _job_liveness / _stop_child /
_cleanup_staging; _process_job claims the asset, then hands the heavy work to the child.
- app/downloads/runner.py: the child entrypoint.
- docker-compose.localdev.yml: init:true on the worker so tini (PID 1) reaps orphaned
ffmpeg grandchildren left by a hard-kill (the worker reaps its own direct children).
- tests: watchdog decision logic + SIGTERM->SIGKILL escalation (fake proc, no real subprocess).
_client_ip trusted the first X-Forwarded-For hop unconditionally, so anyone able to
reach the app port could forge XFF and dodge the login/register/reset/demo rate limits.
Now trust XFF ONLY when the request's socket peer is a configured reverse proxy
(settings.trusted_proxy_ips, e.g. the VPS Caddy's WireGuard peer IP), and take the
RIGHTMOST entry — the client our proxy actually saw and appended, immune to a client
pre-seeding a fake XFF. A request from any other peer (hitting the port directly) is
keyed on its real socket IP, so XFF can't be forged to bypass the limits.
New TRUSTED_PROXY_IPS env (empty default = no proxy, use the direct peer). Documented in
.env.example, docs/self-hosting.md, README. Unit-verified against spoof-through-proxy and
direct-bypass cases.
stream.py enforced a hardcoded _MAX_SESSIONS = 4 and never read the
DB-overridable plex_max_transcodes ConfigSpec, so the Configuration → Plex →
"Max concurrent transcodes" knob was dead (same env-vs-DB drift class as the
Downloads/Admin fixes). _enforce_cap(cap) now takes the cap from
sysconfig.get_int(db, "plex_max_transcodes") (>=1 so playback can't be capped to
zero). Bumped the config default 1→4 so the effective default matches the old
hardcoded cap (no behavior change for non-overriders; the knob now works).
Completes the two-way watch-state sync with two scheduler jobs:
- plex_watch_sync (default 30m): pull recent Plex-side changes (watch history + on-deck, filtered
to the owner account) into Siftlode under last-write-wins (_pull_apply + a _same_state ping-pong
guard + skew tolerance), then re-push any still-unsynced local states.
- plex_watch_reconcile (default daily): full section rescan; uses synced_to_plex to settle what the
incremental feed can't — notably propagating a Plex-side un-watch (clear a previously-mirrored row
Plex no longer has) — while re-pushing never-synced local states and never touching hidden
(Siftlode-only) rows. Union-preserving.
PlexClient gains accounts/watch_history/on_deck; _scan_plex_states is factored out and shared with
the one-time import. Owner accountID is resolved once and cached on the link. Both jobs are
registered in the scheduler (pause-skip, activity tracking, run-now, admin-tunable intervals) with
trilingual (HU/EN/DE) labels + descriptions. New config default plex_watch_reconcile_interval_min.
Verified live against the real Plex server: read feeds, last-write-wins, dirty re-push, the
incremental job end-to-end, and a full reconcile that cleared exactly the one un-watched item with
zero collateral across 17976 scanned.
Plex records watch state per Plex account; Siftlode per user in plex_states. A new plex_link table
maps a Siftlode user to a Plex account; for the owner (MVP) the row uses the server admin token
(uses_admin=True), so no separate Plex login. app/plex/watch_sync.py reads the owner account's
viewCount/viewOffset/lastViewedAt (already present in the catalog mirror's section listing) and
upserts them into the owner's plex_states — 'Plex is master' on this first import, but only where
Plex has a watch record (Siftlode-only states are preserved; union on the intersection). Idempotent.
Admin routes: GET/POST /api/plex/watch/link (status + enable/disable; first enable runs the import)
and POST /api/plex/watch/import (re-run). Migration 0051_plex_link. Two-way push (Phase B) and the
incremental Plex→Siftlode reconcile (Phase C) build on this in later ships.
Validated end-to-end on a real remux file (h264+ac3 mkv). The riskiest part of the
Plex epic — on-the-fly playback from the local file — now proven.
- app/plex/stream.py: seek-restart HLS session model (one ffmpeg per item, restarted
at a seek offset via -ss). Video stream-COPY (I/O-bound, CPU-light — fine on the
GPU-less prod host) + audio→aac only when not already aac. Maps first video+audio,
drops subtitles (no VTT rendition clutter). Session cap + idle reaper.
- routes: POST /stream/{rk}/session?start= (direct→raw url, remux→HLS session,
transcode→501 P3), GET /stream/{rk}/file (206 range, direct), /index.m3u8, /seg_n.ts.
- KEY FINDINGS (hard-won): (1) ffmpeg -hls_playlist_type VOD does NOT write the
playlist mid-run (only at finalize) → use EVENT (append-only, appears immediately),
which suits the seek-restart model; (2) naive per-segment copy + keyframe-indexed
extraction are both unreliable (non-uniform/imprecise segments) — ffmpeg's own HLS
muxer is the only correct segmenter; (3) dev slowness was the Windows /downloads
bind-mount write + SMB read → PLEX_HLS_DIR env points scratch at fast container-local
/var/tmp on dev (prod keeps download_root, fast local disk).
YouTube returns flaky per-client responses: the web clients intermittently mislabel normal
videos as 'DRM protected' (like tv did), and android_vr can get bot-flagged under heavy load.
Neither client alone is reliable. So:
- PRIMARY player client = android_vr: high-quality DASH (399+251), no PO token / Deno needed,
reliable for normal use
- on a bot/DRM/format failure the worker retries with the FALLBACK set (web_safari,web,android
+ POT sidecar + Deno) — bypasses bot-detection when android_vr is flagged
- both client sets admin-tunable (DOWNLOAD_PLAYER_CLIENTS[_FALLBACK])
Verified: the previously DRM-failing video now downloads steadily via android_vr, no DRM error.
The web player clients need two things beyond the PO token: (1) a JavaScript runtime for
YouTube's 'n' signature challenge — install Deno (v2.9.1) + yt-dlp[default] (bundles yt-dlp-ejs
solver scripts); auto-detected on PATH. (2) an audio source — the web clients only expose
video-only DASH, so add the 'android' client (also PO-token-backed) for audio/muxed formats.
Player clients: web_safari,web,android (dropped tv — it falsely reports normal videos as DRM).
End-to-end verified in the worker: extraction (POT + Deno n-challenge) + download completes,
no bot/DRM/format errors. Full YouTube stack now: force-ipv4 + bgutil POT sidecar + Deno.
YouTube increasingly demands a Proof-of-Origin token (or shows 'confirm you're not a bot').
Instead of manual cookies, run the bgutil POT provider as a sidecar that mints tokens on demand:
- compose: bgutil-pot service (brainicism/bgutil-ytdlp-pot-provider:1.3.1, port 4416, init)
- requirements: bgutil-ytdlp-pot-provider==1.3.1 plugin (version pinned to the image)
- config: DOWNLOAD_POT_BASE_URL (http://bgutil-pot:4416) + DOWNLOAD_PLAYER_CLIENTS
(web_safari,web,tv — POT-capable; the default android_vr client never requests a token)
- formats.build_ydl_opts wires extractor_args youtube:player_client + youtubepot-bgutilhttp:base_url
Verified: the sidecar mints a valid PO token (BotGuard challenge solved) and yt-dlp fetches it
from the provider. NOTE: end-to-end download couldn't be confirmed here because the test IP got
hard-flagged by YouTube from heavy testing (~16 GB + dozens of extractions) — a hard abuse block
needs cooldown/cookies, which POT doesn't lift; POT bypasses the normal soft bot-check.
- Remove owner-specific / legacy deploy files (home/prod/server compose, deploy/).
The home compose stays as a local untracked file for the maintainer's own deploy.
- Genericise infra-specific code comments (egress-proxy examples) to neutral wording.
- Replace the hardcoded contact email on the legal pages with the instance operator's
configured admin email, served via the public /auth/config and shown with a neutral
fallback — so each self-hosted instance shows its own contact.
- Rewrite README for the current app + a copy-paste self-hosting quick start (prebuilt
image + first-run wizard) with a build-from-source alternative; tidy .env.example.
- Live-search results stay ephemeral: the discovery-cleanup job now also reclaims un-kept
via_search videos (no watch state / not playlisted / channel not subscribed) after a grace
period (search_grace_days), and POST /api/search/clear discards a given result set 'as if
never added' (drops the user's search-finds + deletes the now-orphaned, un-kept videos).
Admin POST /api/admin/purge-discovery runs it on demand (grace 0).
- Count selector: GET /api/search/youtube gains a 'limit' — the free scrape source pages
through continuations until that many results are gathered (no manual load-more); the API
source stays one ≤50 page (cost).
- Per-user channel blocklist (migration 0034 blocked_channels): block/unblock/list endpoints;
blocked channels' videos are dropped from live search before ingest and hidden from the feed
/ Library / explore / channel page; explore refuses a blocked channel.
- New-first ordering: results you already have (subscribed channel, watched/in-progress/saved/
playlisted) sink below genuinely-new discoveries, preserving relevance within each group.
Adds the backend for a dedicated channel page + ephemeral browsing of un-subscribed
channels:
- migration 0033: channels.{total_view_count,published_at,banner_url,external_links,
from_explore}, videos.via_explore, explored_channels (per-user, grace-clocked).
- enrich channels with part=brandingSettings (banner/links) + statistics.viewCount +
snippet.publishedAt — no extra quota.
- GET /api/channels/{id}: About detail + this user's relationship; lazy-enriches the new
About fields (published_at sentinel).
- POST /api/channels/{id}/explore (require_human, quota-reserve guarded): records the
exploration, flags the channel ephemeral unless followed, ingests one page of uploads
(via_explore) + enriches; returns next_page_token for load-more.
- feed: per-explorer gate — a via_explore video is visible only to users who explored or
subscribe to its channel, so exploration never leaks into everyone's catalog.
- subscribe = keep: clears from_explore + via_explore on the channel's videos.
- scheduler explore_cleanup job + explore_grace_days config: hard-delete explored-but-
unkept channels and their untouched ephemeral videos after a grace period.
Live YouTube search can now use YouTube's internal InnerTube endpoint instead
of search.list, materialising results through the same enrich/provenance path
at zero API quota (search.list costs 100 units/page; scrape costs nothing, only
the cheap shared videos.list enrich is charged).
- youtube/search_scrape.py: InnerTube search returning the same page shape as
YouTubeClient.search_videos (items + continuation cursor); SOCS consent cookie,
videoRenderer walk (channels/playlists/Shorts shelves naturally excluded),
type:video filter, cached InnerTube key/version with constant fallbacks.
- routes/search.py: admin-selectable source (search_source, default scrape).
Scrape path skips the 100-unit budget pre-check and logs a zero-cost search
event so the per-user daily cap still counts it; api path unchanged. Response
carries the active source.
- sysconfig/config: new search_source key (scrape|api).
- quota.log_action: record a zero-cost action event for per-user rate limits.
Add a live YouTube search that materialises results into the shared catalog so
they render with the normal feed cards + in-app player and gain per-user state.
- YouTubeClient.search_videos(): search.list (100 units), embeddable-only, returns
flat stubs + nextPageToken; surfaces liveBroadcastContent for live filtering.
- routes/search.py GET /api/search/youtube: require_human + per-user daily cap
(search_daily_limit_per_user, default 70) + can_spend pre-check (429 on either);
drops live/upcoming, upserts channel stubs (channels.list) + video stubs, enriches
(videos.list), runs the youtube.com/shorts probe, then excludes Shorts/live and
returns feed cards in relevance order with the YouTube pageToken as the cursor.
- Provenance: videos.via_search / channels.from_search (migration 0028) flag
search-discovered rows; the feed hides them from the Library (scope=all) by default
via exclude_search_discovered, leaving the Mine feed untouched.
- quota.actions_today() counts a user's per-action events today for the cap; only the
search.list call is attributed VIDEOS_SEARCH so the counter is exactly 1 per search.
New youtube_api_proxy setting (env fallback + admin Config UI, env YOUTUBE_API_PROXY):
when set, the YouTubeClient routes all its httpx traffic through that HTTP(S) proxy.
Lets a dynamic-IP host send API calls through a fixed-IP host (e.g. the server over
private tunnel) so an IP-restricted API key keeps working. Empty = direct.
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
- Link a Google account to a password account, and adopt the Google identity onto a matching
email account instead of 500ing on a duplicate; set or change a password from Settings.
- Expose has_google/has_password on /api/me for the Sign-in methods UI.
- Mark Google logins email-verified (backfill existing rows, migration 0024); stop a routine
login from clobbering an admin-assigned role (env ADMIN_EMAILS stays the bootstrap admin).
- Suspension login-gates (password + Google callback + current_user) with a rate-limited
'suspended' notice; shared purge_user (cascade delete + access-request cleanup + Google-grant
revoke) behind self- and admin-deletion; single app_base source for user-facing email links.
A security audit of every mutating endpoint found no isolation gaps — all routes
scope by current_user, admin routes are gated, and the demo account is sandboxed
in its own user_id space (can't reach others' data, escalate, or hit admin
routes). The remaining demo concern is communal pollution of its own shared state,
so add an automatic reset: a new 'demo_reset' scheduler job (admin-tunable
interval, default 12h) reuses the manual reset logic, and no-ops without ever
creating a demo account if none exists. Verified: a triggered run wiped the demo's
video states and re-seeded its sample playlists.
Add email+password auth alongside Google: argon2id hashing; users gains
password_hash/email_verified/is_active and google_sub becomes nullable (migration
0022); a single-use, hashed auth_tokens table for email verification + password
reset. Registration creates a pending (inactive, unverified) account + an access
request; sign-in needs verified email AND admin approval. Anti-enumeration: uniform
register/login/reset responses (a correct-password owner still gets a specific
pending reason). allow_registration flag (DB-tunable). Admin users-list + role
endpoint (guards: not self, not demo, not the last admin); approving access (or a
manual whitelist add) now activates the matching pending account. current_user
rejects deactivated accounts. Verified end-to-end via curl + DB.
Add a daily maintenance/validation job that detects videos which can't be played
anywhere and retires them safely. Two phases: re-check already-flagged videos
(recover if available again, else hard-delete once the grace period elapses,
cascading to states/playlist items), and a rolling re-validation of the
least-recently-checked currently-available videos that flags newly-unplayable
ones (hidden from the feed immediately via unavailable_since).
Detection is ~free: a video missing from the videos.list response is
deleted-or-private; an `upcoming` premiere >2 days past its scheduled start that
never went live is abandoned. A still-live broadcast is kept (legit 24/7 stream).
Enrichment now also fetches part=status to populate the status columns. Grace is
7 days for removed videos, none for abandoned. Before deleting, affected users
get one batched notification (never per-video). Interval is admin-tunable via the
Scheduler dashboard; batch size and grace are config. Quota-attributed to the
system and bounded by the same backfill reserve as the other jobs.
Mirror each user's own YouTube playlists into local source='youtube' playlists,
one-way (YT -> local). New client methods iter_my_playlists / iter_my_playlist_video_ids
(OAuth, so private playlists work); sync/playlists.py reconciles the mirror (matching YT
order) and ingests any playlist videos not in the shared catalog yet (with stub channels).
POST /api/playlists/sync-youtube for manual sync (read-scope gated, per-user quota) plus a
scheduler job (playlist_sync_minutes, default 6h) that syncs all read-scope users. YouTube's
Watch Later / History are not API-accessible and are never synced.
peter