Deep-review round 2 on the round-1 fix:
- Race (medium): the round-1 identity guard did the supersede check and the
respawn in TWO separate _lock sections. A newer same-key session installed in
the gap was still torn down by _spawn's unconditional pop+kill (the round-1
comment even had the direction backwards). Fold the check into _spawn via a
new replace_only= param: the identity test and the pop+replace now share ONE
lock, so the respawn is a true no-op when superseded/culled.
- Honest states (low): the superseded branch returned current_session(key),
which is None when the key was cap-culled during our out-of-lock wait; the
route turns None into a dishonest 'Media file not found on disk' 404 for a
file that exists. Yield 'current_session(key) or s' so a culled session
degrades to the route's retryable 'not available yet' path instead.
- Hygiene: _want_hw docstring corrected (it is a test-only shim now; production
decides via _select_encode) and the two stale _want_hw references refreshed
(stream.py header, config.py plex_transcode_hw).
- tests: the cap-cull no-respawn/no-None sub-case; the _opencl_present except
OSError branch (glob raising, not just empty); tightened a tautological -vf
assertion. Backend gate green (ruff + full pytest).
The software-fallback re-encode now runs deprioritized and admin-configurable.
The SW transcode ffmpeg child is niced (preexec_fn → os.nice, applied per-child
because the Popen lives in the long-lived API process, which must NOT itself be
niced — unlike the download worker's whole-process _deprioritize) so a CPU-bound
re-encode can't starve request handling or playback; the VAAPI path needs none of
it.
New sysconfig knobs (plex group, rendered on the admin Configuration page, i18n
HU/EN): plex_transcode_hw (auto/on/off, layered onto _want_hw), plex_transcode_
preset, plex_transcode_crf, plex_transcode_max_height (software-only downscale-to
cap; the GPU path encodes at source resolution). plex_max_transcodes stays the
concurrency cap (comment corrected — the fallback is shipped, not "later").
pytest extends the command-builder + _want_hw coverage (preset/crf/max_height
plumbing, the on/off/auto policy). Browser-UAT'd: max_height=240 drops the live
resolution to 330x240; the SW ffmpeg child runs at nice 10 (ppid=api process).
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.
- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
gates library actions. Set at every finalize (download probe, editor carries the
source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
places it in a per-user .uploads tree, and creates an already-ready asset + done job
(the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
(download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
application/octet-stream) on both the in-app and public endpoints, so uploaded
HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API,
Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and
starved web requests + playback until it finished. Full-scope S4 mitigations:
- Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp
descendant): CPU nice increment + best-effort idle ionice class + an
OMP_NUM_THREADS budget, all config-gated and independently disable-able.
- Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode
reserves a core for foreground work instead of claiming all of them.
- Throttle the retention GC's whole-tree orphan sweep (yield IO after each
batch of files) so its rglob can't monopolize disk IO in the API process.
- Idle-back-off the Download Center (2s→15s when nothing queued/running) and
UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the
next fetch after an enqueue (invalidateDownloads refetches immediately).
Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence,
_deprioritize (nice/ionice/OMP branches), GC sweep throttle batching.
Make Plex the source of truth for watch state:
- Write-back during playback: item_progress now pushes the resume position to
Plex on the periodic checkpoint too (not only the final flush), throttled to
one push per item every ~10s (in-memory, single-process rate limiter). The FE
now also fires an immediate (debounced) push on seek.
- Faster incremental pull: 30 → 15 min (plex_watch_sync_interval_min).
- Plex-authoritative pull: _pull_apply resolves a genuine divergence to Plex's
state instead of last-write-wins; the _same_state + synced_to_plex short-circuit
still prevents a freshly-pushed row from bouncing. Removed the now-unused
_LWW_TOLERANCE_S / _sift_ts.
Tests: 5 new in test_plex_watch_sync.py (Plex-wins resume/watched, no-bounce,
throttle final vs periodic).
Matches each mirrored Plex movie/show to TMDB by its stored canonical guid,
fetches once (append_to_response), and PERSISTS the enrichment onto the row —
served from the DB, never a live TMDB call at serve time. Additive: the
Plex-sourced imdb_rating/rating are untouched; tmdb_rating is the distinct
TMDb community score (ratings scope = TMDb + Plex's IMDb/audience, no OMDb/RT).
- config.py: tmdb_api_key + api_base/language/interval/batch defaults.
- sysconfig SPECS: tmdb_api_key (secret, Fernet) + tmdb_language, auto-rendered
in the admin Config page; read like the Plex token.
- migration 0071 (ADD-only nullable, reversible): plex_items + plex_shows gain
tmdb_id / tmdb_enriched_at (watermark) / tmdb_rating / tmdb_vote_count /
budget / revenue / keywords / crew / studios / production_countries /
recommendations / backdrops; plex_people gains tmdb_person_id (the S2
foundation for S6 person bios). No watermark reset needed.
- app/plex/tmdb.py: TmdbClient (mirrors PlexClient; throttled; 404->None,
transient->TmdbError), parse_tmdb / apply_tmdb_detail (one-parser rule),
guid_to_tmdb_id (tmdb: direct, imdb:/tvdb: via /find), enrich_batch (movies
then shows, transient error leaves the watermark NULL to retry, definitive
no-match still stamps so it isn't re-fetched), _link_people (upserts
tmdb_person_id by name), serve_block (recommendations resolved to library
rows by guid), run_enrich (no-op without a key).
- scheduler: tmdb_enrich nightly job (default 120 min, no YouTube quota),
runtime-tunable + "run now" from the Scheduler dashboard.
- detail endpoints emit `tmdb` (movies + shows) so it's curl-verifiable now;
the FE render lands in S5.
Note: _link_people inserts plex_people rows for TMDB cast/crew; unreferenced
crew (composers etc., not in cast_names/directors) get pruned by the orphan
sweep — acceptable, the person page is cast-driven and cast rows survive.
Gate: ruff + py_compile green; backend test lane FULL suite 387 passed
(incl. migration drift downgrade->re-upgrade + test_plex_tmdb 9 tests).
Frontend unchanged since S1. Not merged (epic branch, stacks on S1).
This codebase leans on "why" comments as its architecture doc, so a header that
claims the opposite of the code below it is worse than none. Fix the seven the
review flagged: channels.py (subscribe/unsubscribe shipped), playlists.py +
sync/playlists.py (YouTube two-way sync + write-back shipped), setup.py (drop the
"epic 6c" ref — the wizard steps exist), config.py ("auth._client_ip" ->
"routes.auth._client_ip"), db.py ("routes/plex.py" -> "routes/plex/" after the
package split), sysconfig.py (the "Phase C" note — the watch-sync job that reads
plex_watch_sync_interval_min has landed; clarified it's a boot default, not a knob).
Docs only.
Adds the storage + sync for the Hungarian-metadata overlay (the browsable EN sections gain HU
twin fields written from a paired HU section, matched by guid; no browsable HU rows).
Schema (2 migrations, ADD-only; run before the first HU sync while *_hu is NULL so the generated
column build is trivial):
- 0063: plex_libraries.metadata_lang ('en' default) + self-FK overlay_of.
- 0064: plex_items/plex_shows.guid (indexed) + title_hu/summary_hu/genres_hu (+ GIN) + a STORED
generated search_vector_hu mirroring the EN vector (title_hu A / people_text B / summary_hu C) + GIN.
Sync:
- includeGuids=1 on section_items so the canonical guid rides the cheap listing.
- _canonical_guid(meta) (tmdb > imdb > tvdb) captured onto every EN row in _apply_item/_sync_shows.
- Two-pass sync(): pass 1 mirrors browsable (non-overlay) sections; pass 2 reads plex_overlay_map
('huKey:enKey', new sysconfig), designates each HU section (metadata_lang='hu' + overlay_of), and
_sync_overlay writes title_hu/summary_hu/genres_hu onto the guid-matched EN rows.
Browse: _content_libs now excludes overlay libraries (overlay_of is None); the unified_library and
facets inline library partitions fold into it, and list_libraries/collections exclude overlays too.
Tests: DB-lane coverage of the overlay write (matched vs unmatched rows, no browsable HU rows, the
generated search_vector_hu populates), the guid parse, and the browse exclusion. Full suite 272
passed; migrations verified reversible (upgrade→downgrade→upgrade). S5b (persist the live detail
fields + serve item/show detail from DB) is the follow-up.
Defense-in-depth after the ReDoS incident. The heavy work (yt-dlp download / ffmpeg edit)
now runs in a child process (app.downloads.runner) supervised by the worker thread, not in
the thread itself. A thread can't be killed and a C-level spin can't even be cooperatively
cancelled, so one wedged download used to pin a core, hold the GIL and freeze the whole
queue; a child process can be SIGTERM/SIGKILLed.
The supervisor kills the child (and its process group, so ffmpeg grandchildren die too)
when: a route pauses/cancels the job, it makes no progress for download_stall_timeout_seconds
(150s), it runs past download_wall_clock_max_seconds (2h), the worker is shutting down, or
the child crashes without setting state. On a kill it frees the asset and fails the job with
a retryable network error (stalls are not auto-requeued). A clean exit is left alone — the
child already wrote the terminal state, including a cooperative pause via the progress hook.
- config.py: stall / wall-clock / supervisor-poll / kill-grace settings.
- worker.py: _run_supervised + _spawn_download_child / _job_liveness / _stop_child /
_cleanup_staging; _process_job claims the asset, then hands the heavy work to the child.
- app/downloads/runner.py: the child entrypoint.
- docker-compose.localdev.yml: init:true on the worker so tini (PID 1) reaps orphaned
ffmpeg grandchildren left by a hard-kill (the worker reaps its own direct children).
- tests: watchdog decision logic + SIGTERM->SIGKILL escalation (fake proc, no real subprocess).
_client_ip trusted the first X-Forwarded-For hop unconditionally, so anyone able to
reach the app port could forge XFF and dodge the login/register/reset/demo rate limits.
Now trust XFF ONLY when the request's socket peer is a configured reverse proxy
(settings.trusted_proxy_ips, e.g. the VPS Caddy's WireGuard peer IP), and take the
RIGHTMOST entry — the client our proxy actually saw and appended, immune to a client
pre-seeding a fake XFF. A request from any other peer (hitting the port directly) is
keyed on its real socket IP, so XFF can't be forged to bypass the limits.
New TRUSTED_PROXY_IPS env (empty default = no proxy, use the direct peer). Documented in
.env.example, docs/self-hosting.md, README. Unit-verified against spoof-through-proxy and
direct-bypass cases.
stream.py enforced a hardcoded _MAX_SESSIONS = 4 and never read the
DB-overridable plex_max_transcodes ConfigSpec, so the Configuration → Plex →
"Max concurrent transcodes" knob was dead (same env-vs-DB drift class as the
Downloads/Admin fixes). _enforce_cap(cap) now takes the cap from
sysconfig.get_int(db, "plex_max_transcodes") (>=1 so playback can't be capped to
zero). Bumped the config default 1→4 so the effective default matches the old
hardcoded cap (no behavior change for non-overriders; the knob now works).
Completes the two-way watch-state sync with two scheduler jobs:
- plex_watch_sync (default 30m): pull recent Plex-side changes (watch history + on-deck, filtered
to the owner account) into Siftlode under last-write-wins (_pull_apply + a _same_state ping-pong
guard + skew tolerance), then re-push any still-unsynced local states.
- plex_watch_reconcile (default daily): full section rescan; uses synced_to_plex to settle what the
incremental feed can't — notably propagating a Plex-side un-watch (clear a previously-mirrored row
Plex no longer has) — while re-pushing never-synced local states and never touching hidden
(Siftlode-only) rows. Union-preserving.
PlexClient gains accounts/watch_history/on_deck; _scan_plex_states is factored out and shared with
the one-time import. Owner accountID is resolved once and cached on the link. Both jobs are
registered in the scheduler (pause-skip, activity tracking, run-now, admin-tunable intervals) with
trilingual (HU/EN/DE) labels + descriptions. New config default plex_watch_reconcile_interval_min.
Verified live against the real Plex server: read feeds, last-write-wins, dirty re-push, the
incremental job end-to-end, and a full reconcile that cleared exactly the one un-watched item with
zero collateral across 17976 scanned.
Plex records watch state per Plex account; Siftlode per user in plex_states. A new plex_link table
maps a Siftlode user to a Plex account; for the owner (MVP) the row uses the server admin token
(uses_admin=True), so no separate Plex login. app/plex/watch_sync.py reads the owner account's
viewCount/viewOffset/lastViewedAt (already present in the catalog mirror's section listing) and
upserts them into the owner's plex_states — 'Plex is master' on this first import, but only where
Plex has a watch record (Siftlode-only states are preserved; union on the intersection). Idempotent.
Admin routes: GET/POST /api/plex/watch/link (status + enable/disable; first enable runs the import)
and POST /api/plex/watch/import (re-run). Migration 0051_plex_link. Two-way push (Phase B) and the
incremental Plex→Siftlode reconcile (Phase C) build on this in later ships.
Validated end-to-end on a real remux file (h264+ac3 mkv). The riskiest part of the
Plex epic — on-the-fly playback from the local file — now proven.
- app/plex/stream.py: seek-restart HLS session model (one ffmpeg per item, restarted
at a seek offset via -ss). Video stream-COPY (I/O-bound, CPU-light — fine on the
GPU-less prod host) + audio→aac only when not already aac. Maps first video+audio,
drops subtitles (no VTT rendition clutter). Session cap + idle reaper.
- routes: POST /stream/{rk}/session?start= (direct→raw url, remux→HLS session,
transcode→501 P3), GET /stream/{rk}/file (206 range, direct), /index.m3u8, /seg_n.ts.
- KEY FINDINGS (hard-won): (1) ffmpeg -hls_playlist_type VOD does NOT write the
playlist mid-run (only at finalize) → use EVENT (append-only, appears immediately),
which suits the seek-restart model; (2) naive per-segment copy + keyframe-indexed
extraction are both unreliable (non-uniform/imprecise segments) — ffmpeg's own HLS
muxer is the only correct segmenter; (3) dev slowness was the Windows /downloads
bind-mount write + SMB read → PLEX_HLS_DIR env points scratch at fast container-local
/var/tmp on dev (prod keeps download_root, fast local disk).
YouTube returns flaky per-client responses: the web clients intermittently mislabel normal
videos as 'DRM protected' (like tv did), and android_vr can get bot-flagged under heavy load.
Neither client alone is reliable. So:
- PRIMARY player client = android_vr: high-quality DASH (399+251), no PO token / Deno needed,
reliable for normal use
- on a bot/DRM/format failure the worker retries with the FALLBACK set (web_safari,web,android
+ POT sidecar + Deno) — bypasses bot-detection when android_vr is flagged
- both client sets admin-tunable (DOWNLOAD_PLAYER_CLIENTS[_FALLBACK])
Verified: the previously DRM-failing video now downloads steadily via android_vr, no DRM error.
The web player clients need two things beyond the PO token: (1) a JavaScript runtime for
YouTube's 'n' signature challenge — install Deno (v2.9.1) + yt-dlp[default] (bundles yt-dlp-ejs
solver scripts); auto-detected on PATH. (2) an audio source — the web clients only expose
video-only DASH, so add the 'android' client (also PO-token-backed) for audio/muxed formats.
Player clients: web_safari,web,android (dropped tv — it falsely reports normal videos as DRM).
End-to-end verified in the worker: extraction (POT + Deno n-challenge) + download completes,
no bot/DRM/format errors. Full YouTube stack now: force-ipv4 + bgutil POT sidecar + Deno.
YouTube increasingly demands a Proof-of-Origin token (or shows 'confirm you're not a bot').
Instead of manual cookies, run the bgutil POT provider as a sidecar that mints tokens on demand:
- compose: bgutil-pot service (brainicism/bgutil-ytdlp-pot-provider:1.3.1, port 4416, init)
- requirements: bgutil-ytdlp-pot-provider==1.3.1 plugin (version pinned to the image)
- config: DOWNLOAD_POT_BASE_URL (http://bgutil-pot:4416) + DOWNLOAD_PLAYER_CLIENTS
(web_safari,web,tv — POT-capable; the default android_vr client never requests a token)
- formats.build_ydl_opts wires extractor_args youtube:player_client + youtubepot-bgutilhttp:base_url
Verified: the sidecar mints a valid PO token (BotGuard challenge solved) and yt-dlp fetches it
from the provider. NOTE: end-to-end download couldn't be confirmed here because the test IP got
hard-flagged by YouTube from heavy testing (~16 GB + dozens of extractions) — a hard abuse block
needs cooldown/cookies, which POT doesn't lift; POT bypasses the normal soft bot-check.
- Remove owner-specific / legacy deploy files (home/prod/server compose, deploy/).
The home compose stays as a local untracked file for the maintainer's own deploy.
- Genericise infra-specific code comments (egress-proxy examples) to neutral wording.
- Replace the hardcoded contact email on the legal pages with the instance operator's
configured admin email, served via the public /auth/config and shown with a neutral
fallback — so each self-hosted instance shows its own contact.
- Rewrite README for the current app + a copy-paste self-hosting quick start (prebuilt
image + first-run wizard) with a build-from-source alternative; tidy .env.example.
- Live-search results stay ephemeral: the discovery-cleanup job now also reclaims un-kept
via_search videos (no watch state / not playlisted / channel not subscribed) after a grace
period (search_grace_days), and POST /api/search/clear discards a given result set 'as if
never added' (drops the user's search-finds + deletes the now-orphaned, un-kept videos).
Admin POST /api/admin/purge-discovery runs it on demand (grace 0).
- Count selector: GET /api/search/youtube gains a 'limit' — the free scrape source pages
through continuations until that many results are gathered (no manual load-more); the API
source stays one ≤50 page (cost).
- Per-user channel blocklist (migration 0034 blocked_channels): block/unblock/list endpoints;
blocked channels' videos are dropped from live search before ingest and hidden from the feed
/ Library / explore / channel page; explore refuses a blocked channel.
- New-first ordering: results you already have (subscribed channel, watched/in-progress/saved/
playlisted) sink below genuinely-new discoveries, preserving relevance within each group.
Adds the backend for a dedicated channel page + ephemeral browsing of un-subscribed
channels:
- migration 0033: channels.{total_view_count,published_at,banner_url,external_links,
from_explore}, videos.via_explore, explored_channels (per-user, grace-clocked).
- enrich channels with part=brandingSettings (banner/links) + statistics.viewCount +
snippet.publishedAt — no extra quota.
- GET /api/channels/{id}: About detail + this user's relationship; lazy-enriches the new
About fields (published_at sentinel).
- POST /api/channels/{id}/explore (require_human, quota-reserve guarded): records the
exploration, flags the channel ephemeral unless followed, ingests one page of uploads
(via_explore) + enriches; returns next_page_token for load-more.
- feed: per-explorer gate — a via_explore video is visible only to users who explored or
subscribe to its channel, so exploration never leaks into everyone's catalog.
- subscribe = keep: clears from_explore + via_explore on the channel's videos.
- scheduler explore_cleanup job + explore_grace_days config: hard-delete explored-but-
unkept channels and their untouched ephemeral videos after a grace period.
Live YouTube search can now use YouTube's internal InnerTube endpoint instead
of search.list, materialising results through the same enrich/provenance path
at zero API quota (search.list costs 100 units/page; scrape costs nothing, only
the cheap shared videos.list enrich is charged).
- youtube/search_scrape.py: InnerTube search returning the same page shape as
YouTubeClient.search_videos (items + continuation cursor); SOCS consent cookie,
videoRenderer walk (channels/playlists/Shorts shelves naturally excluded),
type:video filter, cached InnerTube key/version with constant fallbacks.
- routes/search.py: admin-selectable source (search_source, default scrape).
Scrape path skips the 100-unit budget pre-check and logs a zero-cost search
event so the per-user daily cap still counts it; api path unchanged. Response
carries the active source.
- sysconfig/config: new search_source key (scrape|api).
- quota.log_action: record a zero-cost action event for per-user rate limits.
Add a live YouTube search that materialises results into the shared catalog so
they render with the normal feed cards + in-app player and gain per-user state.
- YouTubeClient.search_videos(): search.list (100 units), embeddable-only, returns
flat stubs + nextPageToken; surfaces liveBroadcastContent for live filtering.
- routes/search.py GET /api/search/youtube: require_human + per-user daily cap
(search_daily_limit_per_user, default 70) + can_spend pre-check (429 on either);
drops live/upcoming, upserts channel stubs (channels.list) + video stubs, enriches
(videos.list), runs the youtube.com/shorts probe, then excludes Shorts/live and
returns feed cards in relevance order with the YouTube pageToken as the cursor.
- Provenance: videos.via_search / channels.from_search (migration 0028) flag
search-discovered rows; the feed hides them from the Library (scope=all) by default
via exclude_search_discovered, leaving the Mine feed untouched.
- quota.actions_today() counts a user's per-action events today for the cap; only the
search.list call is attributed VIDEOS_SEARCH so the counter is exactly 1 per search.
New youtube_api_proxy setting (env fallback + admin Config UI, env YOUTUBE_API_PROXY):
when set, the YouTubeClient routes all its httpx traffic through that HTTP(S) proxy.
Lets a dynamic-IP host send API calls through a fixed-IP host (e.g. the server over
private tunnel) so an IP-restricted API key keeps working. Empty = direct.
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
- Link a Google account to a password account, and adopt the Google identity onto a matching
email account instead of 500ing on a duplicate; set or change a password from Settings.
- Expose has_google/has_password on /api/me for the Sign-in methods UI.
- Mark Google logins email-verified (backfill existing rows, migration 0024); stop a routine
login from clobbering an admin-assigned role (env ADMIN_EMAILS stays the bootstrap admin).
- Suspension login-gates (password + Google callback + current_user) with a rate-limited
'suspended' notice; shared purge_user (cascade delete + access-request cleanup + Google-grant
revoke) behind self- and admin-deletion; single app_base source for user-facing email links.
A security audit of every mutating endpoint found no isolation gaps — all routes
scope by current_user, admin routes are gated, and the demo account is sandboxed
in its own user_id space (can't reach others' data, escalate, or hit admin
routes). The remaining demo concern is communal pollution of its own shared state,
so add an automatic reset: a new 'demo_reset' scheduler job (admin-tunable
interval, default 12h) reuses the manual reset logic, and no-ops without ever
creating a demo account if none exists. Verified: a triggered run wiped the demo's
video states and re-seeded its sample playlists.
Add email+password auth alongside Google: argon2id hashing; users gains
password_hash/email_verified/is_active and google_sub becomes nullable (migration
0022); a single-use, hashed auth_tokens table for email verification + password
reset. Registration creates a pending (inactive, unverified) account + an access
request; sign-in needs verified email AND admin approval. Anti-enumeration: uniform
register/login/reset responses (a correct-password owner still gets a specific
pending reason). allow_registration flag (DB-tunable). Admin users-list + role
endpoint (guards: not self, not demo, not the last admin); approving access (or a
manual whitelist add) now activates the matching pending account. current_user
rejects deactivated accounts. Verified end-to-end via curl + DB.
Add a daily maintenance/validation job that detects videos which can't be played
anywhere and retires them safely. Two phases: re-check already-flagged videos
(recover if available again, else hard-delete once the grace period elapses,
cascading to states/playlist items), and a rolling re-validation of the
least-recently-checked currently-available videos that flags newly-unplayable
ones (hidden from the feed immediately via unavailable_since).
Detection is ~free: a video missing from the videos.list response is
deleted-or-private; an `upcoming` premiere >2 days past its scheduled start that
never went live is abandoned. A still-live broadcast is kept (legit 24/7 stream).
Enrichment now also fetches part=status to populate the status columns. Grace is
7 days for removed videos, none for abandoned. Before deleting, affected users
get one batched notification (never per-video). Interval is admin-tunable via the
Scheduler dashboard; batch size and grace are config. Quota-attributed to the
system and bounded by the same backfill reserve as the other jobs.
Mirror each user's own YouTube playlists into local source='youtube' playlists,
one-way (YT -> local). New client methods iter_my_playlists / iter_my_playlist_video_ids
(OAuth, so private playlists work); sync/playlists.py reconciles the mirror (matching YT
order) and ingests any playlist videos not in the shared catalog yet (with stub channels).
POST /api/playlists/sync-youtube for manual sync (read-scope gated, per-user quota) plus a
scheduler job (playlist_sync_minutes, default 6h) that syncs all read-scope users. YouTube's
Watch Later / History are not API-accessible and are never synced.
Make APP_VERSION come from the VERSION file (Dockerfile reads it for the SPA build
and ships it for the backend to read at runtime) instead of depending solely on a
deploy-time build-arg. This keeps the version correct even for a plain
`docker compose build` and removes the first-deploy bootstrap gap. git_sha/build_date
stay best-effort build-args.
Add a VERSION file (0.1.0) and inject APP_VERSION/GIT_SHA/BUILD_DATE as Docker
build-args (both stages; Vite inlines them into the SPA). New public GET
/api/version returns app_version, git_sha, build_date and the Alembic head as the
database revision. deploy.sh and the localdev build pass the args.
Base login now requests only openid/email/profile (non-sensitive), so a new user
gets a clean Google consent with no "unverified app" warning and no 7-day refresh
token expiry. YouTube read (youtube.readonly) and write (youtube) are granted later
by the onboarding wizard via a parameterized /auth/upgrade?access=read|write.
Security fixes folded in from the baseline audit:
- config: refuse to boot in production (https OAUTH_REDIRECT_URL) with the
placeholder/short SECRET_KEY or a missing TOKEN_ENCRYPTION_KEY, closing a
session-forgery / admin-impersonation hole.
- main: mark the session cookie Secure when served over HTTPS.
- me: expose can_read; sync/subscriptions returns a friendly 403 (not a 500)
until YouTube read access is granted.
Move the access whitelist from the ALLOWED_EMAILS env var into a DB Invite table
(env kept as bootstrap fallback), and add a self-service request + admin approval
flow with fail-soft email.
- models: Invite(email, status pending|approved|denied, requested_at, decided_*)
- migration 0008: invites table; seed env ALLOWED_EMAILS u ADMIN_EMAILS as approved
- auth: is_allowed() (DB-first, env fallback); a denied Google login records a pending
request and bounces to /?access=requested instead of a raw 403; public POST
/auth/request-access; upsert is idempotent so repeats don't re-spam admins
- routes/admin.py (admin-only): list/approve/deny invites + manual add
- email.py: smtplib + Gmail App Password, fail-soft (skips if SMTP unset)
- /api/me exposes pending_invites; config + .env.example gain SMTP_*
- UI: Login 'Request access' form + access=requested/denied handling; Settings ->
Access requests (approve/deny + add); admin nudge toast on pending requests
Verified locally: request-access creates a pending invite and emails the admin;
seed approved npeter83; guinea-pig yt.trash2023 denied until approved.
- Language detection: classify one cleaned+concatenated blob (strip emoji,
@mentions, #tags, numbers, punctuation); fixes caps/emoji-heavy channels
(e.g. Nessaj -> Hungarian, no more bogus Chinese/Korean)
- Feed now joins the user's subscriptions, so unsubscribing on YouTube removes a
channel from the feed; periodic subscription re-sync job picks up changes
- Watched/Saved/Hidden views ignore the Shorts/live default-hiding so the full
set is visible (fixes hidden videos missing from the Hidden view)
- Persist feed filters + search across reloads (localStorage)
- 3D polish: cards lift with shadow on hover; chips/buttons get depth and a
press effect; undo toast lasts longer
- Shorts: confirm via youtube.com/shorts/<id> probe (SOCS cookie bypasses the
consent redirect) instead of a 60s heuristic; concurrent probing, shorts_probed
flag, scheduled refinement (migration 0005)
- Search: match title + channel name only (descriptions caused noisy results)
- Faceted tag filtering: AND across categories (language AND topic narrows),
OR within a category; any/all toggle applies to topics
- Language detection: majority vote over individual titles (fixes misdetections
like multipoleguy -> English; drops bogus Polish/Romanian)
- Login: drop forced consent so returning sign-in is quick (select_account)
- Feed cards: clickable channel name (opens channel), persistent saved badge,
undo toast on hide, Hidden view to restore; tag chips show counts in tooltip
- Tag and ChannelTag models + migration 0003 (partial unique indexes split
system vs per-user tag names)
- Offline language detection (py3langid) constrained to a curated language set,
with the channel's declared default language as a strong prior
- Topic tags mapped from YouTube topicDetails + dominant video category; the
generic "Lifestyle" catch-all is intentionally dropped
- System (auto) tags are regenerated idempotently and never touch user tags;
orphaned system tags are cleaned up
- GET /api/tags and admin POST /api/tags/recompute; scheduled autotag job
- Free per-channel RSS reader for quota-less fresh-video detection
- Recent-first backfill (configurable: 100 videos / 1 year) plus resumable deep
backfill from the uploads playlist
- Enrichment via videos.list: duration, view/like counts, category, topics,
language, Shorts heuristic and livestream/premiere classification
- Reusable sync runners + APScheduler jobs (rss / enrich / backfill), all
quota-aware with a reserve so backfill never starves fresh enrichment
- Manual triggers: POST /api/sync/{rss,backfill,enrich}
- Exact insert counting via RETURNING with in-batch de-duplication
- Channel/Subscription/Video/ApiQuotaUsage models + migration 0002
- Synchronous YouTube Data API client with OAuth token refresh and per-call
quota accounting (API key preferred for public reads when configured)
- Subscription import: upsert channels + subscription links, prune unsubscribed,
fetch channel details (uploads playlist, topics, stats, handle, country)
- Central quota guard tracking units per US-Pacific day against a daily budget
- POST /api/sync/subscriptions and GET /api/sync/status endpoints
- docker-compose with Postgres 16 + slim Python API image
- FastAPI app with session middleware, health endpoint, static login page
- Google OAuth (Authlib) with email invite-list whitelist; admin role support
- User + OAuthToken models; refresh tokens encrypted at rest (Fernet)
- Alembic migrations, run automatically on container startup
- Postgres backup/restore scripts for portability between machines
peter