The log now has a stored, queryable copy for the diagnostics console:
every WARNING and above plus the request line of every request, kept
for log_retention_days (default 7, no "forever") and pruned hourly by
the new log_gc job.
Rows are built on the logging thread from the same redacted fields as
the stdout line, queued without blocking and written in batches by one
background thread on its own connection. A full queue or a failed write
drops and reports instead of retrying. Flood control keeps 30 of any
one message per minute plus a count of the rest; anonymous request
lines are capped at 60 a minute per client, and the client address is
never stored. Deleting an account deletes its stored events.
Measured on prod first: a YouTube RSS outage wrote 11k warnings in four
hours, one per channel per poll. The poll now writes one warning with
per-reason counts and a few examples, and a failure on our side gets
one traceback per poll instead of one per channel.
A rotated or missing TOKEN_ENCRYPTION_KEY makes every stored secret undecryptable.
sysconfig._raw_override called security.decrypt() with no guard, and it sits on
hot paths (/api/me via google_enabled, login, every YouTube scheduler job,
account purge) — so a key mismatch raised InvalidToken and 500'd the whole app,
the SPA wouldn't even boot, with no documented recovery.
Catch the decrypt failure in _raw_override -> warn once per key and fall back to
the env/config default. The YouTube client's refresh-token decrypt now raises a
clear YouTubeError ("reconnect") that every caller already handles, and
purge_user uses decrypt_optional so a best-effort revoke can't 500 the delete.
Document that the key must never change once secrets are stored (recovery: clear
the affected system_config rows and re-enter).
Transaction ownership (C-A5):
- quota.record_usage / log_action write in their OWN SessionLocal(), committed
immediately, and no longer take/commit the caller's db. The spend stays DURABLE
regardless of the caller (read-only YouTube paths never commit; a failed job rolls
back) — under-counting the shared daily budget would risk overspending the real
YouTube quota — AND quota stops flushing a job's partial state mid-run behind its
own `except: db.rollback()`. units_used_today switches to a scalar SELECT so
measured()'s before/after diff sees the separate session's commits (READ COMMITTED).
Call sites updated (youtube/client.py ×3, routes/search.py ×1).
- plex sync commits PER SECTION (moved the end-of-loop commit inside), so a PlexError
on a later library keeps the ones already mirrored (idempotent; next run reconciles).
The sync jobs did NOT rely on record_usage's incidental commit for persistence — their
per-item functions (apply_rss_feed, backfill_channel_recent, import_subscriptions,
sync_user_playlists, apply_channel_autotags) all commit internally, so the in-loop
rollback handlers already isolate a failed item. No loop restructuring needed.
DB-lane tests (test_quota_ownership, 4): spend survives a caller rollback while the
caller's uncommitted row does NOT leak; accumulation; measured-style mid-txn read.
Gate: ruff clean; DB lane 180 green.
- never prune subscriptions on an empty fetch: one anomalous empty-but-200
response wiped every subscription (and its deep_requested flag) in one run
- push_playlist: stop after 5 consecutive write failures — a quota-dead or
scope-revoked account otherwise burns 50 units per remaining item
- RSS: raise RssError instead of returning [] on a transport/HTTP failure, so a
failed poll no longer stamps last_rss_at and a 404 feed stops looking healthy
- worker: exit when the schema never appears instead of "continuing" into a
crash one call later
From the auth /security-review (contained fixes; architectural SA3 proxy-trust +
SA4 session-revocation deferred to the user):
- SB1: encrypt the OAuth access_token at rest (was plaintext while refresh_token
was encrypted) — a ~1h Google bearer credential. New security.decrypt_optional()
falls back to a refresh for legacy plaintext tokens; column is unbounded String.
E2E-verified: token refreshed → stored as Fernet ciphertext → 319-subscription
YouTube sync succeeded.
- SA5: password_login is no longer a timing/enumeration oracle — it always runs
argon2 (against a decoy hash for unknown/passwordless emails), so response time
can't reveal whether an account exists.
- SB2: Google login only ADOPTS+activates a pre-existing password account when
Google actually attests email_verified (defense-in-depth against takeover); and
the email sync won't overwrite with a value another account owns (avoids a 500).
- demo_login clears the wallet first (demo can't switch to / act as a real account
via the multi-account header); switch_account rejects a suspended target (which
would otherwise clear the whole session on the next request).
Re-review clean; ruff clean; localdev boots; YouTube auth path E2E-verified.
- feed.py _filtered_query: drop the dead 2nd return element (status_expr was
used only internally for WHERE filters; all 3 callers discarded it as _status).
Now returns (query, rank_expr); fixed the stale docstring.
- youtube/client.py: extract _iter_playlist_items() — iter_my_playlist_video_ids
and iter_playlist_items_with_ids were near-identical playlistItems paging loops
(jscpd [173-187]≈[267-281]); both now map over the shared generator.
- sync/videos.py: extract _apply_video_batch() with an on_missing callback —
enrich_pending and refresh_live shared the fetch-map-apply skeleton, differing
only in the query and how they retire rows YouTube omits.
- models.py: add LIVE_OR_UPCOMING = ("live","upcoming"); replace the 4 duplicated
copies (feed.HIDDEN_LIVE, search._LIVE_HIDDEN, videos.py + channels.py inline).
Behavior-neutral. ruff clean on touched files, localdev boots, feed/worker healthy.
Machine-baseline harvest (ruff + knip), all tsc/parse-green, no runtime change:
- backend: remove 3 unused imports (channels/playlists/youtube) via ruff; drop the
unused `job` binding in unshare_download (keep the _own_job ownership guard call).
- frontend: remove `export` from 23 internally-used-only symbols (knip "unused
exports") to shrink the public surface; delete 2 genuinely-dead declarations
(PlexBrowseResult — leftover from the removed /browse route; WIDGET_TITLES —
hardcoded English titles superseded by i18n).
Held back for a decision (unused here = possibly-unwired, NOT dead — flagged, not
removed): e2ee.lock()/clearDevice() (security primitives never wired to logout / a
"forget device" feature) and loadDefaultViewFilters (App reimplements it inline — a
DRY issue). See siftlode-ops/CODE-HYGIENE.md.
The YouTubeClient issued httpx requests directly, so a network/transport
failure (egress proxy unreachable, DNS, timeout, connection reset) escaped
as a raw httpx.ConnectError. Callers only guard against YouTubeError, so such
a fault propagated uncaught and surfaced as a 500 — e.g. opening an
un-enriched channel's page (GET /api/channels/{id} lazily enriches About
data) popped a blocking "Server error (500)" modal whenever the fixed-IP
egress proxy was down.
Route all client HTTP through a _send() helper that wraps httpx.HTTPError in
YouTubeError, so every existing 'except YouTubeError' degrades gracefully:
channel detail returns un-enriched (200), explore returns 422 (quiet), and
scheduler jobs log-and-continue instead of crashing.
- Remove owner-specific / legacy deploy files (home/prod/server compose, deploy/).
The home compose stays as a local untracked file for the maintainer's own deploy.
- Genericise infra-specific code comments (egress-proxy examples) to neutral wording.
- Replace the hardcoded contact email on the legal pages with the instance operator's
configured admin email, served via the public /auth/config and shown with a neutral
fallback — so each self-hosted instance shows its own contact.
- Rewrite README for the current app + a copy-paste self-hosting quick start (prebuilt
image + first-run wizard) with a build-from-source alternative; tidy .env.example.
Adds the backend for a dedicated channel page + ephemeral browsing of un-subscribed
channels:
- migration 0033: channels.{total_view_count,published_at,banner_url,external_links,
from_explore}, videos.via_explore, explored_channels (per-user, grace-clocked).
- enrich channels with part=brandingSettings (banner/links) + statistics.viewCount +
snippet.publishedAt — no extra quota.
- GET /api/channels/{id}: About detail + this user's relationship; lazy-enriches the new
About fields (published_at sentinel).
- POST /api/channels/{id}/explore (require_human, quota-reserve guarded): records the
exploration, flags the channel ephemeral unless followed, ingests one page of uploads
(via_explore) + enriches; returns next_page_token for load-more.
- feed: per-explorer gate — a via_explore video is visible only to users who explored or
subscribe to its channel, so exploration never leaks into everyone's catalog.
- subscribe = keep: clears from_explore + via_explore on the channel's videos.
- scheduler explore_cleanup job + explore_grace_days config: hard-delete explored-but-
unkept channels and their untouched ephemeral videos after a grace period.
Live YouTube search can now use YouTube's internal InnerTube endpoint instead
of search.list, materialising results through the same enrich/provenance path
at zero API quota (search.list costs 100 units/page; scrape costs nothing, only
the cheap shared videos.list enrich is charged).
- youtube/search_scrape.py: InnerTube search returning the same page shape as
YouTubeClient.search_videos (items + continuation cursor); SOCS consent cookie,
videoRenderer walk (channels/playlists/Shorts shelves naturally excluded),
type:video filter, cached InnerTube key/version with constant fallbacks.
- routes/search.py: admin-selectable source (search_source, default scrape).
Scrape path skips the 100-unit budget pre-check and logs a zero-cost search
event so the per-user daily cap still counts it; api path unchanged. Response
carries the active source.
- sysconfig/config: new search_source key (scrape|api).
- quota.log_action: record a zero-cost action event for per-user rate limits.
Add a live YouTube search that materialises results into the shared catalog so
they render with the normal feed cards + in-app player and gain per-user state.
- YouTubeClient.search_videos(): search.list (100 units), embeddable-only, returns
flat stubs + nextPageToken; surfaces liveBroadcastContent for live filtering.
- routes/search.py GET /api/search/youtube: require_human + per-user daily cap
(search_daily_limit_per_user, default 70) + can_spend pre-check (429 on either);
drops live/upcoming, upserts channel stubs (channels.list) + video stubs, enriches
(videos.list), runs the youtube.com/shorts probe, then excludes Shorts/live and
returns feed cards in relevance order with the YouTube pageToken as the cursor.
- Provenance: videos.via_search / channels.from_search (migration 0028) flag
search-discovered rows; the feed hides them from the Library (scope=all) by default
via exclude_search_discovered, leaving the Mine feed untouched.
- quota.actions_today() counts a user's per-action events today for the cap; only the
search.list call is attributed VIDEOS_SEARCH so the counter is exactly 1 per search.
New youtube_api_proxy setting (env fallback + admin Config UI, env YOUTUBE_API_PROXY):
when set, the YouTubeClient routes all its httpx traffic through that HTTP(S) proxy.
Lets a dynamic-IP host send API calls through a fixed-IP host (e.g. the server over
private tunnel) so an IP-restricted API key keeps working. Empty = direct.
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
- Build the Google OAuth client lazily from sysconfig (DB override, env fallback) and re-register
when the credentials change, so the install wizard / admin can set them without a restart.
- New 'google' config group (google_client_id/secret, encrypted) on the Configuration page; the
token-refresh reads the creds the same way.
- Public GET /auth/config exposes google_enabled; the login page hides 'Continue with Google'
when Google OAuth isn't configured (email+password still works).
Register youtube_api_key in the sysconfig registry as a secret and resolve it from
the DB-override-or-env layer in YouTubeClient (self.db is in scope). Adds a YouTube
API group to the Configuration page (EN/HU/DE).
Google OAuth client id/secret stay env-only for now: the authlib client is
registered at module load (auth.py), and the token-refresh reads them too, so a
runtime DB override needs a lazy/re-registerable OAuth client — that belongs to the
install-wizard epic (first-boot OAuth config without restart). Admin-role-via-UI
moves to the auth-overhaul epic (user management lives there).
Add a "Discover from playlists" tab to the Channel manager that lists
channels appearing in the user's playlists they don't subscribe to, with
a one-click Subscribe.
- GET /api/channels/discovery: local join (playlist_items -> videos ->
channels) minus the user's subscriptions and their own channel. Enriches
stub channels' metadata up front (title/thumbnail/subscriber count via the
API key) so the user can judge a channel before subscribing; videos are
not pulled (the scheduler picks those up).
- POST /api/channels/{id}/subscribe: write-scope gated, subscriptions.insert
+ local Subscription with the returned resource id.
- YouTubeClient.insert_subscription / get_my_channel_id.
- users.yt_channel_id (migration 0019) caches the user's own channel id so
discovery can exclude it.
- Frontend: ChannelDiscovery DataTable, Channels tab toggle (persisted),
api methods, trilingual strings. Subscribe ships a typed notification
payload (ChannelSubscribedMeta) for the inbox to act on.
Add a daily maintenance/validation job that detects videos which can't be played
anywhere and retires them safely. Two phases: re-check already-flagged videos
(recover if available again, else hard-delete once the grace period elapses,
cascading to states/playlist items), and a rolling re-validation of the
least-recently-checked currently-available videos that flags newly-unplayable
ones (hidden from the feed immediately via unavailable_since).
Detection is ~free: a video missing from the videos.list response is
deleted-or-private; an `upcoming` premiere >2 days past its scheduled start that
never went live is abandoned. A still-live broadcast is kept (legit 24/7 stream).
Enrichment now also fetches part=status to populate the status columns. Grace is
7 days for removed videos, none for abandoned. Before deleting, affected users
get one batched notification (never per-video). Interval is admin-tunable via the
Scheduler dashboard; batch size and grace are config. Quota-attributed to the
system and bounded by the same backfill reserve as the other jobs.
Mirrors (source='youtube') were read-only; per the original two-way design they can
now be edited locally (add/remove/reorder/rename) and pushed back. The read-sync
skips a dirty mirror so it won't clobber unpushed edits; a successful push clears
dirty and lets the mirror refresh. _mark_dirty now covers any YouTube-linked list
(exported local or mirror), and rename marks dirty too. push/delete no longer reject
mirrors. Push reconciles the title as well (cheap playlists.list compare, then
playlists.update only if changed) so a local rename doesn't revert on the next pull.
Frontend: editable = not-built-in (was also excluding mirrors); rows/reorder enabled
for all owned lists; AddToPlaylist popover lists mirrors too (with a YT marker); the
origin chip now reads 'edits sync back'. Trilingual.
Add OAuth-only write helpers to YouTubeClient (each 50 quota units):
create_playlist, add_playlist_item, move_playlist_item, delete_playlist_item,
delete_playlist, plus iter_playlist_items_with_ids for diffing. A shared _write
helper records quota and raises YouTubeError on non-2xx.
Mirror each user's own YouTube playlists into local source='youtube' playlists,
one-way (YT -> local). New client methods iter_my_playlists / iter_my_playlist_video_ids
(OAuth, so private playlists work); sync/playlists.py reconciles the mirror (matching YT
order) and ingests any playlist videos not in the shared catalog yet (with stub channels).
POST /api/playlists/sync-youtube for manual sync (read-scope gated, per-user quota) plus a
scheduler job (playlist_sync_minutes, default 6h) that syncs all read-scope users. YouTube's
Watch Later / History are not API-accessible and are never synced.
- uvicorn --log-config (log_config.json): timestamped formatters for uvicorn
access/error and the app's own "subfeed" loggers (ms precision)
- Log key activity: startup/shutdown, login/denied, token refresh, YouTube API
errors, subscription imports, sync pause/resume
- Background sync jobs no longer swallow errors silently — failures in RSS poll,
backfill, enrichment, autotag and resync are logged with tracebacks
- Shorts: confirm via youtube.com/shorts/<id> probe (SOCS cookie bypasses the
consent redirect) instead of a 60s heuristic; concurrent probing, shorts_probed
flag, scheduled refinement (migration 0005)
- Search: match title + channel name only (descriptions caused noisy results)
- Faceted tag filtering: AND across categories (language AND topic narrows),
OR within a category; any/all toggle applies to topics
- Language detection: majority vote over individual titles (fixes misdetections
like multipoleguy -> English; drops bogus Polish/Romanian)
- Login: drop forced consent so returning sign-in is quick (select_account)
- Feed cards: clickable channel name (opens channel), persistent saved badge,
undo toast on hide, Hidden view to restore; tag chips show counts in tooltip
- Free per-channel RSS reader for quota-less fresh-video detection
- Recent-first backfill (configurable: 100 videos / 1 year) plus resumable deep
backfill from the uploads playlist
- Enrichment via videos.list: duration, view/like counts, category, topics,
language, Shorts heuristic and livestream/premiere classification
- Reusable sync runners + APScheduler jobs (rss / enrich / backfill), all
quota-aware with a reserve so backfill never starves fresh enrichment
- Manual triggers: POST /api/sync/{rss,backfill,enrich}
- Exact insert counting via RETURNING with in-batch de-duplication
- Channel/Subscription/Video/ApiQuotaUsage models + migration 0002
- Synchronous YouTube Data API client with OAuth token refresh and per-call
quota accounting (API key preferred for public reads when configured)
- Subscription import: upsert channels + subscription links, prune unsubscribed,
fetch channel details (uploads playlist, topics, stats, handle, country)
- Central quota guard tracking units per US-Pacific day against a daily budget
- POST /api/sync/subscriptions and GET /api/sync/status endpoints
peter