The ring-based flash relied on ring-accent and a CSS transition that never
restarted on a repeat message, so it was effectively invisible. Replace it with
a keyed inset-glow overlay (animation replays on every flash counter bump, and
inset shadow isn't clipped by the window's overflow-hidden) — flashes whether
the window is expanded, freshly auto-opened, or minimised.
The Messages UI trusted only the per-device IndexedDB key, so if the server's
key record was gone (deleted, DB-restored, admin-reset) while a stale private
key lingered in the browser, the app looked 'unlocked' but no one could be
messaged and no setup was offered. Add useKeyState (server 'configured' AND
local unlock): show setup when the server has no key (setup overwrites the stale
local key), unlock when it has one this device hasn't opened, ready otherwise.
ChatThread is now self-contained. Also fix the header title on the Messages and
Notifications pages (was falling through to 'Channel manager').
The live-message push now carries both parties, so an incoming message opens
that conversation's dock window if it's closed, or flashes it once if it's
already open (expanded or minimised, without disturbing the minimised state) —
only for messages from someone else, never your own echo.
Dock state (open windows + minimised state) is persisted per user, so a reload
restores exactly what was open, minimised, or closed.
Move messaging out of the notification center into its own left-nav module
(own page, own unread badge), so a reload returns to it and notifications stay
separate. Add a Messenger-style floating dock (bottom-right): pop a conversation
out from the page, keep chatting across navigation, minimise (rollup) or close.
- Messages is now a routed page; NotificationsPanel reverts to inbox-only and the
nav badge no longer mixes in messages.
- Extract shared KeyGate + ChatThread (used by both the page and dock windows);
e2ee exposes a shared unlock subscription so page and dock agree.
- ChatDock (always mounted for human users) owns the app-wide live-message
subscription and per-device key restore. EN/HU/DE strings.
A Messages tab in the notification center (hidden for demo): set up secure
messaging with a passphrase (key generated + wrapped in-browser via WebCrypto,
stored non-extractable per device), unlock on other devices, then chat with
end-to-end encrypted, live-delivered messages. The server-authored Siftlode
welcome is readable before any key setup.
- lib/e2ee.ts: ECDH P-256 + HKDF + AES-GCM, PBKDF2-wrapped key, IndexedDB.
- lib/messagesSocket.ts: WebSocket client with backoff reconnect.
- Messages.tsx: key gate, conversation list with decrypted previews, directory,
thread with client-side decrypt + encrypt-on-send. Unread feeds the nav badge.
EN/HU/DE strings.
Private user-to-user messages are end-to-end encrypted: the server only ever
stores ciphertext + iv and acts as a key directory (public keys) plus an opaque
store for each user's private key, wrapped client-side with a passphrase the
server never sees — so not even an admin can read a conversation. A separate
kind=system message (plaintext, no sender) powers a server-authored Siftlode
welcome shown on first open, reusable later for announcements.
- models: rework Message (kind, nullable sender/body, ciphertext+iv) + MessageKey;
migrations 0026 (table) + 0027 (E2EE rework).
- routes/messages.py: key directory/blob endpoints, ciphertext send, conversations
+ threads (system + user), lazy welcome, all gated by require_human.
- realtime.py: in-process WebSocket connection registry; /ws delivers sent
messages to a user's open tabs instantly (sync-callable push, single-process).
Render only the visible rows via @tanstack/react-virtual (useVirtualizer
against the app's <main> scroll container, per-row dynamic measurement),
so the DOM no longer accumulates every loaded card. A ResizeObserver keeps
the responsive grid column count in sync and chunks cards into virtualized
rows; the list view virtualizes single-card rows. Infinite scroll now
triggers from the virtual range and pages via next_cursor instead of
offset; feed/count drops its unused paging args.
Replace deep-OFFSET paging (which scans-and-discards skipped rows on the
233k-row catalog and shifts/duplicates rows as the scheduler ingests mid-
scroll) with stable keyset pagination over an opaque cursor.
A single sort registry drives both the ORDER BY and a generic, NULL-aware
keyset WHERE so they can never drift, with Video.id as the unique final
tiebreaker. Covers every sort: newest/oldest, views, duration, title,
subscribers (joined), priority (coalesced multi-key) and the seeded
shuffle. The response now returns next_cursor instead of offset.
Add docker-compose.home.yml: the app + Postgres run on strong, always-on home
hardware (nested Docker, apparmor:unconfined, Postgres tuned to keep the catalog
hot in cache), and the server reverse-proxies the hostname to it over a private
network link. This moves the heavy feed/count/facet scans off the server while
keeping it as the public TLS front door (the OAuth redirect URL is unchanged).
Retire the the server Forgejo Actions autodeploy (.forgejo/workflows/deploy.yml): the app
no longer runs on the server, so deploys go to the server from the dispatcher instead.
Replace the leftover 'subfeed' name across logger names + log_config,
frontend localStorage keys, Postgres user/db/volume defaults in the
compose files, .env.example, config.py, backup/restore scripts and the
README. Pure rename; no behavioural change. localStorage keys move from
subfeed.* to siftlode.* (one-time UI-state reset is acceptable).
It duplicated the developer-machine quality gate (tsc + npm build + compileall),
ran post-merge on main (not a pre-merge gate), and would put the heavy frontend
build on the server on every push — the very load deploy.yml deliberately
keeps on the dev box. The local pre-ship checks are the CI in this setup.
On a push to main the server-hosted runner SSHes back to the host (forced-command
key, docker bridge gateway, port 2222) and runs deploy/deploy.sh — replacing the
2-minute systemd poller. Build/publish still happens on the dev machine before push.
- /api/me exposes instance-wide google_enabled. The onboarding nudge no longer auto-opens, and
Settings hides the YouTube-access section + the 'Connect Google' option, when the instance has
no Google OAuth configured (e.g. an email+password-only self-host). A linked account still shows
its 'Connected' status.
- docker-compose.selfhost.yml pulls the published image (forge.b1fr0st.eu/peter/siftlode) with a
bundled Postgres — no source build on the operator's host.
- install.sh / install.ps1 generate the four required secrets (.env: POSTGRES_PASSWORD, SECRET_KEY,
a valid Fernet TOKEN_ENCRYPTION_KEY, OAUTH_REDIRECT_URL), start the stack, and print the first-run
setup-wizard URL. Everything else is configured in the web wizard.
- docs/self-hosting.md: the copy-paste playbook (get the files, run the installer, finish in the
wizard, optional Google/SMTP, HTTPS/reverse-proxy, updates + backups).
- Verified end-to-end: a fresh install from the published image boots into setup mode and serves
the wizard.
- Multi-step first-run wizard (intro → admin → Google → SMTP → finish), shown by App while the
instance reports configured=false; the one-time token comes from the setup URL (?token=) and is
sent as X-Setup-Token on every step. Google/SMTP steps appear only when secrets can be stored.
- App holds the 'me' query until setup status is known (so it doesn't 503 against the setup lock);
a missing/invalid token shows a 'use the link from the logs' screen. EN/HU/DE.
- POST /api/setup/admin (create the first admin: active+verified+admin), /config (persist any
registry-backed settings — Google creds, SMTP, quota…), /test-email, and /finish (mark
configured + invalidate the token). All behind require_setup (valid token AND not configured).
- GET /api/setup/info reports secrets_manageable so the wizard hides the Google/SMTP secret steps
on a box without TOKEN_ENCRYPTION_KEY.
- app_state gains configured + setup_token_hash (migration 0025); existing installs (any users)
are marked configured so they never see the wizard, a fresh DB starts in setup mode.
- On first boot the app mints a one-time setup token and logs the wizard URL (only the hash is
stored); the token gates the setup steps (added in 6c).
- A middleware locks all /api + /auth routes (except /api/setup, /api/version, /healthz, static)
to 503 until configured, with a DB-free cached fast-path once setup completes.
- GET /api/setup/status tells the SPA whether to show the wizard.
- Build the Google OAuth client lazily from sysconfig (DB override, env fallback) and re-register
when the credentials change, so the install wizard / admin can set them without a restart.
- New 'google' config group (google_client_id/secret, encrypted) on the Configuration page; the
token-refresh reads the creds the same way.
- Public GET /auth/config exposes google_enabled; the login page hides 'Continue with Google'
when Google OAuth isn't configured (email+password still works).
- Serve real files at the SPA root (e.g. /welcome/*.png, favicon) from the built app — the
landing screenshots were only ever shown by the Vite dev server, never in production.
- Add the Channel-manager screenshot; the two secondary previews are smaller thumbnails that
open in a custom full-size lightbox (fit-to-viewport, Esc/backdrop/✕ to close).
- Drop a suspended/deleted user to the login page on the next 401 (and poll the session only
while signed in, so the public login page no longer flickers); 'suspended' and 'account
deleted' confirmation banners; strip redirect query params for a clean address bar.
- New Users page tabs (Users & roles / Access requests / Demo) and a tab-ified Configuration
page, both via a reusable Tabs component (persisted active tab).
- Admin can suspend/unsuspend (migration 0023 adds users.is_suspended) and delete accounts
from the Users & roles tab, with guards (demo / self / last admin).
- Email notifications to the affected user: suspended (reactive, on a blocked sign-in),
reinstated, role changed, and account deleted; the approval email now carries a clickable
app link. The scheduled/manual demo reset also seeds sample channel subscriptions.
- Hide the 'unverified email' chip for Google accounts (Google attests the email).
- Link a Google account to a password account, and adopt the Google identity onto a matching
email account instead of 500ing on a duplicate; set or change a password from Settings.
- Expose has_google/has_password on /api/me for the Sign-in methods UI.
- Mark Google logins email-verified (backfill existing rows, migration 0024); stop a routine
login from clobbering an admin-assigned role (env ADMIN_EMAILS stays the bootstrap admin).
- Suspension login-gates (password + Google callback + current_user) with a rate-limited
'suspended' notice; shared purge_user (cascade delete + access-request cleanup + Google-grant
revoke) behind self- and admin-deletion; single app_base source for user-facing email links.
The pending-access-requests notification still said "review in Settings -> Account",
but that moved to the new Users page in 5a. Update the text (EN/HU/DE), and give the
notification a durable inbox link ("Review" -> Users page) via a new access-requests
meta kind (persists across reload, unlike a live action callback). Also stop it
piling up: persisted notices reload as dismissed history, so the old dedupe never
matched and a fresh copy was added every load — now removeByMetaKind drops any prior
access-requests notice before re-issuing, keeping exactly one current nudge.
Add DELETE /api/me/account: permanently erases the signed-in account and all its
personal data — OAuth tokens, subscriptions, tags, video states, playlists,
notifications all cascade on the users row (FK ON DELETE CASCADE); quota-audit
events are kept but anonymised (SET NULL); the email's invite/whitelist row is
removed too. Guards: the shared demo account can't be deleted, and the last
remaining admin can't delete itself. Frontend: a Danger zone in Settings -> Account
(non-demo) with a danger-confirm; on success the session clears and the app lands
on the welcome page. EN/HU/DE. Verified via curl: self-delete + session clear +
demo 403.
A security audit of every mutating endpoint found no isolation gaps — all routes
scope by current_user, admin routes are gated, and the demo account is sandboxed
in its own user_id space (can't reach others' data, escalate, or hit admin
routes). The remaining demo concern is communal pollution of its own shared state,
so add an automatic reset: a new 'demo_reset' scheduler job (admin-tunable
interval, default 12h) reuses the manual reset logic, and no-ops without ever
creating a demo account if none exists. Verified: a triggered run wiped the demo's
video states and re-seeded its sample playlists.
Public landing page with the redesigned auth card (email+password, Google,
explicit demo), feature pitch, and demo-screenshot slots. Wires the 5a auth
endpoints with inline errors. Login.tsx replaced by Welcome.tsx.
Replace the old Google-only login card with a public landing page (Welcome.tsx):
hero + auth card (email+password Sign in / Create account / Forgot password,
Continue with Google, explicit Try-the-demo) over a feature pitch (6 pain->solution
cards) and demo-account screenshot slots (graceful placeholder until the images are
dropped into frontend/public/welcome/). Auth forms wire to the 5a endpoints; errors
show inline via a new req() quiet flag (no global modal for a wrong password).
Handles ?verify, ?reset (set-new-password form), ?access banners. EN/HU/DE.
Verified end-to-end on a fresh load: render, register->approve->password-login.
New admin-only Users page (sidebar): Users & roles (list + promote/demote with
confirm; self/demo/last-admin guarded server-side), plus the Access requests
(Invite whitelist) and Demo whitelist+reset migrated out of Settings → Account
(same data/tables — UI relocated only). Settings → Account now holds personal
content only. ConfigPanel learns a boolean field type (toggle) for the new
allow_registration setting. api methods, new 'users' page/route/nav/header, and
EN/HU/DE strings (new users namespace + access group).
Add email+password auth alongside Google: argon2id hashing; users gains
password_hash/email_verified/is_active and google_sub becomes nullable (migration
0022); a single-use, hashed auth_tokens table for email verification + password
reset. Registration creates a pending (inactive, unverified) account + an access
request; sign-in needs verified email AND admin approval. Anti-enumeration: uniform
register/login/reset responses (a correct-password owner still gets a specific
pending reason). allow_registration flag (DB-tunable). Admin users-list + role
endpoint (guards: not self, not demo, not the last admin); approving access (or a
manual whitelist add) now activates the matching pending account. current_user
rejects deactivated accounts. Verified end-to-end via curl + DB.
Cut a version marker for the accumulated work staged on dev (epics 1-4): reusable
Banner; Discover total-videos column + subscribe confirm; Stats consolidation +
quota-action taxonomy; DB-backed system_config + admin Configuration page. NOT
deployed to prod yet — dev/localdev report 0.12.0, prod stays 0.11.2 until a ship.
Alembic head at this marker: 0021.
Move youtube_api_key into the DB-backed system_config (encrypted). Google OAuth
creds + admin-role UI deferred to the wizard/auth epics (see commit body).
Register youtube_api_key in the sysconfig registry as a secret and resolve it from
the DB-override-or-env layer in YouTubeClient (self.db is in scope). Adds a YouTube
API group to the Configuration page (EN/HU/DE).
Google OAuth client id/secret stay env-only for now: the authlib client is
registered at module load (auth.py), and the token-refresh reads them too, so a
runtime DB override needs a lazy/re-registerable OAuth client — that belongs to the
install-wizard epic (first-boot OAuth config without restart). Admin-role-via-UI
moves to the auth-overhaul epic (user management lives there).
Move the operational params (quota budget/reserve, backfill window, shorts-probe,
enrich/autotag batch) off env into the DB-backed system_config + admin Configuration
page, wiring each read through the resolver.
Add the four new groups to the page's group order and EN/HU/DE labels + field
hints for the 8 newly DB-overridable settings. The page is registry-driven, so
this is purely presentational (group order + i18n).
Register 8 more keys in the sysconfig registry (quota daily budget + backfill
reserve, recent-backfill window, shorts-probe params, enrich/autotag batch sizes)
and route their reads through the DB-override-or-env resolver at every call site
(quota.py, sync/runner.py, sync/videos.py, sync/autotag.py, sync/maintenance.py,
routes/scheduler.py). All read sites already had a db session in scope. Reads are
read-through (no cache), matching app.state; admin can tune them live, no redeploy.
Per-field Save buttons were an unconventional, endpoint-driven choice. Replace
them with a single drafted Save/Discard bar (consistent with the Settings page):
edits are buffered and applied together on Save (each via the existing per-key
PATCH/DELETE), Discard reverts. Reset-to-default is a per-field affordance applied
on Save too — clearing a non-secret falls back to default; a secret (which can't
be cleared by emptying) gets a toggle that resets it on save. Backend unchanged.
DB-backed system_config infrastructure (registry + resolver + Fernet-encrypted
secrets + admin API) and a registry-driven admin Configuration page, with the
Email/SMTP group wired end-to-end as the first vertical slice.
New admin-only Configuration page that renders the system_config registry grouped
(Email/SMTP first), with per-field save/reset, write-only encrypted secret fields
(disabled with a hint when TOKEN_ENCRYPTION_KEY is unset), and a Send-test-email
button. New 'config' page + sidebar nav item + header title; api methods and
EN/HU/DE strings.
Add a generic admin-editable config layer (epic 4a): system_config KV table
(migration 0021), a sysconfig registry that is the single source of truth for
DB-overridable keys (type/group/default/bounds/secret) + a DB-override-or-env
resolver, and admin endpoints (GET/PATCH/DELETE /api/admin/config + a test-email
probe). Secrets are Fernet-encrypted at rest (TOKEN_ENCRYPTION_KEY); without it
they stay env-only. First group wired end-to-end: Email/SMTP — email.py now
reads host/port/user/from/password via the resolver (own session, since email
is sent from background tasks).
The Settings > Sync tab moves into the Stats module: Stats is now a per-user
page (Overview tab = sync status + your own API usage + manual actions) with the
admin instance-wide quota dashboard + background-sync pause as an admin-only
System tab. The Stats nav item is visible to all users (was admin-only); the
Settings Sync tab is removed. Sync i18n strings move from settings.sync.* to
stats.sync.* (EN/HU/DE).
The quota-attribution action keys were inconsistent (mixed verbs, ad-hoc names,
English-only labels). Introduce a QuotaAction constants holder (one source of
truth), rename every attribution call site to it, and add migration 0020 to
rename historical quota_events.action values. The display label now resolves
from i18n (quotaActions.<key>, EN/HU/DE) instead of a hard-coded English map.
Scheduler job ids and progress phase labels are a separate namespace and are
left untouched.
Add a "Videos" column (channel total uploads, already in the discovery
response) to the Discover-from-playlists table, and guard the Subscribe button
with a confirm dialog warning it changes the real YouTube account and spends
quota (mirrors the unsubscribe guard). Strings added in EN/HU/DE.
Both top-of-app bars shared the same shell (icon + content + border-b bar);
factor it into a tone-driven Banner with optional action/dismiss. The
version/demo logic stays in the wrappers. Leaves a clean extension point for
a future news-ticker variant.
Same fix as the subscribe reconcile, applied to the remaining endpoints that
wrapped a YouTubeError in a 502: video lookup (feed) and playlist refresh,
push-plan, sync, and delete. A 502 is treated by the client as a transient
"connection lost" blip, so a real, explainable YouTube failure showed the wrong
message. 422 carries the clear detail to the error dialog instead.
Subscribing to a channel the user already follows on YouTube (a local/YouTube
desync — e.g. a stale import dropped the local row, so it resurfaced in the
discovery list) made YouTube return 400 "subscription already exists", which
the endpoint reported as a 502 — and the client shows a 502 as a transient
"connection lost" blip, not the real reason. Treat the duplicate as success:
record the subscription locally (the desired end state already holds; the next
resync fills in the resource id). Surface any other YouTube error as 422 with a
clear message, on both subscribe and unsubscribe, so it isn't mistaken for a
connection drop.
A focus-channel deep-link sets (and persists) the channel-name column filter
via the DataTable's externalFilter. That persisted value then survived into
unrelated visits: clicking the header's "N without full history" link landed
on the right tab with the right status chip, but a leftover name filter (e.g.
a channel focused in a past session) hid every row — "No channels".
Add a resetFiltersToken to DataTable that clears its column filters when it
advances, and bump it from the header intent. The token's ref starts at 0 so
the clear fires even when the table remounts on navigation, while a plain
reload (token still 0) keeps the user's persisted filters.
The Channel manager's tab (subscriptions vs playlist discovery) was persisted
locally, so the header's "N without full history" link and the focus-channel
jump (subscribe notice, tag manager) could dump the user on the Discovery tab
while quietly applying a status/name filter that only affects the subscriptions
table — the targeted channel was there, just on the hidden tab.
Lift the tab state to App alongside the status filter it pairs with, and have
those navigation intents switch it back to "subscribed".
The header polled every 30s, so the live "syncing" state (now gated on a real
running job) was usually missed between ticks. Poll every 8s while a job is
running or channels are pending, easing back to 30s once everything's settled,
via react-query's functional refetchInterval.
The "avatar + name + open-on-YouTube" cell and the @handle-or-/channel/<id>
URL were copy-pasted across the channel manager, the discovery tab, the
subscribe notice and the player. Extract a single ChannelLink component
(optional in-app onView, middle-click opens YouTube) and a channelYouTubeUrl
helper, and route all four through them. Removes the NameCell / DiscoveryNameCell
duplication (the latter introduced with the discovery tab).
Extend the reload-safe meta+link pattern (introduced for "Subscribed") to
more inbox entries, so every notice that can point somewhere does:
- A "Marked watched" activity notice now offers "Find in feed" (jumping to
the feed filtered to Watched + that channel), alongside Unwatch — matching
what the "Hidden" notice already does. VideoWatchedMeta carries the
channel, and locate() handles both hidden and watched.
- A scheduler job-completion notice (System) now offers "Open Scheduler",
jumping to the dashboard.
Maintenance notices reference already-removed videos, so they stay
informational with nothing to navigate to.
The header's sync indicator spun "fetching history" whenever any channel
still lacked full history — i.e. perpetually, even when the scheduler was
idle between its periodic runs. It now spins only while a channel-sync job
(backfill or RSS poll) is actually running; otherwise pending deep-history
work shows as the calm, static "N without full history" link, and recent
work queued for the next run shows a static "N queued". This makes the
header coherent with the Scheduler dashboard's live state.
Backend exposes running_job_ids() from the scheduler activity and a derived
sync_active flag on /api/sync/my-status.
The dashboard now renders a progress bar for any running job — determinate
when counts are reported, an indeterminate "working" sliver otherwise — so a
scheduled run is as visible as a manual one (progress was never manual-only;
the wiring is shared, but only some jobs reported and the display gated on
counts).
Poll faster (1.5s) while any job runs, easing back to 4s when idle, derived
from the freshest data by react-query's functional refetchInterval. The
earlier React-state approach to this stalled the live updates (the row froze
on the first sampled value); useLiveQuery now accepts a function of the data.
Trilingual phase labels for the newly-reporting jobs.
Every background job now reports progress, so the dashboard can show a live
bar for any run (not only enrich/backfill/shorts/maintenance): rss_poll and
subscriptions (runner), autotag, and playlist_sync gained progress.report
calls over their loops.
RSS polling now fetches the channel feeds concurrently (16 workers) instead
of one-at-a-time — a slow/unreachable feed no longer blocks the rest, cutting
a full poll of the catalogue from minutes to seconds. Feed fetches are
network-bound and run in the pool; DB inserts stay on the session's thread
(SQLAlchemy sessions aren't thread-safe), applied as each fetch completes.
Split the DB-write half of poll_rss_channel into apply_rss_feed so the fetch
and apply phases compose cleanly.
The "Subscribed on YouTube" inbox entry now names the channel and offers
two convenience links that survive a reload (driven by the typed payload,
not the live callback): "Channel manager" (jumps to the manager focused on
that channel) and "Open on YouTube". Wires the panel to the app's
focus-channel navigation; trilingual strings.
Add a "Discover from playlists" tab to the Channel manager that lists
channels appearing in the user's playlists they don't subscribe to, with
a one-click Subscribe.
- GET /api/channels/discovery: local join (playlist_items -> videos ->
channels) minus the user's subscriptions and their own channel. Enriches
stub channels' metadata up front (title/thumbnail/subscriber count via the
API key) so the user can judge a channel before subscribing; videos are
not pulled (the scheduler picks those up).
- POST /api/channels/{id}/subscribe: write-scope gated, subscriptions.insert
+ local Subscription with the returned resource id.
- YouTubeClient.insert_subscription / get_my_channel_id.
- users.yt_channel_id (migration 0019) caches the user's own channel id so
discovery can exclude it.
- Frontend: ChannelDiscovery DataTable, Channels tab toggle (persisted),
api methods, trilingual strings. Subscribe ships a typed notification
payload (ChannelSubscribedMeta) for the inbox to act on.
The note claimed 'leaving the page with unsaved changes asks first', but the
beforeunload prompt was deliberately removed (native dialog; we use in-app
ones). Only in-app section switching is guarded — reword to say exactly that.
peter