fix(e12): close the eighteen findings the E12 review round left standing

The deep review of the branch raised 47 findings, 18 of which survived adversarial
verification. Grouped by what they actually were:

**The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user
already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one
user over quota was the only one downloading with no ceiling at all. `check_enqueue`
could not catch it: it ran while the job was queued and the user was still under the cap.
The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision
lives, and where it can be tested), and the worker now fails the job outright rather than
proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help
until space is freed.

**Silent failures in the new account panel.** All three actions went through `quiet: true`
endpoints, which suppress the global error dialog on the promise that the caller reports
inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On
the broken-SMTP instance this panel exists for, an admin would read silence as "the mail
went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`,
shared with the pending-access queue, so the two copies of these mutations cannot drift
apart again — which is how half of this was lost in the first place.

**A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps
back one migration, but a release routinely ships several (0.80.0 shipped four): it drops
the newest one's columns and leaves the schema at a revision the older image also cannot
resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is
now the recommended path, and the alembic route asks the target version for its own head
revision instead of counting backwards.

Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they
share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as
a 500; the pull cooldown extended to the sibling route spending the same QuotaAction;
restore stops the app before dropping its tables; the PowerShell cleanup can no longer
decide the script's verdict on 5.1, nor report the wrong exit code; the documented
one-liners hardened to match the scripts; the registry's retention window stated where
pinning is advertised; the two banners made mutually exclusive so neither can shadow the
other; `hover:bg-hover` (a utility this project never generated) replaced by the
`selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to
the two callers that discarded it.

Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset
banner UAT'd on localdev.
This commit is contained in:
2026-09-05 03:41:58 +02:00
parent f98b060e2b
commit fe38dfbcca
28 changed files with 475 additions and 84 deletions
+3 -1
View File
@@ -96,7 +96,9 @@ docker compose -f docker-compose.selfhost.yml up -d
That follows `latest`, which moves with every release. Set `IMAGE_TAG=0.80.0` in `.env` to pin a That follows `latest`, which moves with every release. Set `IMAGE_TAG=0.80.0` in `.env` to pin a
version and upgrade on your own schedule; released versions are listed in [`CHANGELOG.md`](CHANGELOG.md) version and upgrade on your own schedule; released versions are listed in [`CHANGELOG.md`](CHANGELOG.md)
and in the app's Release Notes. and in the app's Release Notes. The registry keeps only the newest few of those, so a pin decides
*when* you take the next hop rather than letting you stay put forever — see
[docs/self-hosting.md](docs/self-hosting.md#upgrading).
**Back up before upgrading.** Migrations run automatically on startup and are not undone by pulling **Back up before upgrading.** Migrations run automatically on startup and are not undone by pulling
the old image back. A backup is the database dump **plus your `.env`** — `TOKEN_ENCRYPTION_KEY` is the old image back. A backup is the database dump **plus your `.env`** — `TOKEN_ENCRYPTION_KEY` is
+27
View File
@@ -8,9 +8,19 @@ explanation (and decides whether a retry makes sense). The raw text stays availa
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this". `_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
""" """
# Failure texts the WORKER itself raises, as opposed to ones yt-dlp/ffmpeg produce. They live here,
# beside the rules that classify them, because a message and its category drifting apart is exactly
# how a category silently becomes unreachable — the rule below matches a fragment of each, and
# test_download_errors pins that they still classify as intended.
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
# (marker, category) — first match wins, so order most-specific → most-general. Matching is # (marker, category) — first match wins, so order most-specific → most-general. Matching is
# case-insensitive on a substring, against the raw yt-dlp/ffmpeg message. # case-insensitive on a substring, against the raw yt-dlp/ffmpeg message.
_RULES: tuple[tuple[str, str], ...] = ( _RULES: tuple[tuple[str, str], ...] = (
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
# first — it is our wording, so it can never be a false match on an upstream message.
("download storage is full", "quota_exceeded"),
# Login / age / members / bot-check — the user must be signed in (we deliberately are not). # Login / age / members / bot-check — the user must be signed in (we deliberately are not).
("sign in to confirm your age", "login_required"), ("sign in to confirm your age", "login_required"),
("confirm your age", "login_required"), ("confirm your age", "login_required"),
@@ -91,3 +101,20 @@ def classify(msg: str | None) -> str:
if marker in low: if marker in low:
return category return category
return "unknown" return "unknown"
def user_error_text(source_kind: str, msg: str, code: str) -> str:
"""The failure text to STORE on the job, given the raw message and its category.
SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection
refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner with
extra steps, since the raw text is shown to the user behind the details disclosure. `urlguard`
stops the fetch aiming at our own network; this stops the remaining readout.
Only the network class, and only for ``url`` jobs: a YouTube job has no user-chosen host, and
every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis. A separate
function rather than an inline branch in the worker so it can be pinned by a test — the raw
text leaking back is a security regression that would otherwise pass a green suite."""
if source_kind == "url" and code == "network":
return NETWORK_REDACTED_MESSAGE
return msg
+15 -15
View File
@@ -440,13 +440,19 @@ def _download(
cookiefile = credentials.resolve_cookiefile( cookiefile = credentials.resolve_cookiefile(
cdb, user_id, url, asset_sig, spec, staging cdb, user_id, url, asset_sig, spec, staging
) )
# SEC-13: what this user has left. `check_enqueue` only compares the CURRENT footprint # SEC-13: what this user has left. `check_enqueue` only compared the footprint as it
# against the cap — a download's size is unknown until it finishes, so a single job could # stood when the job was QUEUED — a download's size is unknown until it finishes, so a
# previously blow through the cap by any amount. Handing the remainder to yt-dlp as # single job could previously blow through the cap by any amount. Handing the remainder
# `max_filesize` makes the format selector refuse an oversized file up front instead. # to yt-dlp as `max_filesize` makes the format selector refuse an oversized file up
lim = quota.resolve(cdb, user_id) # front instead.
if not lim.unlimited: remaining_bytes = quota.remaining_bytes(cdb, user_id)
remaining_bytes = max(0, lim.max_bytes - quota.footprint(cdb, user_id)) # …and zero is a REFUSAL, not "no limit". Earlier jobs of this user may have filled the cap
# while this one sat in the queue, and 0 is not an expressible yt-dlp `max_filesize`, so
# passing it down would leave the ceiling unset — the one user already at their cap would be
# the only one downloading unbounded. Fail the job here instead. Not retryable until they
# free space, which is what the `quota_exceeded` category tells them.
if remaining_bytes == 0:
raise RuntimeError(dlerrors.QUOTA_FULL_MESSAGE)
info = _media_info( info = _media_info(
_extract_with_fallback( _extract_with_fallback(
job_id, job_id,
@@ -551,14 +557,8 @@ def _download(
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
msg = str(exc)[:500] msg = str(exc)[:500]
code = dlerrors.classify(msg) code = dlerrors.classify(msg)
# SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection # SEC-12: redact a transport-level readout for user-supplied URLs (see user_error_text).
# refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner msg = dlerrors.user_error_text(source_kind, msg, code)
# with extra steps, since the raw text is shown to the user behind the details disclosure.
# The urlguard above stops it aiming at our own network; this stops the remaining readout.
# Only the network class, and only for `url` jobs: a YouTube job has no user-chosen host, and
# every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis.
if source_kind == "url" and code == "network":
msg = "The server could not reach that address."
with SessionLocal() as db: with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id) asset = db.get(MediaAsset, asset_id)
if asset: if asset:
+18 -1
View File
@@ -6,7 +6,9 @@ Limits come from a per-user DownloadQuota row if the admin set one, else the sys
`unlimited` (e.g. the admin's own account) bypasses the byte cap. `unlimited` (e.g. the admin's own account) bypasses the byte cap.
Enforcement split: max_jobs + max_bytes are checked at enqueue (they bound holdings); per-user Enforcement split: max_jobs + max_bytes are checked at enqueue (they bound holdings); per-user
max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads). max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads). max_bytes
is checked a SECOND time in the worker, via `remaining_bytes` — enqueue can only compare the
footprint as it stood when the job was created, and earlier jobs may have filled the cap since.
""" """
from dataclasses import dataclass from dataclasses import dataclass
@@ -122,6 +124,21 @@ def check_enqueue(db: Session, user_id: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, used) raise QuotaExceeded("max_bytes", lim.max_bytes, used)
def remaining_bytes(db: Session, user_id: int) -> int | None:
"""Storage this user may still spend on ONE download: ``None`` for an unlimited user, else the
cap minus the current footprint, floored at 0.
Zero is a real answer, not "no limit" — it means the cap is already met and the caller must
REFUSE rather than proceed unbounded. `check_enqueue` cannot make that refusal on the caller's
behalf: it ran when the job was queued, while the user was still under the cap, and the jobs
that filled it have landed since. Keeping the arithmetic here (rather than inline in the worker)
is what keeps every max_bytes decision in this module."""
lim = resolve(db, user_id)
if lim.unlimited:
return None
return max(0, lim.max_bytes - footprint(db, user_id))
def check_upload(db: Session, user_id: int, add_bytes: int) -> None: def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
"""Like check_enqueue, but for an upload whose size is KNOWN at ingest: also reject when adding """Like check_enqueue, but for an upload whose size is KNOWN at ingest: also reject when adding
this file would push the footprint over max_bytes. A download can't pre-check its size (unknown this file would push the footprint over max_bytes. A download can't pre-check its size (unknown
+23 -6
View File
@@ -10,11 +10,19 @@ What this closes: a URL whose host resolves to a loopback / private / link-local
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
differently between the two moments, and the worker is where the socket is actually opened. differently between the two moments, and the worker is where the socket is actually opened.
What it does NOT close, stated plainly rather than implied: a host that passes the worker's check What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
and then answers yt-dlp's own resolution with an internal address (DNS rebinding). Closing that the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
needs the connection itself to be gated — a custom socket factory inside yt-dlp — which is a much
larger change than this one. The two checks here stop the straightforward case (typing * DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
``http://127.0.0.1:6379`` or an internal hostname into the box), which is the reachable one. with an internal address.
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
open redirect reaches it, so this is the cheaper of the two — but it still buys only what a
media-shaped internal response yields, since a non-media body fails the mp4 remux and stores
nothing. Tracked as SEC-17.
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
hostname into the box), which is the reachable one.
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
user-controlled destination to guard. user-controlled destination to guard.
@@ -28,6 +36,7 @@ from urllib.parse import urlparse
# vocabulary (frontend `localizeDetail`) — keep them stable. # vocabulary (frontend `localizeDetail`) — keep them stable.
REASON_SCHEME = "scheme" # not http(s) REASON_SCHEME = "scheme" # not http(s)
REASON_NO_HOST = "no_host" # URL carries no hostname REASON_NO_HOST = "no_host" # URL carries no hostname
REASON_BAD_PORT = "bad_port" # ":99999" / ":abc" — urlparse only raises on ACCESS, so guard it
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked") REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
@@ -79,7 +88,15 @@ def check_url(raw: str) -> None:
host = u.hostname host = u.hostname
if not host: if not host:
raise BlockedUrl(REASON_NO_HOST) raise BlockedUrl(REASON_NO_HOST)
for addr in _resolve(host, u.port): # `.port` is a lazily-parsed property: it raises ValueError for ":99999" or ":abc" on ACCESS,
# not at urlparse() time, and `.hostname` above happily returns first. Unguarded, one typo
# escapes the caller's `except BlockedUrl` as a 500 instead of the structured 422 this whole
# module exists to return.
try:
port = u.port
except ValueError as exc:
raise BlockedUrl(REASON_BAD_PORT) from exc
for addr in _resolve(host, port):
try: try:
ip = ipaddress.ip_address(addr) ip = ipaddress.ip_address(addr)
except ValueError: # getaddrinfo returned something unparseable; treat as unusable except ValueError: # getaddrinfo returned something unparseable; treat as unusable
+15 -2
View File
@@ -29,9 +29,22 @@ router = APIRouter(prefix="/api/playlists", tags=["playlists"])
# caller owns and then every item in each, against the instance-wide YouTube budget. Its own # caller owns and then every item in each, against the instance-wide YouTube budget. Its own
# limiter rather than a shared one — spending the manual subscription-import allowance must not # limiter rather than a shared one — spending the manual subscription-import allowance must not
# also close the playlist button, and vice versa. # also close the playlist button, and vice versa.
#
# It guards EVERY route that spends under QuotaAction.PLAYLISTS_PULL, not just the bulk one: a
# scripted loop over the single-playlist re-pull drains the same shared budget, and limiting one of
# two routes with the same quota action is the asymmetric fix this codebase keeps re-learning.
# Deliberately NOT on the push side (`push-plan` / `push`): those run on every push attempt,
# including the no-op "already up to date" path, so a 3-per-10-minute allowance would refuse
# legitimate work. Their exposure is tracked as SEC-18 instead.
_playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600) _playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600)
def _check_pull_cooldown(user: User) -> None:
"""Shared gate so the two PLAYLISTS_PULL routes cannot drift apart again."""
if not _playlist_pull_limiter.allow(str(user.id)):
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
def _own_playlist(db: Session, user: User, playlist_id: int) -> Playlist: def _own_playlist(db: Session, user: User, playlist_id: int) -> Playlist:
pl = db.get(Playlist, playlist_id) pl = db.get(Playlist, playlist_id)
if pl is None or pl.user_id != user.id: if pl is None or pl.user_id != user.id:
@@ -291,8 +304,7 @@ def sync_youtube(
status_code=403, status_code=403,
detail="Connect YouTube (read access) to sync your playlists.", detail="Connect YouTube (read access) to sync your playlists.",
) )
if not _playlist_pull_limiter.allow(str(user.id)): _check_pull_cooldown(user)
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL): with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return sync_user_playlists(db, user) return sync_user_playlists(db, user)
@@ -313,6 +325,7 @@ def revert_youtube(
raise HTTPException( raise HTTPException(
status_code=403, detail="Connect YouTube (read access) to sync your playlists." status_code=403, detail="Connect YouTube (read access) to sync your playlists."
) )
_check_pull_cooldown(user)
try: try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL): with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return repull_playlist(db, user, pl) return repull_playlist(db, user, pl)
+25
View File
@@ -85,3 +85,28 @@ def test_the_cooldown_is_per_user_not_global(db, as_user, make_user, monkeypatch
ca.post("/api/sync/subscriptions") ca.post("/api/sync/subscriptions")
assert ca.post("/api/sync/subscriptions").status_code == 429 assert ca.post("/api/sync/subscriptions").status_code == 429
assert as_user(b).post("/api/sync/subscriptions").status_code == 200 assert as_user(b).post("/api/sync/subscriptions").status_code == 200
def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_user, monkeypatch):
# SEC-14, the sweep half: `revert-youtube` spends under the SAME QuotaAction.PLAYLISTS_PULL key
# as the bulk `sync-youtube`, so limiting only one of them left the identical drain open one
# route over. Both must draw on the one allowance.
from app.models import Playlist
from app.routes import playlists as plroutes
user = make_user()
pl = Playlist(user_id=user.id, name="Linked", kind="youtube", yt_playlist_id="PL123")
db.add(pl)
db.commit()
monkeypatch.setattr(plroutes, "has_read_scope", lambda u: True)
monkeypatch.setattr(plroutes, "repull_playlist", lambda d, u, p: {"ok": True})
monkeypatch.setattr(plroutes, "sync_user_playlists", lambda d, u: {"ok": True})
c = as_user(user)
codes = [c.post(f"/api/playlists/{pl.id}/revert-youtube").status_code for _ in range(4)]
assert codes[:3] == [200, 200, 200]
assert codes[3] == 429
# One allowance, not one each: the bulk route is now on cooldown too.
r = c.post("/api/playlists/sync-youtube")
assert r.status_code == 429
assert r.json()["detail"] == {"code": "sync_cooldown"}
+20
View File
@@ -1,5 +1,6 @@
"""Category classification of raw yt-dlp / ffmpeg download-failure strings (S2).""" """Category classification of raw yt-dlp / ffmpeg download-failure strings (S2)."""
from app.downloads import errors as dlerrors
from app.downloads.errors import classify from app.downloads.errors import classify
@@ -54,3 +55,22 @@ def test_login_beats_generic_unavailable_when_both_present():
"This video is unavailable" "This video is unavailable"
) )
assert classify(msg) == "login_required" assert classify(msg) == "login_required"
def test_our_own_worker_refusals_classify_as_themselves():
# The worker raises these itself (not yt-dlp), and the rule that catches each one matches only a
# fragment of the text. Pinning message and rule together is what stops an edit to the wording
# from silently orphaning the category — the UI would then fall back to "Download failed".
assert classify(dlerrors.QUOTA_FULL_MESSAGE) == "quota_exceeded"
def test_a_url_jobs_network_failure_is_redacted_but_youtube_keeps_its_text():
# SEC-12: the raw transport text for a user-chosen host:port is a port-scanner readout, so it is
# replaced for `url` jobs. Everything else must survive verbatim, or a real diagnosis is lost.
raw = "ERROR: Unable to download webpage: <urlopen error [Errno 111] Connection refused>"
assert dlerrors.user_error_text("url", raw, "network") == dlerrors.NETWORK_REDACTED_MESSAGE
# Same failure, but the destination was never user-chosen — nothing to leak, keep the detail.
assert dlerrors.user_error_text("youtube", raw, "network") == raw
# A user-chosen host, but a category that says something real about the CONTENT, not the network.
login = "ERROR: [facebook] 123: You must be logged in to see this post"
assert dlerrors.user_error_text("url", login, "login_required") == login
+59
View File
@@ -10,6 +10,7 @@ Uses the `db` fixture (real Postgres) — skipped when no database is reachable.
its OWN session, so every test COMMITS its setup before calling it. its OWN session, so every test COMMITS its setup before calling it.
""" """
from app.downloads import quota
from app.models import DownloadJob, DownloadQuota, MediaAsset, User from app.models import DownloadJob, DownloadQuota, MediaAsset, User
from app.worker import _claim_job from app.worker import _claim_job
@@ -169,3 +170,61 @@ def test_list_downloads_is_capped(db, monkeypatch):
monkeypatch.setattr(dl, "_MAX_LIST", 2) monkeypatch.setattr(dl, "_MAX_LIST", 2)
rows = dl.list_downloads(user=u, db=db) rows = dl.list_downloads(user=u, db=db)
assert len(rows) == 2 # bounded by the cap, not the 3 rows on disk assert len(rows) == 2 # bounded by the cap, not the 3 rows on disk
# --- E12 · S3 · the worker-side max_bytes remainder ------------------------------------------
def _sized_asset(db, size_bytes, ref):
a = MediaAsset(
source_kind="url",
source_ref=ref,
format_sig="s",
status="ready",
ref_count=1,
size_bytes=size_bytes,
)
db.add(a)
db.flush()
return a
def test_remaining_bytes_is_none_for_an_unlimited_user(db):
u = User(email="unlimited@example.com")
db.add(u)
db.flush()
_quota(db, u.id, max_concurrent=2, unlimited=True)
db.flush()
# None, not a number: "no cap" and "no budget left" must never be the same value downstream —
# build_ydl_opts leaves max_filesize unset for both, so only the caller can tell them apart.
assert quota.remaining_bytes(db, u.id) is None
def test_remaining_bytes_is_the_unspent_part_of_the_cap(db):
u = User(email="partial@example.com")
db.add(u)
db.flush()
db.add(
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False)
)
a = _sized_asset(db, 400, "partial-1")
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
db.flush()
assert quota.remaining_bytes(db, u.id) == 600
def test_remaining_bytes_is_zero_once_the_cap_is_met(db):
# THE case this exists for: `check_enqueue` let these jobs in while the user was under the cap,
# and an earlier one has filled it since. Zero is what makes the worker refuse; if this ever
# returned the raw negative or None instead, the user already over quota would be the only one
# downloading with no size ceiling at all.
u = User(email="atcap@example.com")
db.add(u)
db.flush()
db.add(
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False)
)
a = _sized_asset(db, 1500, "atcap-1") # overshot: one job finished larger than the remainder
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
db.flush()
assert quota.remaining_bytes(db, u.id) == 0
+9
View File
@@ -85,3 +85,12 @@ def test_non_http_schemes_are_refused_before_any_lookup(monkeypatch, url):
def test_a_url_without_a_host_is_refused(): def test_a_url_without_a_host_is_refused():
assert _blocked_reason("http:///just/a/path") == urlguard.REASON_NO_HOST assert _blocked_reason("http:///just/a/path") == urlguard.REASON_NO_HOST
def test_a_malformed_port_is_a_refusal_not_a_crash(monkeypatch):
# `urlparse(...).port` raises ValueError on ACCESS, not at parse time, and `.hostname` returns
# first — so an unguarded read turns a user typo into a 500 that escapes the caller's
# `except BlockedUrl` and loses the localized detail this module exists to produce.
_stub_dns(monkeypatch, ["93.184.216.34"])
assert _blocked_reason("http://example.com:99999/v.mp4") == urlguard.REASON_BAD_PORT
assert _blocked_reason("http://example.com:abc/v.mp4") == urlguard.REASON_BAD_PORT
+61 -12
View File
@@ -172,22 +172,41 @@ Dump and restore the database with these two commands. They also exist as script
(`scripts/backup.sh` + `scripts/restore.sh`, or `backup.ps1` / `restore.ps1` on Windows) if you have (`scripts/backup.sh` + `scripts/restore.sh`, or `backup.ps1` / `restore.ps1` on Windows) if you have
the source checkout: the source checkout:
```bash If you customised `POSTGRES_USER` / `POSTGRES_DB` in `.env`, substitute them for `siftlode` below —
# Back up (writes backups/siftlode-<timestamp>.dump) the scripts read them from the environment, these one-liners cannot.
mkdir -p backups && docker compose -f docker-compose.selfhost.yml exec -T db \
pg_dump -U siftlode -Fc siftlode > "backups/siftlode-$(date +%Y%m%d-%H%M%S).dump"
# Restore into the running instance ```bash
# Back up. Write to a .part name and rename only on success: redirecting straight into the final
# name creates the file BEFORE pg_dump runs, so a failed dump leaves a plausible-looking 0-byte
# file in backups/ that you will not discover until the day you need it.
mkdir -p backups && out="backups/siftlode-$(date +%Y%m%d-%H%M%S).dump" && \
docker compose -f docker-compose.selfhost.yml exec -T db \
pg_dump -U siftlode -Fc siftlode > "$out.part" \
&& [ -s "$out.part" ] && mv "$out.part" "$out" && echo "wrote $out" \
|| { rm -f "$out.part"; echo "BACKUP FAILED"; }
# Restore. Stop the app first: pg_restore --clean DROPs each object in its own transaction, and a
# running api holds locks that make those DROPs queue — abort the wait and you are left with a
# half-dropped database. Restoring with the app down also means nothing writes into a schema that
# is only partly back.
docker compose -f docker-compose.selfhost.yml stop api worker
docker compose -f docker-compose.selfhost.yml exec -T db \ docker compose -f docker-compose.selfhost.yml exec -T db \
pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
docker compose -f docker-compose.selfhost.yml up -d
``` ```
On Windows, do **not** pipe the dump through PowerShell — it corrupts binary data. Write it inside the On Windows, do **not** pipe the dump through PowerShell — it corrupts binary data. Write it inside the
container and copy it out instead: container and copy it out instead. Check the exit code between the two steps and use a timestamped
name: the dump inside the container keeps its name, so a `cp` that runs after a *failed* dump would
copy out the previous run's file and quietly overwrite your only backup with a stale one.
```powershell ```powershell
New-Item -ItemType Directory -Force -Path backups | Out-Null
$out = "backups\siftlode-$(Get-Date -Format yyyyMMdd-HHmmss).dump"
docker compose -f docker-compose.selfhost.yml exec -T db pg_dump -U siftlode -Fc siftlode -f /tmp/s.dump docker compose -f docker-compose.selfhost.yml exec -T db pg_dump -U siftlode -Fc siftlode -f /tmp/s.dump
docker compose -f docker-compose.selfhost.yml cp db:/tmp/s.dump backups\siftlode.dump if ($LASTEXITCODE -ne 0) { throw "BACKUP FAILED - pg_dump returned $LASTEXITCODE" }
docker compose -f docker-compose.selfhost.yml cp db:/tmp/s.dump $out
docker compose -f docker-compose.selfhost.yml exec -T db rm -f /tmp/s.dump
``` ```
## Upgrading ## Upgrading
@@ -211,23 +230,53 @@ Then `pull` + `up -d` stays on that version until you change the line. Released
changed in them are listed in the app itself, under **Release Notes** in the account menu, and in changed in them are listed in the app itself, under **Release Notes** in the account menu, and in
[`CHANGELOG.md`](../CHANGELOG.md). [`CHANGELOG.md`](../CHANGELOG.md).
**The registry keeps only the newest few releases, not every version the changelog lists.** A pin is
"decide when you take the next hop", not "stay here forever": once you are several releases behind,
your pinned tag can no longer be pulled — which matters the day you rebuild the host, prune your
Docker images, or move to a new machine. A container that is already running is unaffected (Docker
keeps the image locally), and you can always go back to `latest` or build the image yourself from the
repository. If you want to sit on one version long term, keep your own copy:
`docker save git.b1fr0st.eu/peter/siftlode:0.80.0 -o siftlode-0.80.0.tar`.
### Rollback ### Rollback
If a release misbehaves, set `IMAGE_TAG` back to the previous version and `up -d`. That is enough If a release misbehaves, set `IMAGE_TAG` back to the previous version and `up -d`. That is enough
**as long as the release did not change the database schema**. If it did, the older image will refuse **as long as the release did not change the database schema**. If it did, the older image will refuse
to start against the newer schema, and you need to step the schema back first: to start against the newer schema.
**Restoring your pre-upgrade dump is the recommended way out of that** — it puts the database back
exactly where the old image expects it, in one step, with no revision arithmetic:
```bash ```bash
# 1. stop the app (leave the database up) # 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker docker compose -f docker-compose.selfhost.yml stop api worker
# 2. step the schema back one revision # 2. restore the dump you took before upgrading (see Backups above)
docker compose -f docker-compose.selfhost.yml run --rm api alembic downgrade -1 docker compose -f docker-compose.selfhost.yml exec -T db \
pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
# 3. pin the previous version in .env, then start again # 3. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d docker compose -f docker-compose.selfhost.yml up -d
``` ```
If you are not sure whether the schema moved, restoring your pre-upgrade dump is the safer path — it If you have no usable dump, you can step the schema back instead — but **not with `alembic downgrade
puts the database back exactly where the old image expects it. -1`**. One release routinely ships several migrations (0.80.0 shipped four), so `-1` undoes only the
newest one and leaves the schema at a revision the older image *also* cannot resolve: you lose that
migration's data and still get a crash loop. Downgrade to the revision the target version expects,
which that version can tell you itself:
```bash
# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. ask the version you are rolling back TO which schema revision it expects
IMAGE_TAG=0.79.0 docker compose -f docker-compose.selfhost.yml run --rm api alembic heads
# 3. step the schema back to exactly that revision — run this with the CURRENT (newer) image,
# because it is the one that ships the migrations being undone
docker compose -f docker-compose.selfhost.yml run --rm api alembic downgrade <revision from step 2>
# 4. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d
```
A downgrade drops the columns the newer version added, so anything stored only in those columns is
gone. That is the price of not having the dump.
### A migration failed on start ### A migration failed on start
+24 -10
View File
@@ -7,6 +7,8 @@ import { LoadingState, StateMessage } from "./QueryState";
import { api } from "../lib/api"; import { api } from "../lib/api";
import { qk } from "../lib/queryKeys"; import { qk } from "../lib/queryKeys";
import { notify } from "../lib/notifications"; import { notify } from "../lib/notifications";
import { copyText } from "../lib/clipboard";
import { invalidateUserAccess, isEmailOff } from "../lib/adminUsers";
import type { AdminUserAccess, AdminUserRow } from "../lib/api/admin"; import type { AdminUserAccess, AdminUserRow } from "../lib/api/admin";
// The admin's answer to "why can this person not get in?", and the place every recovery action for // The admin's answer to "why can this person not get in?", and the place every recovery action for
@@ -57,35 +59,47 @@ export default function AccountPanel({
queryFn: () => api.adminUserAccess(user.id), queryFn: () => api.adminUserAccess(user.id),
}); });
// Every action here goes through a `quiet: true` endpoint, which suppresses the GLOBAL error
// dialog on the promise that the caller reports the failure inline. Without an `onError` that
// promise is broken and a refusal renders as nothing at all — on the broken-SMTP instance this
// panel exists for, the admin would read silence as "the mail went out".
const issue = useMutation({ const issue = useMutation({
mutationFn: () => api.createSetPasswordLink(user.id), mutationFn: () => api.createSetPasswordLink(user.id),
onSuccess: (r) => setLink({ url: r.url, expires_at: r.expires_at }), onSuccess: (r) => setLink({ url: r.url, expires_at: r.expires_at }),
onError: () => notify({ level: "error", message: t("users.account.linkFailed") }),
}); });
const resend = useMutation({ const resend = useMutation({
mutationFn: () => api.resendUserVerification(user.id), mutationFn: () => api.resendUserVerification(user.id),
onSuccess: () => notify({ level: "success", message: t("users.account.resendSent") }), onSuccess: () => notify({ level: "success", message: t("users.account.resendSent") }),
onError: (err) =>
notify({
level: "error",
message: isEmailOff(err)
? t("users.pendingAccess.resendEmailOff")
: t("users.pendingAccess.resendFailed"),
}),
}); });
const activate = useMutation({ const activate = useMutation({
mutationFn: () => api.activateUser(user.id), mutationFn: () => api.activateUser(user.id),
onSuccess: () => { onSuccess: () => invalidateUserAccess(qc, user.id),
qc.invalidateQueries({ queryKey: qk.adminUsers() }); onError: () => notify({ level: "error", message: t("users.pendingAccess.activateFailed") }),
qc.invalidateQueries({ queryKey: qk.adminUserAccess(user.id) });
},
}); });
const copy = async () => { const copy = async () => {
if (!link) return; if (!link) return;
try { // Via the shared helper, whose synchronous `execCommand` path is what makes copying work on a
await navigator.clipboard.writeText(link.url); // plain-http LAN origin — where `navigator.clipboard` is undefined, and which is exactly the
// minimal self-host this no-email link was built for.
if (await copyText(link.url)) {
setCopied(true); setCopied(true);
window.setTimeout(() => setCopied(false), 2000); window.setTimeout(() => setCopied(false), 2000);
} catch { return;
// Clipboard access can be refused (permissions, an insecure origin). The link is on screen and
// selectable, so this is a convenience that failed — not the action failing.
notify({ level: "warning", message: t("users.account.copyFailed") });
} }
// The link is on screen and selectable, so a failed copy is a convenience that failed — not the
// action failing. Say so rather than leaving the button looking inert.
notify({ level: "warning", message: t("users.account.copyFailed") });
}; };
const a: AdminUserAccess | undefined = q.data; const a: AdminUserAccess | undefined = q.data;
+6 -13
View File
@@ -24,6 +24,7 @@ import {
type PendingAccessRow, type PendingAccessRow,
} from "../lib/api"; } from "../lib/api";
import { notify } from "../lib/notifications"; import { notify } from "../lib/notifications";
import { invalidateUserAccess, isEmailOff } from "../lib/adminUsers";
import { ADMIN_USERS_TAB_KEY } from "../lib/adminUsersTab"; import { ADMIN_USERS_TAB_KEY } from "../lib/adminUsersTab";
import Tooltip from "./Tooltip"; import Tooltip from "./Tooltip";
import InlineNotice from "./InlineNotice"; import InlineNotice from "./InlineNotice";
@@ -215,7 +216,7 @@ function UsersRoles({ me }: { me: Me }) {
<button <button
onClick={() => setPanelFor(u)} onClick={() => setPanelFor(u)}
title={t("users.account.open")} title={t("users.account.open")}
className="min-w-0 flex-1 text-left rounded-lg -m-1 p-1 hover:bg-hover transition" className="selectable cursor-pointer min-w-0 flex-1 text-left rounded-lg -m-1 p-1"
> >
<div className="text-sm truncate"> <div className="text-sm truncate">
{u.display_name ?? u.email.split("@")[0]} {u.display_name ?? u.email.split("@")[0]}
@@ -325,12 +326,9 @@ function PendingAccess() {
queryKey: qk.adminPendingAccess(), queryKey: qk.adminPendingAccess(),
queryFn: () => api.adminPendingAccess(), queryFn: () => api.adminPendingAccess(),
}); });
// Activating changes /api/me's badge count too, so refresh both or the number lies until reload. // Activating changes /api/me's badge count too, so refresh all of them or the number lies until
const refresh = () => { // reload. Shared with the account panel's copy of this mutation (see lib/adminUsers).
qc.invalidateQueries({ queryKey: qk.adminPendingAccess() }); const refresh = () => invalidateUserAccess(qc);
qc.invalidateQueries({ queryKey: qk.me() });
qc.invalidateQueries({ queryKey: qk.adminUsers() });
};
const activate = useMutation({ const activate = useMutation({
mutationFn: (u: PendingAccessRow) => api.activateUser(u.id), mutationFn: (u: PendingAccessRow) => api.activateUser(u.id),
@@ -353,7 +351,7 @@ function PendingAccess() {
notify({ notify({
level: "error", level: "error",
message: message:
err instanceof HttpError && isEmailOff(err) isEmailOff(err)
? t("users.pendingAccess.resendEmailOff") ? t("users.pendingAccess.resendEmailOff")
: t("users.pendingAccess.resendFailed"), : t("users.pendingAccess.resendFailed"),
}), }),
@@ -390,11 +388,6 @@ function PendingAccess() {
} }
/** Did the resend fail because this instance has no working SMTP? */ /** Did the resend fail because this instance has no working SMTP? */
function isEmailOff(err: HttpError): boolean {
const d = err.detailData as { code?: string; reason?: string } | null | undefined;
return d?.code === "resend" && d.reason === "email_off";
}
// One stuck sign-up. The row names WHICH gate is shut rather than just offering buttons — the same // One stuck sign-up. The row names WHICH gate is shut rather than just offering buttons — the same
// reasoning as the sign-in card's blocked panel, from the admin's side: an admin about to let someone // reasoning as the sign-in card's blocked panel, from the admin's side: an admin about to let someone
// in should see that the address has not been confirmed yet, because on a no-SMTP instance that is // in should see that the address has not been confirmed yet, because on a no-SMTP instance that is
@@ -5,6 +5,7 @@ import { qk } from "../lib/queryKeys";
import { AlertTriangle, Check, Loader2, RefreshCw, ShieldCheck, X, Youtube } from "lucide-react"; import { AlertTriangle, Check, Loader2, RefreshCw, ShieldCheck, X, Youtube } from "lucide-react";
import { api, type Me } from "../lib/api"; import { api, type Me } from "../lib/api";
import { beginGrant, endOnboarding } from "../lib/onboarding"; import { beginGrant, endOnboarding } from "../lib/onboarding";
import { notifyActionError } from "../lib/actionErrors";
import Overlay from "./Overlay"; import Overlay from "./Overlay";
// A first-login wizard that walks the user through granting YouTube access one scope at a // A first-login wizard that walks the user through granting YouTube access one scope at a
@@ -78,6 +79,10 @@ export default function OnboardingWizard({ me, onClose }: { me: Me; onClose: ()
qc.invalidateQueries({ queryKey: qk.channels() }); qc.invalidateQueries({ queryKey: qk.channels() });
qc.invalidateQueries({ queryKey: qk.myStatus() }); qc.invalidateQueries({ queryKey: qk.myStatus() });
}, },
// Say something rather than nothing: this route is rate-limited (SEC-14) and can 403 without a
// read grant, and a swallowed rejection here leaves the wizard looking like it simply did not
// run. The server's own reason wins over the fallback.
onError: (e) => notifyActionError(e, t("onboarding.importFailed")),
}); });
// First time we see read access with zero imported channels, pull the user's // First time we see read access with zero imported channels, pull the user's
+5 -1
View File
@@ -6,6 +6,7 @@ import { History, Pause, Play, RefreshCw } from "lucide-react";
import { api, type AdminQuotaRow, type Me } from "../lib/api"; import { api, type AdminQuotaRow, type Me } from "../lib/api";
import { formatEta, quotaActionLabel } from "../lib/format"; import { formatEta, quotaActionLabel } from "../lib/format";
import { notify } from "../lib/notifications"; import { notify } from "../lib/notifications";
import { notifyActionError } from "../lib/actionErrors";
import { LS, useAccountPersistedState } from "../lib/storage"; import { LS, useAccountPersistedState } from "../lib/storage";
import Tooltip from "./Tooltip"; import Tooltip from "./Tooltip";
import { LoadingState, StateMessage } from "./QueryState"; import { LoadingState, StateMessage } from "./QueryState";
@@ -64,7 +65,10 @@ function Overview({ me }: { me: Me }) {
message: t("stats.sync.synced", { count: r.subscriptions ?? 0 }), message: t("stats.sync.synced", { count: r.subscriptions ?? 0 }),
}); });
}, },
onError: () => notify({ level: "error", message: t("stats.sync.syncFailed") }), // Via notifyActionError so the server's own reason wins over the generic fallback: this route
// is rate-limited (SEC-14), and "Sync failed" would report a pace limit as a fault while the
// same button on the Channels page explains it. A zero-arg onError discards that detail.
onError: (e) => notifyActionError(e, t("stats.sync.syncFailed")),
}); });
const deepAll = useMutation({ const deepAll = useMutation({
mutationFn: () => api.deepAll(true), mutationFn: () => api.deepAll(true),
+6 -1
View File
@@ -17,7 +17,12 @@ export default function VerifyBanner({ entry }: { entry: VerifyEntry }) {
// A reset link opened here can only be ACTED on signed out, so say that rather than nothing — // A reset link opened here can only be ACTED on signed out, so say that rather than nothing —
// silence is precisely what made the verify half a bug. The token stays in the address bar, so // silence is precisely what made the verify half a bug. The token stays in the address bar, so
// signing out from this same tab picks it up. // signing out from this same tab picks it up.
if (!entry.outcome && entry.resetPending) { //
// Tested on `resetPending` ALONE, not `!outcome && resetPending`: the hook keeps the two states
// mutually exclusive (each new link clears the other), so an extra `!outcome` here would only add
// an ordering dependency on the early return below — tidy that return upwards one day and the
// banner silently becomes unreachable, which is the bug this branch was added to fix.
if (entry.resetPending) {
return ( return (
<Banner <Banner
tone="warning" tone="warning"
+6 -1
View File
@@ -234,7 +234,8 @@
"no_host": "That link doesn't contain a website address.", "no_host": "That link doesn't contain a website address.",
"unresolvable": "That address couldn't be looked up. Check the link and try again.", "unresolvable": "That address couldn't be looked up. Check the link and try again.",
"private": "That address is on the server's own network, so it can't be downloaded from.", "private": "That address is on the server's own network, so it can't be downloaded from.",
"generic": "That address can't be downloaded from." "generic": "That address can't be downloaded from.",
"bad_port": "That link's port number isn't valid."
}, },
"fail": { "fail": {
"title": "Download failed", "title": "Download failed",
@@ -280,6 +281,10 @@
"unknown": { "unknown": {
"short": "Download failed", "short": "Download failed",
"body": "Something went wrong while downloading this item. The technical details below may help." "body": "Something went wrong while downloading this item. The technical details below may help."
},
"quota_exceeded": {
"short": "Storage full",
"body": "This download was stopped because your storage quota is already used up. Delete something from your library, then try again."
} }
} }
}, },
+2 -1
View File
@@ -25,5 +25,6 @@
"title": "All set", "title": "All set",
"body": "Read and unsubscribe access are both granted. You can review or revoke either one anytime in Settings → Account, or from your Google Account.", "body": "Read and unsubscribe access are both granted. You can review or revoke either one anytime in Settings → Account, or from your Google Account.",
"done": "Done" "done": "Done"
} },
"importFailed": "Couldn't import your subscriptions. You can try again from Statistics."
} }
+2 -1
View File
@@ -91,6 +91,7 @@
"resend": "Send a confirmation link", "resend": "Send a confirmation link",
"resendSent": "Confirmation link sent.", "resendSent": "Confirmation link sent.",
"setPassword": "Create a set-password link", "setPassword": "Create a set-password link",
"setPasswordAgain": "Create another link" "setPasswordAgain": "Create another link",
"linkFailed": "Couldn't create the link. Please try again."
} }
} }
+6 -1
View File
@@ -234,7 +234,8 @@
"no_host": "Ez a hivatkozás nem tartalmaz webcímet.", "no_host": "Ez a hivatkozás nem tartalmaz webcímet.",
"unresolvable": "Ezt a címet nem sikerült feloldani. Ellenőrizd a hivatkozást, és próbáld újra.", "unresolvable": "Ezt a címet nem sikerült feloldani. Ellenőrizd a hivatkozást, és próbáld újra.",
"private": "Ez a cím a szerver saját hálózatán van, ezért nem tölthető le róla.", "private": "Ez a cím a szerver saját hálózatán van, ezért nem tölthető le róla.",
"generic": "Erről a címről nem lehet letölteni." "generic": "Erről a címről nem lehet letölteni.",
"bad_port": "Ennek a hivatkozásnak érvénytelen a portszáma."
}, },
"fail": { "fail": {
"title": "A letöltés nem sikerült", "title": "A letöltés nem sikerült",
@@ -280,6 +281,10 @@
"unknown": { "unknown": {
"short": "A letöltés nem sikerült", "short": "A letöltés nem sikerült",
"body": "Valami hiba történt az elem letöltése közben. Az alábbi technikai részletek segíthetnek." "body": "Valami hiba történt az elem letöltése közben. Az alábbi technikai részletek segíthetnek."
},
"quota_exceeded": {
"short": "Betelt a tárhely",
"body": "Ez a letöltés leállt, mert a tárhelykereted már elfogyott. Törölj valamit a könyvtáradból, aztán próbáld újra."
} }
} }
}, },
+2 -1
View File
@@ -25,5 +25,6 @@
"title": "Minden kész", "title": "Minden kész",
"body": "Az olvasási és a leiratkozási hozzáférés is engedélyezve van. Bármelyiket bármikor áttekintheted vagy visszavonhatod a Beállítások → Fiók menüben, vagy a Google-fiókodban.", "body": "Az olvasási és a leiratkozási hozzáférés is engedélyezve van. Bármelyiket bármikor áttekintheted vagy visszavonhatod a Beállítások → Fiók menüben, vagy a Google-fiókodban.",
"done": "Kész" "done": "Kész"
} },
"importFailed": "Nem sikerült importálni a feliratkozásaidat. Újrapróbálhatod a Statisztika oldalon."
} }
+2 -1
View File
@@ -91,6 +91,7 @@
"resend": "Megerősítő link küldése", "resend": "Megerősítő link küldése",
"resendSent": "Megerősítő link elküldve.", "resendSent": "Megerősítő link elküldve.",
"setPassword": "Jelszóbeállító link készítése", "setPassword": "Jelszóbeállító link készítése",
"setPasswordAgain": "Új link készítése" "setPasswordAgain": "Új link készítése",
"linkFailed": "Nem sikerült létrehozni a linket. Próbáld újra."
} }
} }
+32
View File
@@ -0,0 +1,32 @@
import type { QueryClient } from "@tanstack/react-query";
import { HttpError } from "./api";
import { qk } from "./queryKeys";
// Shared bits of the admin's user-management surface, extracted because the SAME two mutations
// (activate, resend-verification) are now offered from two places — the pending-access queue and
// the account panel — and the second copy quietly dropped half of what the first one does.
//
// Grouped invalidation lives per domain rather than in `qk` (see the contract note there);
// `invalidateDownloads` is the same pattern for the downloads domain.
/** Everything that goes stale when a user's ACCESS state changes (activate, approve, suspend).
*
* `me` is in the set because the admin badge counts pending activations, and `admin-pending-access`
* because the queue is exactly the list of accounts this action removes one from — refreshing only
* `admin-users` leaves the badge and the queue asserting a person is still stuck after they were
* let in. One function so the two call sites cannot drift apart again. */
export function invalidateUserAccess(qc: QueryClient, userId?: number) {
qc.invalidateQueries({ queryKey: qk.adminUsers() });
qc.invalidateQueries({ queryKey: qk.adminPendingAccess() });
qc.invalidateQueries({ queryKey: qk.me() });
if (userId !== undefined) qc.invalidateQueries({ queryKey: qk.adminUserAccess(userId) });
}
/** Did an admin resend fail because the instance has no working mail, rather than for a real
* error? The admin-side resend deliberately reports failures (the public one is silent), so the
* message must be able to say WHICH — a no-SMTP instance would otherwise read as a bug. */
export function isEmailOff(err: unknown): boolean {
if (!(err instanceof HttpError)) return false;
const d = err.detailData as { code?: string; reason?: string } | null | undefined;
return d?.code === "resend" && d.reason === "email_off";
}
+13 -2
View File
@@ -59,7 +59,15 @@ export function useVerifyEntry(): VerifyEntry {
// Both halves of the mailed pair arrive the same way — pasted into a running tab, which is a // Both halves of the mailed pair arrive the same way — pasted into a running tab, which is a
// same-document navigation the import-time snapshot never sees. The verify half got this // same-document navigation the import-time snapshot never sees. The verify half got this
// listener in E9.5 S2; the reset half was left out, so it stayed silent (DX-17). // listener in E9.5 S2; the reset half was left out, so it stayed silent (DX-17).
if (hash.get("reset")) setResetPending(true); if (hash.get("reset")) {
setResetPending(true);
// Same rule as a second verify link below: a newly pasted link must not be shadowed by a
// banner still standing from the previous one. Without this the reset guidance is
// unreachable whenever an unread verify outcome is on screen, and `dismiss` then clears the
// pending flag along with the outcome — the DX-17 silence, one state later.
setOutcome(null);
setEmail(null);
}
}; };
// replaceState (how clearEntryUrl strips the fragment) does NOT fire hashchange, so this cannot // replaceState (how clearEntryUrl strips the fragment) does NOT fire hashchange, so this cannot
// feed itself. // feed itself.
@@ -74,9 +82,12 @@ export function useVerifyEntry(): VerifyEntry {
clearEntryUrl(); clearEntryUrl();
if (!token || consumed.current.has(token)) return; if (!token || consumed.current.has(token)) return;
consumed.current.add(token); consumed.current.add(token);
// A second link arriving over an unread first outcome must not leave the OLD banner standing. // A second link arriving over an unread first outcome must not leave the OLD banner standing —
// whichever kind that outcome was. Clearing both keeps the two states mutually exclusive, which
// is what lets the single `dismiss` below clear "whatever is showing" without eating the other.
setOutcome(null); setOutcome(null);
setEmail(null); setEmail(null);
setResetPending(false);
api api
.verifyEmail(token) .verifyEmail(token)
.then((r) => { .then((r) => {
+36
View File
@@ -157,6 +157,42 @@ describe("confirming an emailed verification link", () => {
expect(screen.getByTestId("reset").textContent).toBe("reset-pending"); expect(screen.getByTestId("reset").textContent).toBe("reset-pending");
}); });
// The two banners are mutually exclusive by construction, and each new link clears the other. Once
// they could coexist, the reset guidance was gated behind "no outcome showing" and a standing
// confirmation swallowed it — DX-17 again, one state later — and dismissing that confirmation then
// cleared the pending reset with it, so the guidance could never appear at all.
it("lets a pasted reset link supersede a confirmation banner still on screen", async () => {
const verifyEmail = vi.fn().mockResolvedValue({ ok: true, email: "someone@example.com" });
const outcome = await mountAt("/#verify=tok-first", verifyEmail);
expect(outcome()).toBe("ok:someone@example.com");
await act(async () => {
window.history.replaceState(window.history.state, "", "/#reset=pasted");
window.dispatchEvent(new HashChangeEvent("hashchange"));
});
expect(screen.getByTestId("reset").textContent).toBe("reset-pending");
expect(outcome()).toBe("none:-"); // the old banner is gone, so nothing shadows the guidance
});
it("lets a pasted verify link supersede a pending reset notice", async () => {
// The other direction, for the same reason: whichever link arrived last is the one the user is
// acting on, and only one of the two states may be true for `dismiss` to be unambiguous.
const verifyEmail = vi.fn().mockResolvedValue({ ok: true, email: "second@example.com" });
const outcome = await mountAt("/#reset=first", verifyEmail);
expect(screen.getByTestId("reset").textContent).toBe("reset-pending");
await act(async () => {
window.history.replaceState(window.history.state, "", "/#verify=tok-second");
window.dispatchEvent(new HashChangeEvent("hashchange"));
});
expect(outcome()).toBe("ok:second@example.com");
expect(screen.getByTestId("reset").textContent).toBe("-");
});
// This hook runs on the FIRST commit of App, which is several commits before the setup wizard, the // This hook runs on the FIRST commit of App, which is several commits before the setup wizard, the
// ?link=/?oauth= effects and the lazily-loaded module views read their own parameters. It used to // ?link=/?oauth= effects and the lazily-loaded module views read their own parameters. It used to
// blank the whole query and null history.state, which silently broke first-run setup (the wizard // blank the whole query and null history.state, which silently broke first-run setup (the wizard
+17 -4
View File
@@ -12,6 +12,15 @@ $ErrorActionPreference = "Stop"
# encoding name", so this script simply failed there. # encoding name", so this script simply failed there.
$remote = "/tmp/siftlode-backup.dump" $remote = "/tmp/siftlode-backup.dump"
# Best-effort cleanup of the in-container dump. Deliberately NOT `docker compose ... 2>$null`: on
# Windows PowerShell 5.1 a redirected native stderr becomes an ErrorRecord, and the
# $ErrorActionPreference above makes it TERMINATING — so one stray Compose warning would kill the
# script here, masking the real diagnostic below or reporting a finished backup as a failure. A
# cleanup step must never decide the verdict.
function Remove-RemoteDump($path) {
try { docker compose exec -T db rm -f $path 2>&1 | Out-Null } catch { }
}
# `docker compose` finds docker-compose.yml on its own and honours $env:COMPOSE_FILE. The documented # `docker compose` finds docker-compose.yml on its own and honours $env:COMPOSE_FILE. The documented
# self-host layout (docs/self-hosting.md § Get the files) has NEITHER — just the selfhost file. # self-host layout (docs/self-hosting.md § Get the files) has NEITHER — just the selfhost file.
if (-not $env:COMPOSE_FILE -and -not (Test-Path "docker-compose.yml") -and (Test-Path "docker-compose.selfhost.yml")) { if (-not $env:COMPOSE_FILE -and -not (Test-Path "docker-compose.yml") -and (Test-Path "docker-compose.selfhost.yml")) {
@@ -25,14 +34,18 @@ $ts = Get-Date -Format "yyyyMMdd-HHmmss"
$out = "backups\siftlode-$ts.dump" $out = "backups\siftlode-$ts.dump"
docker compose exec -T db pg_dump -U $user -Fc $dbname -f $remote docker compose exec -T db pg_dump -U $user -Fc $dbname -f $remote
if ($LASTEXITCODE -ne 0) { # Capture BEFORE the cleanup call: any native command overwrites $LASTEXITCODE, so interpolating it
docker compose exec -T db rm -f $remote 2>$null # after Remove-RemoteDump would report the cleanup's code (0) inside a branch that only runs when
throw "BACKUP FAILED — pg_dump returned $LASTEXITCODE; no file was written." # pg_dump failed — a message contradicting itself. Same reason $cpCode is captured below.
$dumpCode = $LASTEXITCODE
if ($dumpCode -ne 0) {
Remove-RemoteDump $remote
throw "BACKUP FAILED — pg_dump returned $dumpCode; no file was written."
} }
docker compose cp "db:$remote" $out docker compose cp "db:$remote" $out
$cpCode = $LASTEXITCODE $cpCode = $LASTEXITCODE
docker compose exec -T db rm -f $remote 2>$null Remove-RemoteDump $remote
if ($cpCode -ne 0) { throw "BACKUP FAILED — could not copy the dump out of the container (exit $cpCode)." } if ($cpCode -ne 0) { throw "BACKUP FAILED — could not copy the dump out of the container (exit $cpCode)." }
if (-not (Test-Path $out) -or (Get-Item $out).Length -eq 0) { if (-not (Test-Path $out) -or (Get-Item $out).Length -eq 0) {
if (Test-Path $out) { Remove-Item $out } if (Test-Path $out) { Remove-Item $out }
+27 -10
View File
@@ -11,6 +11,13 @@ $ErrorActionPreference = "Stop"
# which corrupts binary on 5.1 and cannot express `-Encoding Byte` at all on 7. # which corrupts binary on 5.1 and cannot express `-Encoding Byte` at all on 7.
$remote = "/tmp/siftlode-restore.dump" $remote = "/tmp/siftlode-restore.dump"
# See backup.ps1: `2>$null` on a native command is a TERMINATING error under $ErrorActionPreference
# = "Stop" on Windows PowerShell 5.1, so a cleanup step could kill the script after the restore
# already succeeded — and the obvious next move, re-running the restore, wipes the database again.
function Remove-RemoteDump($path) {
try { docker compose exec -T db rm -f $path 2>&1 | Out-Null } catch { }
}
if (-not $env:COMPOSE_FILE -and -not (Test-Path "docker-compose.yml") -and (Test-Path "docker-compose.selfhost.yml")) { if (-not $env:COMPOSE_FILE -and -not (Test-Path "docker-compose.yml") -and (Test-Path "docker-compose.selfhost.yml")) {
$env:COMPOSE_FILE = "docker-compose.selfhost.yml" $env:COMPOSE_FILE = "docker-compose.selfhost.yml"
} }
@@ -20,16 +27,26 @@ $dbname = if ($env:POSTGRES_DB) { $env:POSTGRES_DB } else { "siftlode" }
if (-not (Test-Path $DumpFile)) { throw "Dump file not found: $DumpFile" } if (-not (Test-Path $DumpFile)) { throw "Dump file not found: $DumpFile" }
docker compose cp $DumpFile "db:$remote" docker compose cp $DumpFile "db:$remote"
if ($LASTEXITCODE -ne 0) { throw "RESTORE FAILED — could not copy the dump into the container (exit $LASTEXITCODE)." } $cpCode = $LASTEXITCODE
if ($cpCode -ne 0) { throw "RESTORE FAILED — could not copy the dump into the container (exit $cpCode)." }
docker compose exec -T db pg_restore -U $user -d $dbname --clean --if-exists $remote # Stop the app before dropping its tables. `--clean` DROPs each object in its own transaction, and a
$restoreCode = $LASTEXITCODE # running api (with its scheduler) holds locks that make those DROPs queue behind it — abort the
docker compose exec -T db rm -f $remote 2>$null # wait and the database is left half-dropped. It also keeps the app from writing into a schema that
# is only partly back. Restarted in `finally`, so a failed restore does not leave the stack down.
docker compose stop api worker
try {
docker compose exec -T db pg_restore -U $user -d $dbname --clean --if-exists $remote
$restoreCode = $LASTEXITCODE
Remove-RemoteDump $remote
# pg_restore exits non-zero for ignorable warnings too (--clean --if-exists on a fresh database # pg_restore exits non-zero for ignorable warnings too (--clean --if-exists on a fresh database
# reports every missing object it tried to drop), so a non-zero code is reported as "check this", # reports every missing object it tried to drop), so a non-zero code is reported as "check this",
# not swallowed and not dressed up as success. # not swallowed and not dressed up as success.
if ($restoreCode -ne 0) { if ($restoreCode -ne 0) {
throw "pg_restore reported errors while restoring $DumpFile (exit $restoreCode) — read the output above before treating this instance as restored." throw "pg_restore reported errors while restoring $DumpFile (exit $restoreCode) — read the output above before treating this instance as restored."
}
Write-Output "Restored $DumpFile"
} finally {
docker compose start api worker | Out-Null
} }
Write-Output "Restored $DumpFile"
+9
View File
@@ -20,6 +20,15 @@ if [ ! -f "$FILE" ]; then echo "Dump file not found: $FILE"; exit 1; fi
USER_NAME="${POSTGRES_USER:-siftlode}" USER_NAME="${POSTGRES_USER:-siftlode}"
DB_NAME="${POSTGRES_DB:-siftlode}" DB_NAME="${POSTGRES_DB:-siftlode}"
# Stop the app before dropping its tables. `--clean` DROPs each object in its own transaction, and a
# running api (with its scheduler) holds locks that make those DROPs queue behind it — abort the wait
# and the database is left half-dropped. It also keeps the app from writing into a schema that is
# only partly back. The trap restarts it however this script ends, so a failed restore does not leave
# the stack down.
docker compose stop api worker
restart_app() { docker compose start api worker >/dev/null 2>&1 || true; }
trap restart_app EXIT
# pg_restore exits non-zero for ignorable warnings too (--clean --if-exists on a fresh database # pg_restore exits non-zero for ignorable warnings too (--clean --if-exists on a fresh database
# reports every missing object it tried to drop), so a non-zero code is reported as "check this", # reports every missing object it tried to drop), so a non-zero code is reported as "check this",
# not swallowed and not dressed up as success. # not swallowed and not dressed up as success.