fix(e12): close what round one broke, and revert two fixes that overreached
Round two of the review loop, on round one's own diff. Two of its findings were regressions the fixes themselves introduced, which is exactly what the loop is for: **A per-user refusal was writing onto a shared row.** MediaAssets are deduped across users, and the generic failure handler puts the message on the asset — right for an upstream failure, wrong for "your storage is full". Two users queuing the same video share one asset, so the first user hitting their cap failed the second user's job with the first one's private reason, telling someone with an empty library to free up space. The refusal now raises `_OwnerRefused`, caught separately: the job fails, the asset goes back to `pending` for whoever else holds it. Pinned by a test that drives `_download` directly — the refusal happens before extraction, so nothing needs stubbing. **The restore scripts restarted the app, undoing the restore.** `docker compose start` re-runs the container command, and that runs `alembic upgrade head` — so on the rollback path this documents as recommended (restore the dump, THEN pin the older tag) the newer image immediately re-applied the migrations the restore had just undone, and on a FAILED restore it put the app back on a half-restored schema. Both are the opposite of why the stop was added. Nothing is restarted automatically now; the scripts say what to run and when, and the docs' snippets say the same. **Reverted rather than patched:** the OnboardingWizard toast (the wizard already renders that failure inline, next to a Retry button — three recoveries for one failure), and `useVerifyEntry`'s "each new link clears the other". The latter blanked the signed-out card's only banner, where nothing renders the reset flag to replace it. The two states are independent again; what changes is that `dismiss` peels ONE layer, which is what the original finding was actually about. Also: `Stats` uses the YouTube helper rather than the bare one, so the route's English 403 detail no longer beats the translated fallback under a Hungarian UI; `urlparse` guarded alongside `.port` (an unclosed bracket raised just as late), the reason renamed `malformed` since "bad port" misdescribes it; three docstrings narrowed to what the code actually does (the worker's byte check is per-job, not an aggregate ledger, and not on the edit path; the redirect residual leaks a reachability oracle, which the note had denied; the invalidation helper does not cover suspend); the pasted Windows recipe wrapped so `throw` can stop it; the bash one-liner's failure branch no longer exits 0; and the tests that were missing for the round-one fixes — the account panel's error paths, the invalidation set, and the shared-asset case above. Gate green: vitest 685, pytest 950.
This commit is contained in:
@@ -8,10 +8,18 @@ explanation (and decides whether a retry makes sense). The raw text stays availa
|
||||
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
|
||||
"""
|
||||
|
||||
# Failure texts the WORKER itself raises, as opposed to ones yt-dlp/ffmpeg produce. They live here,
|
||||
# beside the rules that classify them, because a message and its category drifting apart is exactly
|
||||
# how a category silently becomes unreachable — the rule below matches a fragment of each, and
|
||||
# test_download_errors pins that they still classify as intended.
|
||||
# Two worker-authored strings that this module owns because both are coupled to `_RULES`.
|
||||
#
|
||||
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
|
||||
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
|
||||
# rule drifting apart is exactly how a category silently becomes unreachable.
|
||||
#
|
||||
# NETWORK_REDACTED_MESSAGE is the opposite direction: it REPLACES a message after classification, so
|
||||
# it deliberately has no rule. Nothing re-classifies a stored `job.error` today; if anything ever
|
||||
# does, it must carry the original code rather than re-deriving one from this text.
|
||||
#
|
||||
# Other worker-raised texts (ffmpeg, "produced no output file", the urlguard refusal) are still
|
||||
# inline literals at their raise sites — pinned by markers here, but not owned here.
|
||||
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
|
||||
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
|
||||
|
||||
|
||||
@@ -37,6 +37,18 @@ class _Aborted(Exception):
|
||||
"""Raised inside the progress hook when the job was paused/canceled mid-download."""
|
||||
|
||||
|
||||
class _OwnerRefused(Exception):
|
||||
"""A refusal that belongs to THIS job's owner, not to the source.
|
||||
|
||||
MediaAssets are shared: two users asking for the same video and format get one row (see
|
||||
`service.get_or_create_asset`), and the generic failure handler writes the message onto it —
|
||||
correctly, because an upstream failure really is everyone's. A per-user refusal is not. Written
|
||||
to the shared row it would fail a second user's queued job for the same video with the first
|
||||
user's private reason (`worker._process_job` reads `asset.error` verbatim), telling someone with
|
||||
an empty library to free up space. So this is caught separately: the job fails, the asset goes
|
||||
back to `pending` for whoever else is holding it."""
|
||||
|
||||
|
||||
# Both progress sources call back FAR faster than the database should be touched: yt-dlp fires its
|
||||
# hook per chunk, and ffmpeg's `-progress pipe:1` emits a block of ~12 lines every ~0.5s while
|
||||
# run_ffmpeg calls its callbacks once per LINE. Every callback here is a round-trip (`_job_status`
|
||||
@@ -452,7 +464,7 @@ def _download(
|
||||
# the only one downloading unbounded. Fail the job here instead. Not retryable until they
|
||||
# free space, which is what the `quota_exceeded` category tells them.
|
||||
if remaining_bytes == 0:
|
||||
raise RuntimeError(dlerrors.QUOTA_FULL_MESSAGE)
|
||||
raise _OwnerRefused(dlerrors.QUOTA_FULL_MESSAGE)
|
||||
info = _media_info(
|
||||
_extract_with_fallback(
|
||||
job_id,
|
||||
@@ -554,6 +566,23 @@ def _download(
|
||||
# job's status (the route set it to paused/canceled). ref_count is adjusted by the route.
|
||||
_reset_asset_pending(asset_id)
|
||||
log.info("job %s aborted (pause/cancel)", job_id)
|
||||
except _OwnerRefused as exc:
|
||||
# Fail this job only. The asset is untouched and goes back to `pending`, so another user's
|
||||
# job for the same video is claimed and downloaded normally instead of inheriting a reason
|
||||
# that is about someone else's account.
|
||||
_reset_asset_pending(asset_id)
|
||||
msg = str(exc)[:500]
|
||||
with SessionLocal() as db:
|
||||
service.finalize_if_running(
|
||||
db,
|
||||
job_id,
|
||||
status=JobStatus.ERROR,
|
||||
error=msg,
|
||||
error_code=dlerrors.classify(msg),
|
||||
phase=None,
|
||||
)
|
||||
db.commit()
|
||||
log.info("job %s refused for its owner: %s", job_id, msg)
|
||||
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
|
||||
msg = str(exc)[:500]
|
||||
code = dlerrors.classify(msg)
|
||||
|
||||
@@ -6,9 +6,14 @@ Limits come from a per-user DownloadQuota row if the admin set one, else the sys
|
||||
`unlimited` (e.g. the admin's own account) bypasses the byte cap.
|
||||
|
||||
Enforcement split: max_jobs + max_bytes are checked at enqueue (they bound holdings); per-user
|
||||
max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads). max_bytes
|
||||
is checked a SECOND time in the worker, via `remaining_bytes` — enqueue can only compare the
|
||||
footprint as it stood when the job was created, and earlier jobs may have filled the cap since.
|
||||
max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads).
|
||||
|
||||
DOWNLOAD jobs additionally consult `remaining_bytes` in the worker, because enqueue can only compare
|
||||
the footprint as it stood when the job was created and earlier jobs may have filled the cap since.
|
||||
That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user each read the same
|
||||
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
|
||||
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
|
||||
is unknown until it finishes. Edit jobs do not consult it at all (tracked as SEC-19).
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
@@ -17,9 +17,12 @@ the connection itself with a custom socket factory inside yt-dlp, a much larger
|
||||
with an internal address.
|
||||
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
|
||||
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
|
||||
open redirect reaches it, so this is the cheaper of the two — but it still buys only what a
|
||||
media-shaped internal response yields, since a non-media body fails the mp4 remux and stores
|
||||
nothing. Tracked as SEC-17.
|
||||
open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
|
||||
response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
|
||||
the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
|
||||
class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
|
||||
one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
|
||||
kept). Tracked as SEC-17.
|
||||
|
||||
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
|
||||
hostname into the box), which is the reachable one.
|
||||
@@ -36,7 +39,7 @@ from urllib.parse import urlparse
|
||||
# vocabulary (frontend `localizeDetail`) — keep them stable.
|
||||
REASON_SCHEME = "scheme" # not http(s)
|
||||
REASON_NO_HOST = "no_host" # URL carries no hostname
|
||||
REASON_BAD_PORT = "bad_port" # ":99999" / ":abc" — urlparse only raises on ACCESS, so guard it
|
||||
REASON_MALFORMED = "malformed" # unparseable authority or port; urlparse raises late, so guard it
|
||||
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
|
||||
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
|
||||
|
||||
@@ -82,20 +85,20 @@ def check_url(raw: str) -> None:
|
||||
"""Raise BlockedUrl unless ``raw`` is an http(s) URL whose host resolves only to addresses the
|
||||
server is allowed to fetch from. A host with even ONE forbidden address is refused: a
|
||||
round-robin answer must not become a coin flip."""
|
||||
u = urlparse((raw or "").strip())
|
||||
# `urlparse` itself raises ValueError on a malformed authority ("http://[::1/x" — an unclosed
|
||||
# bracket), and `.port` raises on ":99999" / ":abc" when READ rather than at parse time. Both
|
||||
# escape the caller's `except BlockedUrl` as a 500 unless caught here, which is the opposite of
|
||||
# what this module exists to do: turn a bad URL into a structured, localizable refusal.
|
||||
try:
|
||||
u = urlparse((raw or "").strip())
|
||||
port = u.port
|
||||
except ValueError as exc:
|
||||
raise BlockedUrl(REASON_MALFORMED) from exc
|
||||
if u.scheme not in ("http", "https"):
|
||||
raise BlockedUrl(REASON_SCHEME)
|
||||
host = u.hostname
|
||||
if not host:
|
||||
raise BlockedUrl(REASON_NO_HOST)
|
||||
# `.port` is a lazily-parsed property: it raises ValueError for ":99999" or ":abc" on ACCESS,
|
||||
# not at urlparse() time, and `.hostname` above happily returns first. Unguarded, one typo
|
||||
# escapes the caller's `except BlockedUrl` as a 500 instead of the structured 422 this whole
|
||||
# module exists to return.
|
||||
try:
|
||||
port = u.port
|
||||
except ValueError as exc:
|
||||
raise BlockedUrl(REASON_BAD_PORT) from exc
|
||||
for addr in _resolve(host, port):
|
||||
try:
|
||||
ip = ipaddress.ip_address(addr)
|
||||
|
||||
@@ -66,7 +66,15 @@ def resolve_source(raw: str) -> tuple[str, str]:
|
||||
if _VIDEO_ID.match(raw):
|
||||
return "youtube", raw
|
||||
if raw.startswith(("http://", "https://")):
|
||||
u = urlparse(raw)
|
||||
# `urlparse` raises ValueError on a malformed authority ("http://[::1/x"), and this runs
|
||||
# BEFORE the urlguard check that would otherwise turn it into a structured refusal — so
|
||||
# unguarded, that typo escapes as a 500. Same answer as any other unusable link.
|
||||
try:
|
||||
u = urlparse(raw)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="That doesn't look like a valid video link."
|
||||
) from exc
|
||||
host = u.netloc.lower()
|
||||
if "youtube.com" in host:
|
||||
if u.path.startswith(("/shorts/", "/embed/", "/live/")):
|
||||
|
||||
@@ -95,7 +95,7 @@ def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_u
|
||||
from app.routes import playlists as plroutes
|
||||
|
||||
user = make_user()
|
||||
pl = Playlist(user_id=user.id, name="Linked", kind="youtube", yt_playlist_id="PL123")
|
||||
pl = Playlist(user_id=user.id, name="Linked", source="youtube", yt_playlist_id="PL123")
|
||||
db.add(pl)
|
||||
db.commit()
|
||||
monkeypatch.setattr(plroutes, "has_read_scope", lambda u: True)
|
||||
|
||||
@@ -10,7 +10,8 @@ Uses the `db` fixture (real Postgres) — skipped when no database is reachable.
|
||||
its OWN session, so every test COMMITS its setup before calling it.
|
||||
"""
|
||||
|
||||
from app.downloads import quota
|
||||
from app.config import settings
|
||||
from app.downloads import pipeline, quota
|
||||
from app.models import DownloadJob, DownloadQuota, MediaAsset, User
|
||||
from app.worker import _claim_job
|
||||
|
||||
@@ -228,3 +229,45 @@ def test_remaining_bytes_is_zero_once_the_cap_is_met(db):
|
||||
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
|
||||
db.flush()
|
||||
assert quota.remaining_bytes(db, u.id) == 0
|
||||
|
||||
|
||||
def test_a_refusal_for_one_user_does_not_poison_the_shared_asset(db, monkeypatch, tmp_path):
|
||||
"""A user at their cap fails their own job, and another user's queued job for the same video
|
||||
still downloads.
|
||||
|
||||
MediaAssets are deduped across users, and the generic failure handler writes the message onto
|
||||
that shared row — correct for an upstream failure, wrong for "YOUR storage is full", which
|
||||
`worker._process_job` would then hand verbatim to someone with an empty library.
|
||||
"""
|
||||
monkeypatch.setattr(settings, "download_root", str(tmp_path))
|
||||
over, other = User(email="over@example.com"), User(email="other@example.com")
|
||||
db.add_all([over, other])
|
||||
db.flush()
|
||||
db.add(
|
||||
DownloadQuota(
|
||||
user_id=over.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False
|
||||
)
|
||||
)
|
||||
full = _sized_asset(db, 1500, "already-held")
|
||||
_job(db, over.id, queue_pos=0, status="done", asset_id=full.id)
|
||||
|
||||
shared = MediaAsset(
|
||||
source_kind="youtube", source_ref="x" * 11, format_sig="s", status="pending", ref_count=2
|
||||
)
|
||||
db.add(shared)
|
||||
db.flush()
|
||||
mine = _job(db, over.id, queue_pos=1, status="running", asset_id=shared.id)
|
||||
theirs = _job(db, other.id, queue_pos=2, asset_id=shared.id)
|
||||
db.commit()
|
||||
|
||||
# The refusal happens before extraction, so nothing needs stubbing — if it ever stops happening
|
||||
# first, this test starts reaching the network, which is its own signal.
|
||||
pipeline._download(mine.id, shared.id, "youtube", "x" * 11, {}, over.id)
|
||||
|
||||
db.expire_all()
|
||||
assert db.get(DownloadJob, mine.id).status == "error"
|
||||
assert db.get(DownloadJob, mine.id).error_code == "quota_exceeded"
|
||||
# The asset is released, NOT marked error — otherwise the next line's job inherits the refusal.
|
||||
assert db.get(MediaAsset, shared.id).status == "pending"
|
||||
assert db.get(MediaAsset, shared.id).error is None
|
||||
assert db.get(DownloadJob, theirs.id).status == "queued"
|
||||
|
||||
@@ -87,10 +87,13 @@ def test_a_url_without_a_host_is_refused():
|
||||
assert _blocked_reason("http:///just/a/path") == urlguard.REASON_NO_HOST
|
||||
|
||||
|
||||
def test_a_malformed_port_is_a_refusal_not_a_crash(monkeypatch):
|
||||
# `urlparse(...).port` raises ValueError on ACCESS, not at parse time, and `.hostname` returns
|
||||
# first — so an unguarded read turns a user typo into a 500 that escapes the caller's
|
||||
# `except BlockedUrl` and loses the localized detail this module exists to produce.
|
||||
def test_a_malformed_url_is_a_refusal_not_a_crash(monkeypatch):
|
||||
# Both shapes raise ValueError LATE — `urlparse` on an unclosed bracket, `.port` only when the
|
||||
# property is READ — and `.hostname` happily returns first. Unguarded, either turns a user typo
|
||||
# into a 500 that escapes the caller's `except BlockedUrl` and loses the localized detail this
|
||||
# module exists to produce.
|
||||
_stub_dns(monkeypatch, ["93.184.216.34"])
|
||||
assert _blocked_reason("http://example.com:99999/v.mp4") == urlguard.REASON_BAD_PORT
|
||||
assert _blocked_reason("http://example.com:abc/v.mp4") == urlguard.REASON_BAD_PORT
|
||||
assert _blocked_reason("http://example.com:99999/v.mp4") == urlguard.REASON_MALFORMED
|
||||
assert _blocked_reason("http://example.com:abc/v.mp4") == urlguard.REASON_MALFORMED
|
||||
assert _blocked_reason("http://[::1/v.mp4") == urlguard.REASON_MALFORMED
|
||||
assert _blocked_reason("http://[example.com]/v.mp4") == urlguard.REASON_MALFORMED
|
||||
|
||||
Reference in New Issue
Block a user