fix(downloads,sync): bound what one user can make the server fetch, download and spend
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches it from the WORKER, so any approved user could aim the server at the compose network, the LAN or a cloud metadata endpoint — and read the result off job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a port scanner. downloads/urlguard.py refuses a URL whose host resolves to a loopback/private/link-local/reserved address, at enqueue AND again in the worker (DNS can move in between, and the worker is where the socket opens). A network-class failure on a `url` job now reports one fixed sentence instead of the raw transport text; the error CODE is still classified from the original, so the localized explanation is unchanged. DNS rebinding is explicitly NOT closed — that needs the connection gated inside yt-dlp — and the module says so rather than implying more than it does. SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video, so a URL that is purely a playlist or channel still expanded into every entry — one enqueue, unbounded downloads. `playlist_items` pins it to one. Size was unbounded too: `check_enqueue` compares the CURRENT footprint against the cap, and a download's size is unknown until it lands, so one job could overshoot by any amount; the owner's remaining budget is now handed to yt-dlp as `max_filesize`, applied AFTER the adapter merge so no source adapter can raise a limit the quota layer set. A join's cut-list is capped at 100: an accurate join emits two filters per segment into one ffmpeg filter_complex. SEC-14: the YouTube budget is instance-wide, so one user holding down a manual sync spent the day for everyone. Per-user (not per-IP — the cost belongs to the account) 3-per-10-minutes limiters on the two bulk pulls, subscriptions and playlists, with separate budgets so one does not close the other. The review also listed /rss, /backfill and /enrich; E3 deleted them, and search stays unlimited on purpose — it is interactive, not a bulk pull. Gate: pytest 896 -> 924 (none skipped), vitest 671.
This commit is contained in:
@@ -116,6 +116,7 @@ def build_ydl_opts(
|
||||
progress_hook,
|
||||
extra: dict | None = None,
|
||||
cookiefile: str | None = None,
|
||||
max_bytes: int | None = None,
|
||||
) -> dict:
|
||||
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
|
||||
|
||||
@@ -125,12 +126,20 @@ def build_ydl_opts(
|
||||
|
||||
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
|
||||
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
|
||||
merge so a source adapter can never clobber it."""
|
||||
merge so a source adapter can never clobber it.
|
||||
|
||||
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp refuses a format larger
|
||||
than it rather than filling the disk and only then being caught by the quota accounting."""
|
||||
s = normalize(spec)
|
||||
h = s["max_height"]
|
||||
opts: dict = {
|
||||
"outtmpl": outtmpl,
|
||||
"noplaylist": True,
|
||||
# SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to the single video. A
|
||||
# URL that is PURELY a playlist or channel has no such video to fall back to, so it still
|
||||
# expands into every entry — one enqueue, an unbounded number of downloads. One job means
|
||||
# one item; a user who wants the whole list enqueues it item by item, under their quota.
|
||||
"playlist_items": "1",
|
||||
"quiet": True,
|
||||
"no_warnings": True,
|
||||
"noprogress": True,
|
||||
@@ -154,6 +163,13 @@ def build_ydl_opts(
|
||||
if extra:
|
||||
opts.update(extra)
|
||||
|
||||
# SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
|
||||
# `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set. A
|
||||
# non-positive budget is left unset: "0 bytes left" is a refusal for enqueue to make, not
|
||||
# something to express here as an impossible yt-dlp filter.
|
||||
if max_bytes and max_bytes > 0:
|
||||
opts["max_filesize"] = max_bytes
|
||||
|
||||
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
|
||||
# be able to override the authenticated session the worker resolved for this specific user.
|
||||
if cookiefile:
|
||||
|
||||
Reference in New Issue
Block a user