peter
0cc450d1c9
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches it from the WORKER, so any approved user could aim the server at the compose network, the LAN or a cloud metadata endpoint — and read the result off job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a port scanner. downloads/urlguard.py refuses a URL whose host resolves to a loopback/private/link-local/reserved address, at enqueue AND again in the worker (DNS can move in between, and the worker is where the socket opens). A network-class failure on a `url` job now reports one fixed sentence instead of the raw transport text; the error CODE is still classified from the original, so the localized explanation is unchanged. DNS rebinding is explicitly NOT closed — that needs the connection gated inside yt-dlp — and the module says so rather than implying more than it does. SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video, so a URL that is purely a playlist or channel still expanded into every entry — one enqueue, unbounded downloads. `playlist_items` pins it to one. Size was unbounded too: `check_enqueue` compares the CURRENT footprint against the cap, and a download's size is unknown until it lands, so one job could overshoot by any amount; the owner's remaining budget is now handed to yt-dlp as `max_filesize`, applied AFTER the adapter merge so no source adapter can raise a limit the quota layer set. A join's cut-list is capped at 100: an accurate join emits two filters per segment into one ffmpeg filter_complex. SEC-14: the YouTube budget is instance-wide, so one user holding down a manual sync spent the day for everyone. Per-user (not per-IP — the cost belongs to the account) 3-per-10-minutes limiters on the two bulk pulls, subscriptions and playlists, with separate budgets so one does not close the other. The review also listed /rss, /backfill and /enrich; E3 deleted them, and search stays unlimited on purpose — it is interactive, not a bulk pull. Gate: pytest 896 -> 924 (none skipped), vitest 671.
89 lines
4.1 KiB
Python
89 lines
4.1 KiB
Python
"""Refuse a user-supplied download URL that points back into our own network (SEC-12).
|
|
|
|
The Download Center deliberately accepts an arbitrary ``http(s)`` URL — that is the feature — and
|
|
hands it to yt-dlp running on the WORKER. Without a guard, any approved user can make the server
|
|
fetch anything the server can reach: other containers on the compose network, the LAN, a cloud
|
|
metadata endpoint. The failure text then comes back to them in ``job.error``, so "connection
|
|
refused" vs "HTTP Error 401" vs "timed out" turns the download box into an internal port scanner.
|
|
|
|
What this closes: a URL whose host resolves to a loopback / private / link-local / reserved address
|
|
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
|
|
differently between the two moments, and the worker is where the socket is actually opened.
|
|
|
|
What it does NOT close, stated plainly rather than implied: a host that passes the worker's check
|
|
and then answers yt-dlp's own resolution with an internal address (DNS rebinding). Closing that
|
|
needs the connection itself to be gated — a custom socket factory inside yt-dlp — which is a much
|
|
larger change than this one. The two checks here stop the straightforward case (typing
|
|
``http://127.0.0.1:6379`` or an internal hostname into the box), which is the reachable one.
|
|
|
|
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
|
|
user-controlled destination to guard.
|
|
"""
|
|
|
|
import ipaddress
|
|
import socket
|
|
from urllib.parse import urlparse
|
|
|
|
# Reason codes travel to the client inside the structured 422 detail, so they are part of the API
|
|
# vocabulary (frontend `localizeDetail`) — keep them stable.
|
|
REASON_SCHEME = "scheme" # not http(s)
|
|
REASON_NO_HOST = "no_host" # URL carries no hostname
|
|
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
|
|
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
|
|
|
|
|
|
class BlockedUrl(Exception):
|
|
"""A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
|
|
REASON_* codes above."""
|
|
|
|
def __init__(self, reason: str):
|
|
super().__init__(reason)
|
|
self.reason = reason
|
|
|
|
|
|
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
|
"""Every address family we must not open a connection to. Checked explicitly rather than
|
|
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
|
|
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
|
|
mapped = getattr(ip, "ipv4_mapped", None)
|
|
if mapped is not None:
|
|
ip = mapped
|
|
return bool(
|
|
ip.is_private
|
|
or ip.is_loopback
|
|
or ip.is_link_local
|
|
or ip.is_multicast
|
|
or ip.is_reserved
|
|
or ip.is_unspecified
|
|
)
|
|
|
|
|
|
def _resolve(host: str, port: int | None) -> list[str]:
|
|
"""Every address ``host`` resolves to. Raises BlockedUrl(REASON_UNRESOLVABLE) when DNS has no
|
|
answer — reported as its own outcome, because "we could not look that up" and "we refuse to
|
|
fetch that" are different things to tell a user."""
|
|
try:
|
|
infos = socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
|
|
except socket.gaierror as exc:
|
|
raise BlockedUrl(REASON_UNRESOLVABLE) from exc
|
|
return [info[4][0] for info in infos]
|
|
|
|
|
|
def check_url(raw: str) -> None:
|
|
"""Raise BlockedUrl unless ``raw`` is an http(s) URL whose host resolves only to addresses the
|
|
server is allowed to fetch from. A host with even ONE forbidden address is refused: a
|
|
round-robin answer must not become a coin flip."""
|
|
u = urlparse((raw or "").strip())
|
|
if u.scheme not in ("http", "https"):
|
|
raise BlockedUrl(REASON_SCHEME)
|
|
host = u.hostname
|
|
if not host:
|
|
raise BlockedUrl(REASON_NO_HOST)
|
|
for addr in _resolve(host, u.port):
|
|
try:
|
|
ip = ipaddress.ip_address(addr)
|
|
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
|
|
raise BlockedUrl(REASON_PRIVATE) from None
|
|
if _is_forbidden(ip):
|
|
raise BlockedUrl(REASON_PRIVATE)
|