fix(downloads): a user's download is judged at every connection it makes, not only at the URL typed
The URL guard judged the address the user typed; a public host's 3xx into our network, a page's formats pointing inside, and a name rebound between the check and the fetch all got past it, and the stored error told a live internal port from a dead one. Every non-YouTube download now runs through an egress gate on the worker's loopback: it resolves each host once, refuses any forbidden answer before dialling, and connects to the address it vetted. Measured on the worker image: yt-dlp's proxy reaches the Requests and CurlCFFI handlers, native HLS and ffmpeg alike. Formats fetched over anything but http(s), which ffmpeg would dial directly, are dropped, and one factory opens every yt-dlp so none skips it.
This commit is contained in:
@@ -10,22 +10,12 @@ What this closes: a URL whose host resolves to a loopback / private / link-local
|
||||
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
|
||||
differently between the two moments, and the worker is where the socket is actually opened.
|
||||
|
||||
What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
|
||||
the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
|
||||
|
||||
* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
|
||||
with an internal address.
|
||||
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
|
||||
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
|
||||
open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
|
||||
response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
|
||||
the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
|
||||
class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
|
||||
one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
|
||||
kept). Tracked as SEC-17.
|
||||
|
||||
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
|
||||
hostname into the box), which is the reachable one.
|
||||
What this alone cannot see is every request yt-dlp makes AFTER the typed URL: a public host
|
||||
answering 3xx with an internal ``Location``, and DNS rebinding (a name that passes this check and
|
||||
then resolves inside for yt-dlp's own lookup). Those are closed one layer down, by `egress`: a guarded
|
||||
download's every connection goes through a gate that judges the address it actually dials, with
|
||||
this module's `is_forbidden`. The checks here stay as the early, structured refusal — a user typing
|
||||
``http://127.0.0.1:6379`` hears why at once instead of from a failed job.
|
||||
|
||||
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
|
||||
user-controlled destination to guard.
|
||||
@@ -62,7 +52,7 @@ class BlockedUrl(Exception):
|
||||
self.reason: UrlBlockedReason = reason
|
||||
|
||||
|
||||
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
def is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
||||
"""Every address family we must not open a connection to. Checked explicitly rather than
|
||||
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
|
||||
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
|
||||
@@ -113,5 +103,5 @@ def check_url(raw: str) -> None:
|
||||
ip = ipaddress.ip_address(addr)
|
||||
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
|
||||
raise BlockedUrl(REASON_PRIVATE) from None
|
||||
if _is_forbidden(ip):
|
||||
if is_forbidden(ip):
|
||||
raise BlockedUrl(REASON_PRIVATE)
|
||||
|
||||
Reference in New Issue
Block a user