fix(downloads): a user's download is judged at every connection it makes, not only at the URL typed

The URL guard judged the address the user typed; a public host's 3xx into our
network, a page's formats pointing inside, and a name rebound between the check
and the fetch all got past it, and the stored error told a live internal port
from a dead one. Every non-YouTube download now runs through an egress gate on
the worker's loopback: it resolves each host once, refuses any forbidden answer
before dialling, and connects to the address it vetted. Measured on the worker
image: yt-dlp's proxy reaches the Requests and CurlCFFI handlers, native HLS and
ffmpeg alike. Formats fetched over anything but http(s), which ffmpeg would dial
directly, are dropped, and one factory opens every yt-dlp so none skips it.
This commit is contained in:
2026-11-06 04:54:56 +01:00
parent 995a493bd3
commit 4e4fff87d4
14 changed files with 668 additions and 65 deletions
+15 -1
View File
@@ -28,6 +28,7 @@ import urllib.parse
from collections.abc import Iterator
from app import logs
from app.downloads import egress as egress_gate
log = logs.get_logger(__name__)
@@ -294,6 +295,8 @@ def build_ydl_opts(
extra: dict | None = None,
cookiefile: str | None = None,
max_bytes: int | None = None,
*,
egress: bool,
) -> dict:
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
@@ -309,7 +312,11 @@ def build_ydl_opts(
a file announced larger than it rather than filling the disk and only then being caught by the
quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
pipeline also holds every produced file to what is left by its real size."""
pipeline also holds every produced file to what is left by its real size.
`egress` (SEC-17) routes every fetch through the worker's egress gate, which refuses a connection
into our own network, and keeps only the formats fetched over http(s). Required, with no default:
a new caller has to say which kind of download it builds (`SourceAdapter.egress_guarded`)."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
@@ -364,6 +371,13 @@ def build_ydl_opts(
if cookiefile:
opts["cookiefile"] = cookiefile
# SEC-17. AFTER the adapter merge, like the cookie: an adapter must not be able to point a
# guarded download past the gate.
if egress:
opts["proxy"] = egress_gate.proxy_url()
opts["hls_prefer_native"] = True
opts[egress_gate.GUARDED] = True
if s["mode"] == "a":
opts["format"] = "bestaudio/best"
opts["postprocessors"].append(