fix(downloads): a user's download is judged at every connection it makes, not only at the URL typed
The URL guard judged the address the user typed; a public host's 3xx into our network, a page's formats pointing inside, and a name rebound between the check and the fetch all got past it, and the stored error told a live internal port from a dead one. Every non-YouTube download now runs through an egress gate on the worker's loopback: it resolves each host once, refuses any forbidden answer before dialling, and connects to the address it vetted. Measured on the worker image: yt-dlp's proxy reaches the Requests and CurlCFFI handlers, native HLS and ffmpeg alike. Formats fetched over anything but http(s), which ffmpeg would dial directly, are dropped, and one factory opens every yt-dlp so none skips it.
This commit is contained in:
@@ -28,6 +28,7 @@ import urllib.parse
|
||||
from collections.abc import Iterator
|
||||
|
||||
from app import logs
|
||||
from app.downloads import egress as egress_gate
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
@@ -294,6 +295,8 @@ def build_ydl_opts(
|
||||
extra: dict | None = None,
|
||||
cookiefile: str | None = None,
|
||||
max_bytes: int | None = None,
|
||||
*,
|
||||
egress: bool,
|
||||
) -> dict:
|
||||
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
|
||||
|
||||
@@ -309,7 +312,11 @@ def build_ydl_opts(
|
||||
a file announced larger than it rather than filling the disk and only then being caught by the
|
||||
quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
|
||||
a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
|
||||
pipeline also holds every produced file to what is left by its real size."""
|
||||
pipeline also holds every produced file to what is left by its real size.
|
||||
|
||||
`egress` (SEC-17) routes every fetch through the worker's egress gate, which refuses a connection
|
||||
into our own network, and keeps only the formats fetched over http(s). Required, with no default:
|
||||
a new caller has to say which kind of download it builds (`SourceAdapter.egress_guarded`)."""
|
||||
s = normalize(spec)
|
||||
h = s["max_height"]
|
||||
opts: dict = {
|
||||
@@ -364,6 +371,13 @@ def build_ydl_opts(
|
||||
if cookiefile:
|
||||
opts["cookiefile"] = cookiefile
|
||||
|
||||
# SEC-17. AFTER the adapter merge, like the cookie: an adapter must not be able to point a
|
||||
# guarded download past the gate.
|
||||
if egress:
|
||||
opts["proxy"] = egress_gate.proxy_url()
|
||||
opts["hls_prefer_native"] = True
|
||||
opts[egress_gate.GUARDED] = True
|
||||
|
||||
if s["mode"] == "a":
|
||||
opts["format"] = "bestaudio/best"
|
||||
opts["postprocessors"].append(
|
||||
|
||||
Reference in New Issue
Block a user