peter
0cc450d1c9
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches it from the WORKER, so any approved user could aim the server at the compose network, the LAN or a cloud metadata endpoint — and read the result off job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a port scanner. downloads/urlguard.py refuses a URL whose host resolves to a loopback/private/link-local/reserved address, at enqueue AND again in the worker (DNS can move in between, and the worker is where the socket opens). A network-class failure on a `url` job now reports one fixed sentence instead of the raw transport text; the error CODE is still classified from the original, so the localized explanation is unchanged. DNS rebinding is explicitly NOT closed — that needs the connection gated inside yt-dlp — and the module says so rather than implying more than it does. SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video, so a URL that is purely a playlist or channel still expanded into every entry — one enqueue, unbounded downloads. `playlist_items` pins it to one. Size was unbounded too: `check_enqueue` compares the CURRENT footprint against the cap, and a download's size is unknown until it lands, so one job could overshoot by any amount; the owner's remaining budget is now handed to yt-dlp as `max_filesize`, applied AFTER the adapter merge so no source adapter can raise a limit the quota layer set. A join's cut-list is capped at 100: an accurate join emits two filters per segment into one ffmpeg filter_complex. SEC-14: the YouTube budget is instance-wide, so one user holding down a manual sync spent the day for everyone. Per-user (not per-IP — the cost belongs to the account) 3-per-10-minutes limiters on the two bulk pulls, subscriptions and playlists, with separate budgets so one does not close the other. The review also listed /rss, /backfill and /enrich; E3 deleted them, and search stays unlimited on purpose — it is interactive, not a bulk pull. Gate: pytest 896 -> 924 (none skipped), vitest 671.
88 lines
3.3 KiB
Python
88 lines
3.3 KiB
Python
"""SEC-12 · the download URL guard.
|
|
|
|
The Download Center hands a user-supplied URL to yt-dlp on the worker, so the server fetches
|
|
whatever the user typed. These pin that a URL resolving into our own network is refused, that the
|
|
refusal reasons stay distinct (an unresolvable host is not the same message as a blocked one), and
|
|
that a normal public URL still goes through.
|
|
|
|
DNS is stubbed rather than live: the point under test is the address JUDGEMENT, and a test that
|
|
resolves real names would be slow, offline-fragile, and would silently start passing if the guard
|
|
stopped resolving at all.
|
|
"""
|
|
|
|
import pytest
|
|
|
|
from app.downloads import urlguard
|
|
|
|
|
|
def _stub_dns(monkeypatch, addrs):
|
|
"""Make every lookup answer with `addrs` (a list of textual IPs)."""
|
|
|
|
def fake_getaddrinfo(host, port, *a, **kw):
|
|
return [(None, None, None, "", (addr, port or 0)) for addr in addrs]
|
|
|
|
monkeypatch.setattr(urlguard.socket, "getaddrinfo", fake_getaddrinfo)
|
|
|
|
|
|
def _blocked_reason(url):
|
|
with pytest.raises(urlguard.BlockedUrl) as e:
|
|
urlguard.check_url(url)
|
|
return e.value.reason
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"addr",
|
|
[
|
|
"127.0.0.1", # loopback
|
|
"10.1.2.3", # RFC1918
|
|
"192.168.0.5", # RFC1918
|
|
"172.16.9.9", # RFC1918
|
|
"169.254.169.254", # link-local — the cloud metadata endpoint
|
|
"0.0.0.0", # unspecified
|
|
"::1", # IPv6 loopback
|
|
"fe80::1", # IPv6 link-local
|
|
"fc00::1", # IPv6 unique-local
|
|
"::ffff:127.0.0.1", # loopback wearing an IPv6 hat
|
|
],
|
|
)
|
|
def test_addresses_on_our_own_network_are_refused(monkeypatch, addr):
|
|
_stub_dns(monkeypatch, [addr])
|
|
assert _blocked_reason("http://whatever.example/video") == urlguard.REASON_PRIVATE
|
|
|
|
|
|
def test_one_bad_answer_in_a_round_robin_blocks_the_whole_host(monkeypatch):
|
|
# A host answering with both a public and an internal address must not become a coin flip:
|
|
# yt-dlp would pick for itself, so the guard refuses if ANY answer is forbidden.
|
|
_stub_dns(monkeypatch, ["93.184.216.34", "127.0.0.1"])
|
|
assert _blocked_reason("https://mixed.example/v") == urlguard.REASON_PRIVATE
|
|
|
|
|
|
def test_a_public_address_passes(monkeypatch):
|
|
_stub_dns(monkeypatch, ["93.184.216.34"])
|
|
urlguard.check_url("https://example.com/watch?v=abc") # does not raise
|
|
|
|
|
|
def test_unresolvable_is_its_own_reason(monkeypatch):
|
|
import socket as _socket
|
|
|
|
def boom(*a, **kw):
|
|
raise _socket.gaierror("nope")
|
|
|
|
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
|
|
# Distinct from PRIVATE on purpose: "we couldn't look that up" and "we refuse to fetch that"
|
|
# are different things to tell a user, and the frontend renders them differently.
|
|
assert _blocked_reason("https://nx.example/v") == urlguard.REASON_UNRESOLVABLE
|
|
|
|
|
|
@pytest.mark.parametrize("url", ["file:///etc/passwd", "ftp://host/x", "gopher://host/1"])
|
|
def test_non_http_schemes_are_refused_before_any_lookup(monkeypatch, url):
|
|
def boom(*a, **kw): # a scheme rejection must not have resolved anything first
|
|
raise AssertionError("DNS was consulted for a non-http scheme")
|
|
|
|
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
|
|
assert _blocked_reason(url) == urlguard.REASON_SCHEME
|
|
|
|
|
|
def test_a_url_without_a_host_is_refused():
|
|
assert _blocked_reason("http:///just/a/path") == urlguard.REASON_NO_HOST
|