peter
fe38dfbcca
The deep review of the branch raised 47 findings, 18 of which survived adversarial verification. Grouped by what they actually were: **The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one user over quota was the only one downloading with no ceiling at all. `check_enqueue` could not catch it: it ran while the job was queued and the user was still under the cap. The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision lives, and where it can be tested), and the worker now fails the job outright rather than proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help until space is freed. **Silent failures in the new account panel.** All three actions went through `quiet: true` endpoints, which suppress the global error dialog on the promise that the caller reports inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On the broken-SMTP instance this panel exists for, an admin would read silence as "the mail went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`, shared with the pending-access queue, so the two copies of these mutations cannot drift apart again — which is how half of this was lost in the first place. **A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps back one migration, but a release routinely ships several (0.80.0 shipped four): it drops the newest one's columns and leaves the schema at a revision the older image also cannot resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is now the recommended path, and the alembic route asks the target version for its own head revision instead of counting backwards. Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as a 500; the pull cooldown extended to the sibling route spending the same QuotaAction; restore stops the app before dropping its tables; the PowerShell cleanup can no longer decide the script's verdict on 5.1, nor report the wrong exit code; the documented one-liners hardened to match the scripts; the registry's retention window stated where pinning is advertised; the two banners made mutually exclusive so neither can shadow the other; `hover:bg-hover` (a utility this project never generated) replaced by the `selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to the two callers that discarded it. Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset banner UAT'd on localdev.
106 lines
5.0 KiB
Python
106 lines
5.0 KiB
Python
"""Refuse a user-supplied download URL that points back into our own network (SEC-12).
|
|
|
|
The Download Center deliberately accepts an arbitrary ``http(s)`` URL — that is the feature — and
|
|
hands it to yt-dlp running on the WORKER. Without a guard, any approved user can make the server
|
|
fetch anything the server can reach: other containers on the compose network, the LAN, a cloud
|
|
metadata endpoint. The failure text then comes back to them in ``job.error``, so "connection
|
|
refused" vs "HTTP Error 401" vs "timed out" turns the download box into an internal port scanner.
|
|
|
|
What this closes: a URL whose host resolves to a loopback / private / link-local / reserved address
|
|
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
|
|
differently between the two moments, and the worker is where the socket is actually opened.
|
|
|
|
What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
|
|
the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
|
|
|
|
* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
|
|
with an internal address.
|
|
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
|
|
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
|
|
open redirect reaches it, so this is the cheaper of the two — but it still buys only what a
|
|
media-shaped internal response yields, since a non-media body fails the mp4 remux and stores
|
|
nothing. Tracked as SEC-17.
|
|
|
|
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
|
|
hostname into the box), which is the reachable one.
|
|
|
|
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
|
|
user-controlled destination to guard.
|
|
"""
|
|
|
|
import ipaddress
|
|
import socket
|
|
from urllib.parse import urlparse
|
|
|
|
# Reason codes travel to the client inside the structured 422 detail, so they are part of the API
|
|
# vocabulary (frontend `localizeDetail`) — keep them stable.
|
|
REASON_SCHEME = "scheme" # not http(s)
|
|
REASON_NO_HOST = "no_host" # URL carries no hostname
|
|
REASON_BAD_PORT = "bad_port" # ":99999" / ":abc" — urlparse only raises on ACCESS, so guard it
|
|
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
|
|
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
|
|
|
|
|
|
class BlockedUrl(Exception):
|
|
"""A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
|
|
REASON_* codes above."""
|
|
|
|
def __init__(self, reason: str):
|
|
super().__init__(reason)
|
|
self.reason = reason
|
|
|
|
|
|
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
|
|
"""Every address family we must not open a connection to. Checked explicitly rather than
|
|
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
|
|
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
|
|
mapped = getattr(ip, "ipv4_mapped", None)
|
|
if mapped is not None:
|
|
ip = mapped
|
|
return bool(
|
|
ip.is_private
|
|
or ip.is_loopback
|
|
or ip.is_link_local
|
|
or ip.is_multicast
|
|
or ip.is_reserved
|
|
or ip.is_unspecified
|
|
)
|
|
|
|
|
|
def _resolve(host: str, port: int | None) -> list[str]:
|
|
"""Every address ``host`` resolves to. Raises BlockedUrl(REASON_UNRESOLVABLE) when DNS has no
|
|
answer — reported as its own outcome, because "we could not look that up" and "we refuse to
|
|
fetch that" are different things to tell a user."""
|
|
try:
|
|
infos = socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
|
|
except socket.gaierror as exc:
|
|
raise BlockedUrl(REASON_UNRESOLVABLE) from exc
|
|
return [info[4][0] for info in infos]
|
|
|
|
|
|
def check_url(raw: str) -> None:
|
|
"""Raise BlockedUrl unless ``raw`` is an http(s) URL whose host resolves only to addresses the
|
|
server is allowed to fetch from. A host with even ONE forbidden address is refused: a
|
|
round-robin answer must not become a coin flip."""
|
|
u = urlparse((raw or "").strip())
|
|
if u.scheme not in ("http", "https"):
|
|
raise BlockedUrl(REASON_SCHEME)
|
|
host = u.hostname
|
|
if not host:
|
|
raise BlockedUrl(REASON_NO_HOST)
|
|
# `.port` is a lazily-parsed property: it raises ValueError for ":99999" or ":abc" on ACCESS,
|
|
# not at urlparse() time, and `.hostname` above happily returns first. Unguarded, one typo
|
|
# escapes the caller's `except BlockedUrl` as a 500 instead of the structured 422 this whole
|
|
# module exists to return.
|
|
try:
|
|
port = u.port
|
|
except ValueError as exc:
|
|
raise BlockedUrl(REASON_BAD_PORT) from exc
|
|
for addr in _resolve(host, port):
|
|
try:
|
|
ip = ipaddress.ip_address(addr)
|
|
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
|
|
raise BlockedUrl(REASON_PRIVATE) from None
|
|
if _is_forbidden(ip):
|
|
raise BlockedUrl(REASON_PRIVATE)
|