fix(uploads): make both caps answerable, and stop holding a connection
The epic-end deep review found three defects in P1 and four tests that did not hold what they claimed. They land together because they share the upload route and its test module; splitting by hunk would have produced commits that do not individually run green. * The edge ceiling and the route's own cap are the SAME number at the default configuration, so the edge always answers first: a declared Content-Length over the cap is refused before a byte is read, and an undeclared body trips the middleware on the same chunk the route would have tripped on. The route's numbered 422 -- and the drain written for it last week -- are reachable only once an admin LOWERS the knob, which is the one shape the browser UAT tested. Measured: headroom does not fix this, because the declared-length check fires first for any body big enough to matter and the drain pushes the count back over the edge for the rest. So the edge's own refusal is made readable instead: `errors.tooLarge` localizes `too_large` the way `upload_too_large` was already localized, and the dialog refuses an over-cap file BEFORE sending it, off the `upload_max_bytes` that GET /api/downloads/usage now reports. Measured in Chrome against the API directly: that 413 arrives as `onload` with a readable body in 43 ms. * Reading the body behind the auth dependency put the whole transfer inside the request's database session -- the auth SELECT autobegins a transaction, so one of the twenty pooled connections sat `idle in transaction`, with nothing to time it out, for as long as the client cared to keep sending. That is the invariant db.py sizes the pool on and plex/playback.py records an incident for. The read transaction now ends before the await. Measured on the dev server: an 1818 ms upload shows no idle-in-transaction sample while the bytes move, only afterwards, when the handler actually writes. * The display name moved into a request header, where nothing bounds its length, while the column stayed varchar(255) -- so a long enough name was a DataError 500 after the whole file had already transferred. One `display_text` helper now serves all three writers (the metadata PATCH already truncated, the upload ingest did not), and the dialog stops at the same number. * The gate invariant asked `route.dependant.body_params`, which holds only the route's OWN callable's parameters: a body declared in a SUB-DEPENDENCY -- the upload route's own idiom -- leaves it empty while FastAPI still reads the body before auth. It asks `route.body_field` now, which is what FastAPI itself consults. The AST registry of body-reading call sites had also never followed `capped_stream_to`, so the one route the whole invariant exists for was the single site the drift guard could not see. Four tests that passed over their own mutation are fixed: the cap boundary (a `>` loosened to `>=` survived the entire suite), the percent-decoding (every filename the suite sends is ASCII, where quote() is the identity function), the display-name header (no test at all), and the staging cleanup (deleting the unlink left the suite green while every refused upload leaked its partial bytes onto the volume). Every test added or changed here was mutation-checked.
This commit is contained in:
@@ -74,10 +74,25 @@ DEFAULT_MAX_BODY_BYTES = 1024 * 1024
|
||||
# default, so listing it could only widen what it accepts.
|
||||
ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
|
||||
# Ingesting a local file. The body IS the file since P1 — no multipart envelope to allow for —
|
||||
# so this is exactly the number the route enforces. `sysconfig`'s `download_upload_max_bytes` is
|
||||
# bounded so it can never exceed it (see sysconfig.SPECS), which is what stops an admin from
|
||||
# raising the route's own cap past the edge and collecting 413s from a layer the admin page
|
||||
# never mentions.
|
||||
# so this ceiling and the route's own cap count exactly the same bytes. `sysconfig`'s
|
||||
# `download_upload_max_bytes` is bounded by this same constant (see sysconfig.SPECS), so an
|
||||
# admin can only ever LOWER the route's cap below the edge, never raise it past one.
|
||||
#
|
||||
# WHICH LAYER ANSWERS, measured rather than assumed (deep-review round 1):
|
||||
# * knob LOWERED (an override below the default) — the route's cap is the smaller number, so
|
||||
# `capped_stream_to` trips first and answers 422 `upload_too_large` carrying the admin's own
|
||||
# figure, after draining the rest of the body so the client can read it.
|
||||
# * knob at the DEFAULT — the two numbers are EQUAL, and the edge always wins: a declared
|
||||
# Content-Length over the cap is refused here before a byte is read, and an undeclared body
|
||||
# trips this counter on the same chunk the route would have tripped on. The route's 422 is
|
||||
# unreachable in that configuration, and no amount of headroom would change it — the
|
||||
# declared-length check fires first for any body big enough to matter, and the route's drain
|
||||
# pushes the count past the edge for the rest.
|
||||
#
|
||||
# So this 413 is a refusal a real user meets on a default install, not a theoretical edge, and
|
||||
# it has to be as readable as the route's own: the client localizes `too_large` (`errors.tooLarge`)
|
||||
# exactly as it does `upload_too_large`, and the upload dialog refuses an over-cap file before
|
||||
# sending it at all, off the `upload_max_bytes` that `GET /api/downloads/usage` reports.
|
||||
("POST", "/api/downloads/upload"): settings.download_upload_max_bytes,
|
||||
("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
|
||||
}
|
||||
|
||||
@@ -31,6 +31,26 @@ class EditError(Exception):
|
||||
super().__init__(reason)
|
||||
|
||||
|
||||
# The width of every user-supplied display string on a job (`DownloadJob.display_name`,
|
||||
# `display_uploader`): `String(255)`. `test_upload.py` holds this against the columns
|
||||
# themselves so the two cannot drift — a number in a comment is exactly how that drift happens.
|
||||
DISPLAY_TEXT_MAX = 255
|
||||
|
||||
|
||||
def display_text(raw: str | None) -> str | None:
|
||||
"""A user-supplied display string on its way into one of those columns: stripped, truncated,
|
||||
blank → None.
|
||||
|
||||
One definition because there are three writers and they had drifted: the metadata PATCH
|
||||
truncated, the upload ingest did not, and since P1 the upload's name arrives in a REQUEST HEADER
|
||||
where nothing bounds its length — so a long enough one reached Postgres as a `DataError`, i.e. a
|
||||
500 after the whole file had already been transferred. Truncating is the right answer rather
|
||||
than refusing: the name is a label on a file the user has already sent, and losing the tail of it
|
||||
is not worth failing the upload over.
|
||||
"""
|
||||
return (raw or "").strip()[:DISPLAY_TEXT_MAX] or None
|
||||
|
||||
|
||||
def source_url(source_kind: str, source_ref: str) -> str:
|
||||
"""The URL yt-dlp should fetch. YouTube stores the 11-char id (the adapter reconstructs the watch
|
||||
URL); ad-hoc sources store the URL verbatim."""
|
||||
|
||||
@@ -99,7 +99,7 @@ def ingest_upload(
|
||||
source_kind="upload",
|
||||
source_ref=ref,
|
||||
profile_snapshot={},
|
||||
display_name=(display_name or "").strip() or None,
|
||||
display_name=service.display_text(display_name),
|
||||
status="done",
|
||||
progress=100,
|
||||
queue_pos=service.next_queue_pos(db),
|
||||
|
||||
@@ -462,6 +462,18 @@ async def spool_upload(
|
||||
MOVES it into place on success (the unlink is then a no-op), and on any other exit — cap hit,
|
||||
empty, quota, probe or move error, a client that vanished mid-upload — it is removed."""
|
||||
cap = sysconfig.get_int(db, "download_upload_max_bytes")
|
||||
user_id = user.id
|
||||
# Hand the pooled connection back BEFORE the transfer starts. Resolving `user` and reading the
|
||||
# cap were both SELECTs, and SQLAlchemy 2.0 autobegins a transaction on the first one — so
|
||||
# without this the session sits `idle in transaction` holding one of the twenty pooled
|
||||
# connections for as long as the client cares to keep sending. That is the invariant `db.py`
|
||||
# sizes the pool on, and `plex/playback.py` records the production incident that taught it;
|
||||
# reading the body here instead of letting FastAPI spool it (P1) is what put this route at risk
|
||||
# of breaking it. Auth only reads, so ending the read transaction loses nothing, and the
|
||||
# handler's first query autobegins a fresh one — one ROLLBACK round-trip against a minute of
|
||||
# held connection. Everything this dependency still needs is already an int in hand: touching
|
||||
# an ORM attribute below would reload `user` and check a connection straight back out.
|
||||
db.rollback()
|
||||
staging = Path(settings.download_root) / ".staging" / "uploads"
|
||||
staging.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp = tempfile.mkstemp(dir=str(staging))
|
||||
@@ -477,9 +489,9 @@ async def spool_upload(
|
||||
yield SpooledUpload(
|
||||
path=tmp_path,
|
||||
size=size,
|
||||
user_id=user.id,
|
||||
user_id=user_id,
|
||||
filename=_header_text(request, UPLOAD_FILENAME_HEADER) or "upload",
|
||||
display_name=_header_text(request, UPLOAD_DISPLAY_NAME_HEADER) or None,
|
||||
display_name=service.display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER)),
|
||||
)
|
||||
finally:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
@@ -570,7 +582,20 @@ def list_downloads(
|
||||
|
||||
@router.get("/usage")
|
||||
def my_usage(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
|
||||
return quota.usage(db, user.id)
|
||||
"""The caller's download quota, plus the per-file upload cap.
|
||||
|
||||
The cap rides here rather than in `quota.usage` because it is not a quota value — it is a
|
||||
`sysconfig` knob about one route — and this is the route layer, where composing the two is the
|
||||
job. The upload dialog needs it to refuse an over-cap file BEFORE sending it: both server-side
|
||||
caps answer too late to be pleasant (the edge ceiling refuses the request outright, see
|
||||
`bodylimit.ROUTE_MAX_BODY_BYTES`), and making the user push 600 MB up the wire to be told the
|
||||
limit is 500 is a bad answer even when it is a correct one. It is a convenience, not the
|
||||
boundary: the caps stay where they are and still refuse a client that ignores this.
|
||||
"""
|
||||
return {
|
||||
**quota.usage(db, user.id),
|
||||
"upload_max_bytes": sysconfig.get_int(db, "download_upload_max_bytes"),
|
||||
}
|
||||
|
||||
|
||||
# Priority when a source has several jobs (e.g. a done one plus a fresh queued one).
|
||||
@@ -665,9 +690,9 @@ def update_download_meta(
|
||||
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
|
||||
sent = body.model_fields_set
|
||||
if "display_name" in sent:
|
||||
job.display_name = ((body.display_name or "").strip())[:255] or None
|
||||
job.display_name = service.display_text(body.display_name)
|
||||
if "display_uploader" in sent:
|
||||
job.display_uploader = ((body.display_uploader or "").strip())[:255] or None
|
||||
job.display_uploader = service.display_text(body.display_uploader)
|
||||
if "display_uploader_url" in sent:
|
||||
job.display_uploader_url = _clean_url(body.display_uploader_url)
|
||||
if "extra_links" in sent:
|
||||
|
||||
@@ -137,13 +137,18 @@ SPECS: tuple[ConfigSpec, ...] = (
|
||||
min=1,
|
||||
max=100_000,
|
||||
),
|
||||
# min=1, NOT 0: the per-file cap is enforced as `if cap and size > cap`, so a stored 0 would read
|
||||
# as falsy and SILENTLY disable the limit (an unbounded streamed write to the volume). Floor it at
|
||||
# 1 so the cap is always active; the overall footprint is separately bounded by the byte quota.
|
||||
# `max` is not cosmetic: `bodylimit.ROUTE_MAX_BODY_BYTES` refuses an upload body over roughly
|
||||
# min=1, NOT 0: a stored 0 used to read as "no per-file cap" (the cap was enforced as
|
||||
# `if cap and size > cap`, so a falsy value disabled it). Since P1 the route passes this number
|
||||
# straight to `capped_stream_to`, where 0 would refuse every upload instead — the opposite
|
||||
# failure, and the reason `config.download_upload_max_bytes` carries its own `ge=1`. Either way
|
||||
# a stored 0 is wrong, so it is floored at 1; the overall footprint is separately bounded by the
|
||||
# byte quota.
|
||||
# `max` is not cosmetic: `bodylimit.ROUTE_MAX_BODY_BYTES` refuses an upload body over exactly
|
||||
# this number at the edge, before the route's own check ever sees it. An admin allowed to raise
|
||||
# the knob past the edge would get 413s from a layer the admin page never mentions, so the two
|
||||
# are pinned together here and `tests/test_body_limit.py` holds them that way. `_coerce` also
|
||||
# are pinned together here and `tests/test_body_limit.py` holds them that way. Below the edge is
|
||||
# the useful direction and the one the knob exists for: a lowered knob is what makes the route's
|
||||
# own 422 — the one carrying this figure — the answer the user gets. `_coerce` also
|
||||
# REJECTS a stored value outside these bounds on read (falling back to the env default, with a
|
||||
# warning) — neither bound has always been what it is now, so a legal-when-written override can
|
||||
# be sitting in the database already.
|
||||
|
||||
@@ -204,23 +204,51 @@ def test_the_default_ceiling_clears_the_largest_body_a_default_route_takes():
|
||||
)
|
||||
|
||||
|
||||
# Every `capped_body` call site in the routers, as (module, limit constant). A route that reads its
|
||||
# own body declares the size it MEANS to accept; if that is above the global default and its route
|
||||
# is not in ROUTE_MAX_BODY_BYTES, this middleware refuses a request the route would have taken —
|
||||
# silently, in production, on a path nobody tested. The point of the list is the third entry.
|
||||
# Every body-reading call site in the routers, as (module, function, limit expression). A route that
|
||||
# reads its own body declares the size it MEANS to accept; if that is above the global default and
|
||||
# its route is not in ROUTE_MAX_BODY_BYTES, this middleware refuses a request the route would have
|
||||
# taken — silently, in production, on a path nobody tested. The point of the list is the third entry.
|
||||
#
|
||||
# BOTH helpers, not just `capped_body`: P1 added `capped_stream_to` as its sibling and the registry
|
||||
# did not follow, so the upload route — the one this whole invariant was written for — was the one
|
||||
# site the drift guard could not see (deep-review round 1). The names come from `_common.py` itself
|
||||
# below, so a THIRD sibling cannot be added without this list noticing either.
|
||||
CAPPED_BODY_SITES = {
|
||||
("diagnostics.py", "BODY_MAX_BYTES"),
|
||||
("debug_sessions.py", "debugsession.SCREENSHOT_MAX_BYTES"),
|
||||
("diagnostics.py", "capped_body", "BODY_MAX_BYTES"),
|
||||
("debug_sessions.py", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
|
||||
("downloads.py", "capped_stream_to", "cap"),
|
||||
}
|
||||
|
||||
|
||||
def test_every_capped_body_site_is_known_and_fits_under_its_ceiling():
|
||||
def _body_readers() -> set[str]:
|
||||
"""The names of the body-reading helpers `_common.py` exports — asked of the module rather than
|
||||
listed here, so adding a fourth one cannot quietly leave the registry below behind."""
|
||||
common = pathlib.Path(__file__).resolve().parents[1] / "app" / "routes" / "_common.py"
|
||||
tree = ast.parse(common.read_text("utf-8"), filename=str(common))
|
||||
return {
|
||||
node.name
|
||||
for node in tree.body
|
||||
if isinstance(node, ast.AsyncFunctionDef) and node.name.startswith("capped_")
|
||||
}
|
||||
|
||||
|
||||
def test_the_registry_covers_every_body_reading_helper():
|
||||
"""The mutation guard for the test below: if a helper were renamed or added, the site list would
|
||||
still match itself while the new one went uncovered."""
|
||||
assert _body_readers() == {"capped_body", "capped_stream_to"}, (
|
||||
"routes/_common.py grew or renamed a body-reading helper — add its call sites to "
|
||||
"CAPPED_BODY_SITES and this expectation."
|
||||
)
|
||||
|
||||
|
||||
def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
|
||||
"""Parsed, not grepped. A regex over the call text has to guess how the arguments are spelled,
|
||||
and the shapes it guesses wrong — a bare `4 * 1024 * 1024`, a parameter not named `request` —
|
||||
are the natural ways the NEXT site gets written, so a miss reads as "no new sites" rather than
|
||||
as a failure. `ast` sees every call whatever it looks like."""
|
||||
readers = _body_readers()
|
||||
routes_dir = pathlib.Path(__file__).resolve().parents[1] / "app" / "routes"
|
||||
found: set[tuple[str, str]] = set()
|
||||
found: set[tuple[str, str, str]] = set()
|
||||
for py in routes_dir.rglob("*.py"):
|
||||
tree = ast.parse(py.read_text("utf-8"), filename=str(py))
|
||||
for node in ast.walk(tree):
|
||||
@@ -228,22 +256,27 @@ def test_every_capped_body_site_is_known_and_fits_under_its_ceiling():
|
||||
continue
|
||||
fn = node.func
|
||||
name = fn.attr if isinstance(fn, ast.Attribute) else getattr(fn, "id", None)
|
||||
if name != "capped_body":
|
||||
if name not in readers:
|
||||
continue
|
||||
limit = next(
|
||||
(kw.value for kw in node.keywords if kw.arg == "limit"),
|
||||
node.args[1] if len(node.args) > 1 else None,
|
||||
node.args[2] if name == "capped_stream_to" and len(node.args) > 2 else None,
|
||||
)
|
||||
found.add((py.name, ast.unparse(limit) if limit is not None else "<none>"))
|
||||
if limit is None and name != "capped_stream_to":
|
||||
limit = node.args[1] if len(node.args) > 1 else None
|
||||
found.add((py.name, name, ast.unparse(limit) if limit is not None else "<none>"))
|
||||
assert found == CAPPED_BODY_SITES, (
|
||||
f"a capped_body call site appeared, moved or changed shape: {found ^ CAPPED_BODY_SITES}. "
|
||||
f"a body-reading call site appeared, moved or changed shape: {found ^ CAPPED_BODY_SITES}. "
|
||||
"Each one declares a body size; check it against bodylimit.DEFAULT_MAX_BODY_BYTES and, if "
|
||||
"it is larger, add its route to bodylimit.ROUTE_MAX_BODY_BYTES — then update this list."
|
||||
)
|
||||
# And the two known ones really do satisfy the rule.
|
||||
# And the known ones really do satisfy the rule.
|
||||
assert diagnostics_routes.BODY_MAX_BYTES <= DEFAULT_MAX_BODY_BYTES
|
||||
assert debugsession.SCREENSHOT_MAX_BYTES > DEFAULT_MAX_BODY_BYTES
|
||||
assert ("POST", "/api/debug-sessions/{sid}/screenshots") in bodylimit.ROUTE_MAX_BODY_BYTES
|
||||
# The upload site's limit is the sysconfig knob, which is bounded BY the edge ceiling rather
|
||||
# than being a constant to compare against — that pairing is its own test below.
|
||||
assert ("POST", "/api/downloads/upload") in bodylimit.ROUTE_MAX_BODY_BYTES
|
||||
|
||||
|
||||
def test_the_upload_ceiling_and_the_admin_knob_cannot_drift_apart():
|
||||
@@ -254,6 +287,12 @@ def test_the_upload_ceiling_and_the_admin_knob_cannot_drift_apart():
|
||||
Since P1 the two count the SAME bytes — the body is the file, with no multipart envelope
|
||||
wrapped round it — so the old allowance is gone and the edge is the knob's own maximum,
|
||||
exactly. A gap reappearing means someone re-wrapped the upload in an envelope without saying so.
|
||||
|
||||
Equality also means the EDGE is what answers on a default install, where no override has lowered
|
||||
the knob: the declared-length check refuses before a byte is read. That is not a gap to close
|
||||
with headroom (measured: headroom only moves the boundary by its own size, and the route's drain
|
||||
pushes the count back over it) — it is a refusal the client has to be able to read, which is why
|
||||
`errors.tooLarge` exists and why the dialog checks the file against `upload_max_bytes` first.
|
||||
"""
|
||||
spec = next(s for s in sysconfig.SPECS if s.key == "download_upload_max_bytes")
|
||||
edge = bodylimit.ROUTE_MAX_BODY_BYTES[("POST", "/api/downloads/upload")]
|
||||
@@ -288,19 +327,28 @@ def test_no_route_with_a_raised_ceiling_declares_a_body_parameter(method, templa
|
||||
closed on the upload route, which sat in `ROUTE_MAX_BODY_BYTES` with a 500 MiB ceiling and a
|
||||
`file: UploadFile = File(...)` body for one release.
|
||||
|
||||
Asked of FastAPI itself (`route.dependant.body_params`) rather than parsed or grepped: the
|
||||
router already knows which parameters it will read the body for, and a shape a regex or an AST
|
||||
walk guesses wrong is precisely how the next route slips through."""
|
||||
Asked of FastAPI itself rather than parsed or grepped: the router already knows which parameters
|
||||
it will read the body for, and a shape a regex or an AST walk guesses wrong is precisely how the
|
||||
next route slips through.
|
||||
|
||||
`route.body_field`, NOT `route.dependant.body_params`, and the difference is the whole test.
|
||||
`body_params` holds only the parameters of the route's OWN callable; `body_field` is built from
|
||||
`_get_flat_body_params`, which recurses into sub-dependencies — and that is also what FastAPI
|
||||
itself consults when it decides to read the body. Measured on the pinned FastAPI: a body declared
|
||||
inside a sub-dependency leaves `body_params` EMPTY while the body is still read in full before
|
||||
auth, so the first version of this test was green over a reproduction of the very defect it
|
||||
exists to forbid (deep-review round 1). The upload route's own idiom — a body read from a
|
||||
`yield` dependency — is exactly the shape that would have slipped through."""
|
||||
from app.main import app
|
||||
|
||||
route = _route_objects(app).get((method, template))
|
||||
assert route is not None, f"{method} {template} is not a route any more"
|
||||
body_params = [p.name for p in getattr(route.dependant, "body_params", ())]
|
||||
assert not body_params, (
|
||||
f"{method} {template} declares a body parameter ({', '.join(body_params)}), so FastAPI "
|
||||
"spools it before the first dependency runs and the raised ceiling is all that stands "
|
||||
"between an anonymous caller and the whole allowance. Read the body yourself, from a "
|
||||
"dependency declared AFTER the auth one — see routes/downloads.py:spool_upload."
|
||||
assert route.body_field is None, (
|
||||
f"{method} {template} declares a body ({route.body_field.name}) on the route or in one of "
|
||||
"its dependencies, so FastAPI reads it before the first dependency runs and the raised "
|
||||
"ceiling is all that stands between an anonymous caller and the whole allowance. Read the "
|
||||
"body yourself, from a dependency declared AFTER the auth one — see "
|
||||
"routes/downloads.py:spool_upload."
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -21,10 +21,12 @@ from fastapi import HTTPException
|
||||
|
||||
from app.config import settings
|
||||
from app.routes._common import capped_stream_to
|
||||
from app.downloads import mediakind, quota, storage
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.downloads import mediakind, quota, service, storage
|
||||
from app.downloads import probe as probemod
|
||||
from app.downloads import upload as uploadmod
|
||||
from app.models import DownloadQuota, MediaAsset
|
||||
from app.models import DownloadJob, DownloadQuota, MediaAsset
|
||||
|
||||
|
||||
# --- pure: the kind invariant ---------------------------------------------------------------
|
||||
@@ -369,6 +371,149 @@ def test_upload_route_enforces_per_file_cap(
|
||||
assert r.json()["detail"]["code"] == "upload_too_large"
|
||||
|
||||
|
||||
def test_the_cap_admits_a_body_of_EXACTLY_the_cap_and_refuses_one_byte_more(
|
||||
as_user, make_user, db, _root, monkeypatch, upload_headers
|
||||
):
|
||||
"""The boundary, which the test above cannot see: it sends 16 bytes against a cap of 5, so a
|
||||
`>` silently loosened to `>=` refuses it just the same and the mutation survives the whole
|
||||
suite. The cap is `size > limit`, so a file of exactly the cap must be ACCEPTED.
|
||||
"""
|
||||
from app import sysconfig
|
||||
|
||||
_probe(monkeypatch, (None, None, None, None, None))
|
||||
payload = b"x" * 64
|
||||
sysconfig.set_value(db, "download_upload_max_bytes", len(payload))
|
||||
db.commit()
|
||||
c = as_user(make_user())
|
||||
|
||||
at_the_cap = c.post(
|
||||
"/api/downloads/upload", content=payload, headers=upload_headers("exact.bin")
|
||||
)
|
||||
assert at_the_cap.status_code == 200, at_the_cap.text
|
||||
|
||||
over = c.post(
|
||||
"/api/downloads/upload", content=payload + b"x", headers=upload_headers("over.bin")
|
||||
)
|
||||
assert over.status_code == 422
|
||||
assert over.json()["detail"] == {"code": "upload_too_large", "limit": len(payload)}
|
||||
|
||||
|
||||
def test_a_non_ascii_filename_survives_the_header_round_trip(
|
||||
as_user, make_user, db, _root, monkeypatch, upload_headers
|
||||
):
|
||||
"""The percent-decoding, which every other test in this file leaves vacuous.
|
||||
|
||||
Headers are latin-1 on the wire, so the client `encodeURIComponent`s the name and the route
|
||||
`unquote`s it. Every filename the suite sends is pure ASCII — and `quote()` is the IDENTITY
|
||||
function on ASCII — so deleting the `unquote` call changes nothing anywhere else. A Hungarian
|
||||
name is the only input that can tell the two apart: without the decode the sanitizer would be
|
||||
handed the literal `nyaral%C3%A1s.bin` and would keep the percent-escapes in the stored name.
|
||||
"""
|
||||
_probe(monkeypatch, (None, None, None, None, None))
|
||||
c = as_user(make_user())
|
||||
r = c.post(
|
||||
"/api/downloads/upload", content=b"data", headers=upload_headers("nyaralás ősszel.bin")
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
asset = db.execute(select(MediaAsset).order_by(MediaAsset.id.desc())).scalars().first()
|
||||
assert asset is not None
|
||||
stored = Path(asset.rel_path).name
|
||||
assert "%" not in stored and "c3" not in stored.lower(), stored
|
||||
assert stored.startswith("nyaralas_osszel"), stored
|
||||
|
||||
|
||||
def test_the_display_text_bound_matches_the_columns_it_guards():
|
||||
"""A number living in two places is how the `cancelled`/`canceled` bug happened. `display_text`
|
||||
truncates to `DISPLAY_TEXT_MAX`; the columns it feeds are `String(255)`. Held against the model
|
||||
itself, so widening one without the other fails here rather than as a DataError in production."""
|
||||
for name in ("display_name", "display_uploader"):
|
||||
assert DownloadJob.__table__.c[name].type.length == service.DISPLAY_TEXT_MAX
|
||||
|
||||
|
||||
def test_the_display_name_header_reaches_the_job_and_is_bounded(
|
||||
as_user, make_user, _root, monkeypatch, upload_headers
|
||||
):
|
||||
"""The second header, which had no backend test at all — and the length nothing bounded.
|
||||
|
||||
Since P1 the display name arrives in a REQUEST HEADER rather than as a form field, so the only
|
||||
limit on it was the header size. The column is varchar(255), so a longer one reached Postgres as
|
||||
a DataError: a 500 AFTER the whole file had been transferred. It is truncated now, by the same
|
||||
helper the metadata PATCH uses.
|
||||
"""
|
||||
_probe(monkeypatch, (None, None, None, None, None))
|
||||
c = as_user(make_user())
|
||||
|
||||
ok = c.post(
|
||||
"/api/downloads/upload",
|
||||
content=b"data",
|
||||
headers=upload_headers("clip.bin", "Nyári vakáció"),
|
||||
)
|
||||
assert ok.status_code == 200, ok.text
|
||||
assert ok.json()["display_name"] == "Nyári vakáció"
|
||||
|
||||
long_name = "n" * 300
|
||||
bounded = c.post(
|
||||
"/api/downloads/upload", content=b"data2", headers=upload_headers("clip2.bin", long_name)
|
||||
)
|
||||
assert bounded.status_code == 200, bounded.text
|
||||
assert bounded.json()["display_name"] == "n" * service.DISPLAY_TEXT_MAX
|
||||
|
||||
|
||||
def test_the_upload_holds_no_db_transaction_while_the_body_streams(
|
||||
as_user, make_user, db, _root, monkeypatch, upload_headers
|
||||
):
|
||||
"""P1 moved the body read behind the auth dependency, which put the whole transfer INSIDE the
|
||||
request's database session: resolving the caller and reading the cap are SELECTs, SQLAlchemy 2.0
|
||||
autobegins on the first one, and the session then holds one of the twenty pooled connections —
|
||||
`idle in transaction` to Postgres, with no `idle_in_transaction_session_timeout` to end it — for
|
||||
as long as the client keeps sending. `db.py` sizes the pool on that not happening and
|
||||
`plex/playback.py` records the production incident behind the rule.
|
||||
|
||||
Asserted on the session rather than on `engine.pool.checkedout()`: the transaction is the cause,
|
||||
the checkout is its consequence (measured), and the pool counter is shared with whatever else
|
||||
the test process is holding.
|
||||
"""
|
||||
from app.routes import downloads as downloads_routes
|
||||
|
||||
_probe(monkeypatch, (None, None, None, None, None))
|
||||
seen: dict[str, bool] = {}
|
||||
real = downloads_routes.capped_stream_to
|
||||
|
||||
async def spy(request, out, limit, refusal):
|
||||
seen["in_transaction"] = bool(db.in_transaction())
|
||||
return await real(request, out, limit, refusal)
|
||||
|
||||
monkeypatch.setattr(downloads_routes, "capped_stream_to", spy)
|
||||
c = as_user(make_user())
|
||||
r = c.post("/api/downloads/upload", content=b"data", headers=upload_headers("held.bin"))
|
||||
assert r.status_code == 200, r.text
|
||||
assert seen["in_transaction"] is False, (
|
||||
"the request's session was inside a transaction when the transfer began, so a pooled "
|
||||
"connection is pinned for the whole upload"
|
||||
)
|
||||
|
||||
|
||||
def test_a_refused_upload_leaves_nothing_in_the_staging_directory(
|
||||
as_user, make_user, db, _root, monkeypatch, upload_headers
|
||||
):
|
||||
"""`spool_upload` writes the body to a staging temp file as it arrives, so a refusal happens
|
||||
with a partial file already on disk — its `finally` is what removes it. Nothing asserted that:
|
||||
deleting the unlink left the whole suite green while every refused upload leaked its partial
|
||||
bytes onto the volume, one file per attempt, with no job to ever clean them up.
|
||||
"""
|
||||
from app import sysconfig
|
||||
|
||||
sysconfig.set_value(db, "download_upload_max_bytes", 8)
|
||||
db.commit()
|
||||
staging = Path(settings.download_root) / ".staging" / "uploads"
|
||||
c = as_user(make_user())
|
||||
|
||||
r = c.post("/api/downloads/upload", content=b"x" * 4096, headers=upload_headers("leaky.bin"))
|
||||
assert r.status_code == 422
|
||||
assert staging.exists(), "the dependency did not get as far as creating the staging dir"
|
||||
assert list(staging.iterdir()) == [], "a refused upload left its partial file behind"
|
||||
|
||||
|
||||
def test_upload_cap_cannot_be_disabled_with_zero(db):
|
||||
"""download_upload_max_bytes floors at 1, and set_value must enforce it.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user