peter/siftlode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(uploads): make both caps answerable, and stop holding a connection

Browse Source 
The epic-end deep review found three defects in P1 and four tests that did not
hold what they claimed. They land together because they share the upload route
and its test module; splitting by hunk would have produced commits that do not
individually run green.

* The edge ceiling and the route's own cap are the SAME number at the default
  configuration, so the edge always answers first: a declared Content-Length
  over the cap is refused before a byte is read, and an undeclared body trips
  the middleware on the same chunk the route would have tripped on. The route's
  numbered 422 -- and the drain written for it last week -- are reachable only
  once an admin LOWERS the knob, which is the one shape the browser UAT tested.
  Measured: headroom does not fix this, because the declared-length check fires
  first for any body big enough to matter and the drain pushes the count back
  over the edge for the rest. So the edge's own refusal is made readable
  instead: `errors.tooLarge` localizes `too_large` the way `upload_too_large`
  was already localized, and the dialog refuses an over-cap file BEFORE sending
  it, off the `upload_max_bytes` that GET /api/downloads/usage now reports.
  Measured in Chrome against the API directly: that 413 arrives as `onload`
  with a readable body in 43 ms.

* Reading the body behind the auth dependency put the whole transfer inside the
  request's database session -- the auth SELECT autobegins a transaction, so one
  of the twenty pooled connections sat `idle in transaction`, with nothing to
  time it out, for as long as the client cared to keep sending. That is the
  invariant db.py sizes the pool on and plex/playback.py records an incident
  for. The read transaction now ends before the await. Measured on the dev
  server: an 1818 ms upload shows no idle-in-transaction sample while the bytes
  move, only afterwards, when the handler actually writes.

* The display name moved into a request header, where nothing bounds its length,
  while the column stayed varchar(255) -- so a long enough name was a DataError
  500 after the whole file had already transferred. One `display_text` helper
  now serves all three writers (the metadata PATCH already truncated, the upload
  ingest did not), and the dialog stops at the same number.

* The gate invariant asked `route.dependant.body_params`, which holds only the
  route's OWN callable's parameters: a body declared in a SUB-DEPENDENCY -- the
  upload route's own idiom -- leaves it empty while FastAPI still reads the body
  before auth. It asks `route.body_field` now, which is what FastAPI itself
  consults. The AST registry of body-reading call sites had also never followed
  `capped_stream_to`, so the one route the whole invariant exists for was the
  single site the drift guard could not see.

Four tests that passed over their own mutation are fixed: the cap boundary (a
`>` loosened to `>=` survived the entire suite), the percent-decoding (every
filename the suite sends is ASCII, where quote() is the identity function), the
display-name header (no test at all), and the staging cleanup (deleting the
unlink left the suite green while every refused upload leaked its partial bytes
onto the volume). Every test added or changed here was mutation-checked.
This commit is contained in:
peter
2026-09-22 22:41:22 +02:00
parent 3d7ef9e945
commit efd7b217ca
14 changed files with 428 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tests/test_body_limit.py
+70 -22
View File
@@ -204,23 +204,51 @@ def test_the_default_ceiling_clears_the_largest_body_a_default_route_takes():
)
# Every `capped_body` call site in the routers, as (module, limit constant). A route that reads its
# own body declares the size it MEANS to accept; if that is above the global default and its route
# is not in ROUTE_MAX_BODY_BYTES, this middleware refuses a request the route would have taken —
# silently, in production, on a path nobody tested. The point of the list is the third entry.
# Every body-reading call site in the routers, as (module, function, limit expression). A route that
# reads its own body declares the size it MEANS to accept; if that is above the global default and
# its route is not in ROUTE_MAX_BODY_BYTES, this middleware refuses a request the route would have
# taken — silently, in production, on a path nobody tested. The point of the list is the third entry.
#
# BOTH helpers, not just `capped_body`: P1 added `capped_stream_to` as its sibling and the registry
# did not follow, so the upload route — the one this whole invariant was written for — was the one
# site the drift guard could not see (deep-review round 1). The names come from `_common.py` itself
# below, so a THIRD sibling cannot be added without this list noticing either.
CAPPED_BODY_SITES = {
("diagnostics.py", "BODY_MAX_BYTES"),
("debug_sessions.py", "debugsession.SCREENSHOT_MAX_BYTES"),
("diagnostics.py", "capped_body", "BODY_MAX_BYTES"),
("debug_sessions.py", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
("downloads.py", "capped_stream_to", "cap"),
}
def test_every_capped_body_site_is_known_and_fits_under_its_ceiling():
def _body_readers() -> set[str]:
"""The names of the body-reading helpers `_common.py` exports — asked of the module rather than
listed here, so adding a fourth one cannot quietly leave the registry below behind."""
common = pathlib.Path(__file__).resolve().parents[1] / "app" / "routes" / "_common.py"
tree = ast.parse(common.read_text("utf-8"), filename=str(common))
return {
node.name
for node in tree.body
if isinstance(node, ast.AsyncFunctionDef) and node.name.startswith("capped_")
}
def test_the_registry_covers_every_body_reading_helper():
"""The mutation guard for the test below: if a helper were renamed or added, the site list would
still match itself while the new one went uncovered."""
assert _body_readers() == {"capped_body", "capped_stream_to"}, (
"routes/_common.py grew or renamed a body-reading helper — add its call sites to "
"CAPPED_BODY_SITES and this expectation."
)
def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
"""Parsed, not grepped. A regex over the call text has to guess how the arguments are spelled,
and the shapes it guesses wrong — a bare `4 * 1024 * 1024`, a parameter not named `request` —
are the natural ways the NEXT site gets written, so a miss reads as "no new sites" rather than
as a failure. `ast` sees every call whatever it looks like."""
readers = _body_readers()
routes_dir = pathlib.Path(__file__).resolve().parents[1] / "app" / "routes"
found: set[tuple[str, str]] = set()
found: set[tuple[str, str, str]] = set()
for py in routes_dir.rglob("*.py"):
tree = ast.parse(py.read_text("utf-8"), filename=str(py))
for node in ast.walk(tree):
@@ -228,22 +256,27 @@ def test_every_capped_body_site_is_known_and_fits_under_its_ceiling():
continue
fn = node.func
name = fn.attr if isinstance(fn, ast.Attribute) else getattr(fn, "id", None)
if name != "capped_body":
if name not in readers:
continue
limit = next(
(kw.value for kw in node.keywords if kw.arg == "limit"),
node.args[1] if len(node.args) > 1 else None,
node.args[2] if name == "capped_stream_to" and len(node.args) > 2 else None,
)
found.add((py.name, ast.unparse(limit) if limit is not None else "<none>"))
if limit is None and name != "capped_stream_to":
limit = node.args[1] if len(node.args) > 1 else None
found.add((py.name, name, ast.unparse(limit) if limit is not None else "<none>"))
assert found == CAPPED_BODY_SITES, (
f"a capped_body call site appeared, moved or changed shape: {found ^ CAPPED_BODY_SITES}. "
f"a body-reading call site appeared, moved or changed shape: {found ^ CAPPED_BODY_SITES}. "
"Each one declares a body size; check it against bodylimit.DEFAULT_MAX_BODY_BYTES and, if "
"it is larger, add its route to bodylimit.ROUTE_MAX_BODY_BYTES — then update this list."
)
# And the two known ones really do satisfy the rule.
# And the known ones really do satisfy the rule.
assert diagnostics_routes.BODY_MAX_BYTES <= DEFAULT_MAX_BODY_BYTES
assert debugsession.SCREENSHOT_MAX_BYTES > DEFAULT_MAX_BODY_BYTES
assert ("POST", "/api/debug-sessions/{sid}/screenshots") in bodylimit.ROUTE_MAX_BODY_BYTES
# The upload site's limit is the sysconfig knob, which is bounded BY the edge ceiling rather
# than being a constant to compare against — that pairing is its own test below.
assert ("POST", "/api/downloads/upload") in bodylimit.ROUTE_MAX_BODY_BYTES
def test_the_upload_ceiling_and_the_admin_knob_cannot_drift_apart():
@@ -254,6 +287,12 @@ def test_the_upload_ceiling_and_the_admin_knob_cannot_drift_apart():
Since P1 the two count the SAME bytes — the body is the file, with no multipart envelope
wrapped round it — so the old allowance is gone and the edge is the knob's own maximum,
exactly. A gap reappearing means someone re-wrapped the upload in an envelope without saying so.
Equality also means the EDGE is what answers on a default install, where no override has lowered
the knob: the declared-length check refuses before a byte is read. That is not a gap to close
with headroom (measured: headroom only moves the boundary by its own size, and the route's drain
pushes the count back over it) — it is a refusal the client has to be able to read, which is why
`errors.tooLarge` exists and why the dialog checks the file against `upload_max_bytes` first.
"""
spec = next(s for s in sysconfig.SPECS if s.key == "download_upload_max_bytes")
edge = bodylimit.ROUTE_MAX_BODY_BYTES[("POST", "/api/downloads/upload")]
@@ -288,19 +327,28 @@ def test_no_route_with_a_raised_ceiling_declares_a_body_parameter(method, templa
closed on the upload route, which sat in `ROUTE_MAX_BODY_BYTES` with a 500 MiB ceiling and a
`file: UploadFile = File(...)` body for one release.
Asked of FastAPI itself (`route.dependant.body_params`) rather than parsed or grepped: the
router already knows which parameters it will read the body for, and a shape a regex or an AST
walk guesses wrong is precisely how the next route slips through."""
Asked of FastAPI itself rather than parsed or grepped: the router already knows which parameters
it will read the body for, and a shape a regex or an AST walk guesses wrong is precisely how the
next route slips through.
`route.body_field`, NOT `route.dependant.body_params`, and the difference is the whole test.
`body_params` holds only the parameters of the route's OWN callable; `body_field` is built from
`_get_flat_body_params`, which recurses into sub-dependencies — and that is also what FastAPI
itself consults when it decides to read the body. Measured on the pinned FastAPI: a body declared
inside a sub-dependency leaves `body_params` EMPTY while the body is still read in full before
auth, so the first version of this test was green over a reproduction of the very defect it
exists to forbid (deep-review round 1). The upload route's own idiom — a body read from a
`yield` dependency — is exactly the shape that would have slipped through."""
from app.main import app
route = _route_objects(app).get((method, template))
assert route is not None, f"{method} {template} is not a route any more"
body_params = [p.name for p in getattr(route.dependant, "body_params", ())]
assert not body_params, (
f"{method} {template} declares a body parameter ({', '.join(body_params)}), so FastAPI "
"spools it before the first dependency runs and the raised ceiling is all that stands "
"between an anonymous caller and the whole allowance. Read the body yourself, from a "
"dependency declared AFTER the auth one — see routes/downloads.py:spool_upload."
assert route.body_field is None, (
f"{method} {template} declares a body ({route.body_field.name}) on the route or in one of "
"its dependencies, so FastAPI reads it before the first dependency runs and the raised "
"ceiling is all that stands between an anonymous caller and the whole allowance. Read the "
"body yourself, from a dependency declared AFTER the auth one — see "
"routes/downloads.py:spool_upload."
)