fix(uploads): make both caps answerable, and stop holding a connection

The epic-end deep review found three defects in P1 and four tests that did not
hold what they claimed. They land together because they share the upload route
and its test module; splitting by hunk would have produced commits that do not
individually run green.

* The edge ceiling and the route's own cap are the SAME number at the default
  configuration, so the edge always answers first: a declared Content-Length
  over the cap is refused before a byte is read, and an undeclared body trips
  the middleware on the same chunk the route would have tripped on. The route's
  numbered 422 -- and the drain written for it last week -- are reachable only
  once an admin LOWERS the knob, which is the one shape the browser UAT tested.
  Measured: headroom does not fix this, because the declared-length check fires
  first for any body big enough to matter and the drain pushes the count back
  over the edge for the rest. So the edge's own refusal is made readable
  instead: `errors.tooLarge` localizes `too_large` the way `upload_too_large`
  was already localized, and the dialog refuses an over-cap file BEFORE sending
  it, off the `upload_max_bytes` that GET /api/downloads/usage now reports.
  Measured in Chrome against the API directly: that 413 arrives as `onload`
  with a readable body in 43 ms.

* Reading the body behind the auth dependency put the whole transfer inside the
  request's database session -- the auth SELECT autobegins a transaction, so one
  of the twenty pooled connections sat `idle in transaction`, with nothing to
  time it out, for as long as the client cared to keep sending. That is the
  invariant db.py sizes the pool on and plex/playback.py records an incident
  for. The read transaction now ends before the await. Measured on the dev
  server: an 1818 ms upload shows no idle-in-transaction sample while the bytes
  move, only afterwards, when the handler actually writes.

* The display name moved into a request header, where nothing bounds its length,
  while the column stayed varchar(255) -- so a long enough name was a DataError
  500 after the whole file had already transferred. One `display_text` helper
  now serves all three writers (the metadata PATCH already truncated, the upload
  ingest did not), and the dialog stops at the same number.

* The gate invariant asked `route.dependant.body_params`, which holds only the
  route's OWN callable's parameters: a body declared in a SUB-DEPENDENCY -- the
  upload route's own idiom -- leaves it empty while FastAPI still reads the body
  before auth. It asks `route.body_field` now, which is what FastAPI itself
  consults. The AST registry of body-reading call sites had also never followed
  `capped_stream_to`, so the one route the whole invariant exists for was the
  single site the drift guard could not see.

Four tests that passed over their own mutation are fixed: the cap boundary (a
`>` loosened to `>=` survived the entire suite), the percent-decoding (every
filename the suite sends is ASCII, where quote() is the identity function), the
display-name header (no test at all), and the staging cleanup (deleting the
unlink left the suite green while every refused upload leaked its partial bytes
onto the volume). Every test added or changed here was mutation-checked.
This commit is contained in:
2026-09-22 22:41:22 +02:00
parent 3d7ef9e945
commit efd7b217ca
14 changed files with 428 additions and 67 deletions
+19 -4
View File
@@ -74,10 +74,25 @@ DEFAULT_MAX_BODY_BYTES = 1024 * 1024
# default, so listing it could only widen what it accepts.
ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# Ingesting a local file. The body IS the file since P1 — no multipart envelope to allow for —
# so this is exactly the number the route enforces. `sysconfig`'s `download_upload_max_bytes` is
# bounded so it can never exceed it (see sysconfig.SPECS), which is what stops an admin from
# raising the route's own cap past the edge and collecting 413s from a layer the admin page
# never mentions.
# so this ceiling and the route's own cap count exactly the same bytes. `sysconfig`'s
# `download_upload_max_bytes` is bounded by this same constant (see sysconfig.SPECS), so an
# admin can only ever LOWER the route's cap below the edge, never raise it past one.
#
# WHICH LAYER ANSWERS, measured rather than assumed (deep-review round 1):
# * knob LOWERED (an override below the default) — the route's cap is the smaller number, so
# `capped_stream_to` trips first and answers 422 `upload_too_large` carrying the admin's own
# figure, after draining the rest of the body so the client can read it.
# * knob at the DEFAULT — the two numbers are EQUAL, and the edge always wins: a declared
# Content-Length over the cap is refused here before a byte is read, and an undeclared body
# trips this counter on the same chunk the route would have tripped on. The route's 422 is
# unreachable in that configuration, and no amount of headroom would change it — the
# declared-length check fires first for any body big enough to matter, and the route's drain
# pushes the count past the edge for the rest.
#
# So this 413 is a refusal a real user meets on a default install, not a theoretical edge, and
# it has to be as readable as the route's own: the client localizes `too_large` (`errors.tooLarge`)
# exactly as it does `upload_too_large`, and the upload dialog refuses an over-cap file before
# sending it at all, off the `upload_max_bytes` that `GET /api/downloads/usage` reports.
("POST", "/api/downloads/upload"): settings.download_upload_max_bytes,
("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
}
+20
View File
@@ -31,6 +31,26 @@ class EditError(Exception):
super().__init__(reason)
# The width of every user-supplied display string on a job (`DownloadJob.display_name`,
# `display_uploader`): `String(255)`. `test_upload.py` holds this against the columns
# themselves so the two cannot drift — a number in a comment is exactly how that drift happens.
DISPLAY_TEXT_MAX = 255
def display_text(raw: str | None) -> str | None:
"""A user-supplied display string on its way into one of those columns: stripped, truncated,
blank → None.
One definition because there are three writers and they had drifted: the metadata PATCH
truncated, the upload ingest did not, and since P1 the upload's name arrives in a REQUEST HEADER
where nothing bounds its length — so a long enough one reached Postgres as a `DataError`, i.e. a
500 after the whole file had already been transferred. Truncating is the right answer rather
than refusing: the name is a label on a file the user has already sent, and losing the tail of it
is not worth failing the upload over.
"""
return (raw or "").strip()[:DISPLAY_TEXT_MAX] or None
def source_url(source_kind: str, source_ref: str) -> str:
"""The URL yt-dlp should fetch. YouTube stores the 11-char id (the adapter reconstructs the watch
URL); ad-hoc sources store the URL verbatim."""
+1 -1
View File
@@ -99,7 +99,7 @@ def ingest_upload(
source_kind="upload",
source_ref=ref,
profile_snapshot={},
display_name=(display_name or "").strip() or None,
display_name=service.display_text(display_name),
status="done",
progress=100,
queue_pos=service.next_queue_pos(db),
+30 -5
View File
@@ -462,6 +462,18 @@ async def spool_upload(
MOVES it into place on success (the unlink is then a no-op), and on any other exit — cap hit,
empty, quota, probe or move error, a client that vanished mid-upload — it is removed."""
cap = sysconfig.get_int(db, "download_upload_max_bytes")
user_id = user.id
# Hand the pooled connection back BEFORE the transfer starts. Resolving `user` and reading the
# cap were both SELECTs, and SQLAlchemy 2.0 autobegins a transaction on the first one — so
# without this the session sits `idle in transaction` holding one of the twenty pooled
# connections for as long as the client cares to keep sending. That is the invariant `db.py`
# sizes the pool on, and `plex/playback.py` records the production incident that taught it;
# reading the body here instead of letting FastAPI spool it (P1) is what put this route at risk
# of breaking it. Auth only reads, so ending the read transaction loses nothing, and the
# handler's first query autobegins a fresh one — one ROLLBACK round-trip against a minute of
# held connection. Everything this dependency still needs is already an int in hand: touching
# an ORM attribute below would reload `user` and check a connection straight back out.
db.rollback()
staging = Path(settings.download_root) / ".staging" / "uploads"
staging.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(staging))
@@ -477,9 +489,9 @@ async def spool_upload(
yield SpooledUpload(
path=tmp_path,
size=size,
user_id=user.id,
user_id=user_id,
filename=_header_text(request, UPLOAD_FILENAME_HEADER) or "upload",
display_name=_header_text(request, UPLOAD_DISPLAY_NAME_HEADER) or None,
display_name=service.display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER)),
)
finally:
tmp_path.unlink(missing_ok=True)
@@ -570,7 +582,20 @@ def list_downloads(
@router.get("/usage")
def my_usage(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
return quota.usage(db, user.id)
"""The caller's download quota, plus the per-file upload cap.
The cap rides here rather than in `quota.usage` because it is not a quota value — it is a
`sysconfig` knob about one route — and this is the route layer, where composing the two is the
job. The upload dialog needs it to refuse an over-cap file BEFORE sending it: both server-side
caps answer too late to be pleasant (the edge ceiling refuses the request outright, see
`bodylimit.ROUTE_MAX_BODY_BYTES`), and making the user push 600 MB up the wire to be told the
limit is 500 is a bad answer even when it is a correct one. It is a convenience, not the
boundary: the caps stay where they are and still refuse a client that ignores this.
"""
return {
**quota.usage(db, user.id),
"upload_max_bytes": sysconfig.get_int(db, "download_upload_max_bytes"),
}
# Priority when a source has several jobs (e.g. a done one plus a fresh queued one).
@@ -665,9 +690,9 @@ def update_download_meta(
job = owned_or_404(db, DownloadJob, job_id, user, what="download")
sent = body.model_fields_set
if "display_name" in sent:
job.display_name = ((body.display_name or "").strip())[:255] or None
job.display_name = service.display_text(body.display_name)
if "display_uploader" in sent:
job.display_uploader = ((body.display_uploader or "").strip())[:255] or None
job.display_uploader = service.display_text(body.display_uploader)
if "display_uploader_url" in sent:
job.display_uploader_url = _clean_url(body.display_uploader_url)
if "extra_links" in sent:
+10 -5
View File
@@ -137,13 +137,18 @@ SPECS: tuple[ConfigSpec, ...] = (
min=1,
max=100_000,
),
# min=1, NOT 0: the per-file cap is enforced as `if cap and size > cap`, so a stored 0 would read
# as falsy and SILENTLY disable the limit (an unbounded streamed write to the volume). Floor it at
# 1 so the cap is always active; the overall footprint is separately bounded by the byte quota.
# `max` is not cosmetic: `bodylimit.ROUTE_MAX_BODY_BYTES` refuses an upload body over roughly
# min=1, NOT 0: a stored 0 used to read as "no per-file cap" (the cap was enforced as
# `if cap and size > cap`, so a falsy value disabled it). Since P1 the route passes this number
# straight to `capped_stream_to`, where 0 would refuse every upload instead — the opposite
# failure, and the reason `config.download_upload_max_bytes` carries its own `ge=1`. Either way
# a stored 0 is wrong, so it is floored at 1; the overall footprint is separately bounded by the
# byte quota.
# `max` is not cosmetic: `bodylimit.ROUTE_MAX_BODY_BYTES` refuses an upload body over exactly
# this number at the edge, before the route's own check ever sees it. An admin allowed to raise
# the knob past the edge would get 413s from a layer the admin page never mentions, so the two
# are pinned together here and `tests/test_body_limit.py` holds them that way. `_coerce` also
# are pinned together here and `tests/test_body_limit.py` holds them that way. Below the edge is
# the useful direction and the one the knob exists for: a lowered knob is what makes the route's
# own 422 — the one carrying this figure — the answer the user gets. `_coerce` also
# REJECTS a stored value outside these bounds on read (falling back to the env default, with a
# warning) — neither bound has always been what it is now, so a legal-when-written override can
# be sitting in the database already.