peter/siftlode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(uploads): finish the sweeps round 1 left half-done

Browse Source 
Deep-review round 2, on round 1's own fix commit. It found MORE than round 1
did (28 raw, against 13), and most of it was mine -- a new piece of UI attracts
findings a backend-only diff never does. The pattern in almost all of them is
the same: a principle applied to the site that prompted it and not to its
siblings.

* `display_text` reached three of the FIVE writers of a user-supplied name.
  `POST /api/downloads` and `POST /api/downloads/edit` still handed Postgres an
  unbounded string, so a 300-character name pasted into the Add-download dialog
  or the editor was a bare 500 (main.py re-raises every DataError but the NUL
  case). Four of the seven reviewers found it independently. The bound now lives
  in `models.py`, the three columns are DECLARED with it, and the guard for the
  next writer is parsed rather than remembered: every `display_name` /
  `display_uploader` assignment under `app/routes/` must go through the helper.
  That guard immediately found two more -- the Google sign-in callback writes
  the profile name into the same 255-wide column -- so it is six, not five.

* The connection release reached one of the THREE routes that read their own
  body. The screenshot route (10 MiB, from a phone) and the client log sink both
  still held a pooled connection idle-in-transaction for the length of the
  upload, which is the invariant db.py sizes the pool on. All three now call
  `db.release_read_connection`, whose precondition -- nothing pending on the
  shared session -- is CHECKED rather than described, because a whole-session
  rollback discarding a caller's staged write is a mistake this repo has already
  made once. `test_body_limit.py` holds the rule for the next such route.

* The AST drift registry keyed its sites by (module, limit expression), so a
  second route copied from the first collapsed onto the same entry and went
  uncovered -- the exact hole the registry exists to close. Keyed per call site
  now, and it carries the release requirement too.

* The dialog's own refusal was announced once and never again: two over-cap
  picks leave an identical node with identical text, and an untouched live
  region says nothing. It is keyed per pick now, the drop zone reads as invalid
  and points at the sentence, and the reason is reachable from the control
  rather than only from a one-shot announcement.

* Two tests rebuilt the blind spot they were written to remove: the dialog's cap
  test sat nowhere near the boundary (a `>` loosened to `>=` survived it, which
  is the mutation round 1 fixed on the server side), and the header-precedence
  test only covered an exact-case duplicate while a differently-cased one would
  have been COMBINED onto the wire rather than replaced. Both fixed, and the
  merge now dedupes case-insensitively because header names are.

Also: the upload dialog's unstubbed usage query was firing a real request from
jsdom in a neighbouring suite and leaking a delayed "connection lost" toast
across test boundaries; `GET /api/downloads/usage` had no test at all though the
whole client-side check hangs off it; and the comment claiming the edge 413 is
readable now says what was measured -- readable to 200 MiB over loopback, a bare
`onerror` through a proxy -- so the pre-send check is named as the mechanism and
that branch as the backstop it is.

Every test added or changed here was mutation-checked. One mutation SURVIVED and
is recorded rather than papered over: the `tooBig` term inside `submit()` cannot
be reached, because `ui/Button` swallows the activation itself, so it is marked
as belt and braces instead of being described as the guard.
This commit is contained in:
peter
2026-09-22 23:31:55 +02:00
parent efd7b217ca
commit 7c0647f87a
21 changed files with 566 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tests/test_body_limit.py
+125 -24
View File
@@ -214,9 +214,9 @@ def test_the_default_ceiling_clears_the_largest_body_a_default_route_takes():
# site the drift guard could not see (deep-review round 1). The names come from `_common.py` itself
# below, so a THIRD sibling cannot be added without this list noticing either.
CAPPED_BODY_SITES = {
("diagnostics.py", "capped_body", "BODY_MAX_BYTES"),
("debug_sessions.py", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
("downloads.py", "capped_stream_to", "cap"),
("diagnostics.py", "_parsed_events", "capped_body", "BODY_MAX_BYTES"),
("debug_sessions.py", "_image_body", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
("downloads.py", "spool_upload", "capped_stream_to", "cap"),
}
@@ -228,43 +228,91 @@ def _body_readers() -> set[str]:
return {
node.name
for node in tree.body
if isinstance(node, ast.AsyncFunctionDef) and node.name.startswith("capped_")
if isinstance(node, (ast.AsyncFunctionDef, ast.FunctionDef))
and node.name.startswith("capped_")
}
def test_the_registry_covers_every_body_reading_helper():
"""The mutation guard for the test below: if a helper were renamed or added, the site list would
still match itself while the new one went uncovered."""
"""The mutation guard for the tests below: if a helper were renamed or added, the site list
would still match itself while the new one went uncovered."""
assert _body_readers() == {"capped_body", "capped_stream_to"}, (
"routes/_common.py grew or renamed a body-reading helper — add its call sites to "
"CAPPED_BODY_SITES and this expectation."
)
def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
"""Parsed, not grepped. A regex over the call text has to guess how the arguments are spelled,
and the shapes it guesses wrong — a bare `4 * 1024 * 1024`, a parameter not named `request` —
are the natural ways the NEXT site gets written, so a miss reads as "no new sites" rather than
as a failure. `ast` sees every call whatever it looks like."""
def _body_reading_functions() -> dict[tuple[str, str], dict]:
"""Every function under `app/routes/` that reads a request body itself, keyed by
(module, function).
Parsed, not grepped. A regex over the call text has to guess how the arguments are spelled, and
the shapes it guesses wrong — a bare `4 * 1024 * 1024`, a parameter not named `request` — are
the natural ways the NEXT site gets written, so a miss reads as "no new sites" rather than as a
failure. `ast` sees every call whatever it looks like.
Keyed by the ENCLOSING FUNCTION, not just the module: a set keyed by module and limit
expression collapses two call sites that spell their limit the same way, which is exactly what
a second route copied from the first would do.
"""
readers = _body_readers()
routes_dir = pathlib.Path(__file__).resolve().parents[1] / "app" / "routes"
found: set[tuple[str, str, str]] = set()
for py in routes_dir.rglob("*.py"):
found: dict[tuple[str, str], dict] = {}
for py in sorted(routes_dir.rglob("*.py")):
tree = ast.parse(py.read_text("utf-8"), filename=str(py))
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
for fn in ast.walk(tree):
if not isinstance(fn, (ast.AsyncFunctionDef, ast.FunctionDef)):
continue
fn = node.func
name = fn.attr if isinstance(fn, ast.Attribute) else getattr(fn, "id", None)
if name not in readers:
calls = [
node
for node in ast.walk(fn)
if isinstance(node, ast.Call)
and (
node.func.attr
if isinstance(node.func, ast.Attribute)
else getattr(node.func, "id", None)
)
in readers
]
if not calls:
continue
limit = next(
(kw.value for kw in node.keywords if kw.arg == "limit"),
node.args[2] if name == "capped_stream_to" and len(node.args) > 2 else None,
names = [
node.func.attr
if isinstance(node.func, ast.Attribute)
else getattr(node.func, "id", None)
for node in calls
]
limits = []
for node, name in zip(calls, names):
limit = next((kw.value for kw in node.keywords if kw.arg == "limit"), None)
if limit is None:
pos = 2 if name == "capped_stream_to" else 1
limit = node.args[pos] if len(node.args) > pos else None
limits.append(ast.unparse(limit) if limit is not None else "<none>")
releases = any(
isinstance(node, ast.Call)
and (
node.func.attr
if isinstance(node.func, ast.Attribute)
else getattr(node.func, "id", None)
)
== "release_read_connection"
for node in ast.walk(fn)
)
if limit is None and name != "capped_stream_to":
limit = node.args[1] if len(node.args) > 1 else None
found.add((py.name, name, ast.unparse(limit) if limit is not None else "<none>"))
found[(py.name, fn.name)] = {
"helpers": names,
"limits": limits,
"releases": releases,
}
return found
def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
found = {
(module, fn, site["helpers"][i], site["limits"][i])
for (module, fn), site in _body_reading_functions().items()
for i in range(len(site["helpers"]))
}
assert found == CAPPED_BODY_SITES, (
f"a body-reading call site appeared, moved or changed shape: {found ^ CAPPED_BODY_SITES}. "
"Each one declares a body size; check it against bodylimit.DEFAULT_MAX_BODY_BYTES and, if "
@@ -279,6 +327,59 @@ def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
assert ("POST", "/api/downloads/upload") in bodylimit.ROUTE_MAX_BODY_BYTES
def test_release_read_connection_ends_the_transaction_it_was_called_for(db):
"""The helper itself: a SELECT has autobegun, and afterwards it has not."""
from sqlalchemy import select
from app.db import release_read_connection
db.execute(select(SystemConfig).limit(1))
assert db.in_transaction(), "nothing autobegan, so this test would pass vacuously"
release_read_connection(db)
assert not db.in_transaction()
def test_release_read_connection_refuses_to_discard_pending_work(db):
"""The precondition, ENFORCED rather than described.
The session belongs to the whole request, not to the dependency calling this — so if something
ahead of it ever stages a write, ending the transaction would throw that write away with
nothing in the log. `service.enqueue` already learned that a whole-session rollback takes the
caller's pending work with it, which is why it uses a SAVEPOINT. A comment cannot stop the next
dependency from being added in the wrong place; this can.
"""
from app.db import release_read_connection
db.add(SystemConfig(key="probe_pending_write", value="1"))
with pytest.raises(RuntimeError, match="pending work"):
release_read_connection(db)
db.rollback() # the staged row is this test's own
def test_every_body_reading_site_hands_its_pooled_connection_back_first():
"""Reading the body yourself is what makes a request SLOW inside its own database session.
The auth (or ownership) dependency that must resolve first is a SELECT, so SQLAlchemy has
autobegun and the session is holding one of the pool's connections; awaiting the body then pins
it for as long as the client keeps sending — one connection per in-flight upload, `idle in
transaction`, with nothing to time it out. `db.release_read_connection` ends that transaction,
and every one of these sites needs it.
Held generically because the three sites arrived one at a time and only the newest was fixed:
a fourth body-reading route gets the same requirement automatically.
"""
offenders = [
f"{module}:{fn}"
for (module, fn), site in _body_reading_functions().items()
if not site["releases"]
]
assert not offenders, (
"a route reads its own request body while still inside its read transaction, pinning a "
f"pooled connection for the whole transfer: {offenders}. Call "
"db.release_read_connection(db) before the await — see routes/downloads.py:spool_upload."
)
def test_the_upload_ceiling_and_the_admin_knob_cannot_drift_apart():
"""The upload route's in-handler cap is a `sysconfig` value an admin can change; the edge is a
constant. If the knob could be set above the edge, an admin raising it would silently start