peter/siftlode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(uploads): finish the sweeps round 1 left half-done

Browse Source 
Deep-review round 2, on round 1's own fix commit. It found MORE than round 1
did (28 raw, against 13), and most of it was mine -- a new piece of UI attracts
findings a backend-only diff never does. The pattern in almost all of them is
the same: a principle applied to the site that prompted it and not to its
siblings.

* `display_text` reached three of the FIVE writers of a user-supplied name.
  `POST /api/downloads` and `POST /api/downloads/edit` still handed Postgres an
  unbounded string, so a 300-character name pasted into the Add-download dialog
  or the editor was a bare 500 (main.py re-raises every DataError but the NUL
  case). Four of the seven reviewers found it independently. The bound now lives
  in `models.py`, the three columns are DECLARED with it, and the guard for the
  next writer is parsed rather than remembered: every `display_name` /
  `display_uploader` assignment under `app/routes/` must go through the helper.
  That guard immediately found two more -- the Google sign-in callback writes
  the profile name into the same 255-wide column -- so it is six, not five.

* The connection release reached one of the THREE routes that read their own
  body. The screenshot route (10 MiB, from a phone) and the client log sink both
  still held a pooled connection idle-in-transaction for the length of the
  upload, which is the invariant db.py sizes the pool on. All three now call
  `db.release_read_connection`, whose precondition -- nothing pending on the
  shared session -- is CHECKED rather than described, because a whole-session
  rollback discarding a caller's staged write is a mistake this repo has already
  made once. `test_body_limit.py` holds the rule for the next such route.

* The AST drift registry keyed its sites by (module, limit expression), so a
  second route copied from the first collapsed onto the same entry and went
  uncovered -- the exact hole the registry exists to close. Keyed per call site
  now, and it carries the release requirement too.

* The dialog's own refusal was announced once and never again: two over-cap
  picks leave an identical node with identical text, and an untouched live
  region says nothing. It is keyed per pick now, the drop zone reads as invalid
  and points at the sentence, and the reason is reachable from the control
  rather than only from a one-shot announcement.

* Two tests rebuilt the blind spot they were written to remove: the dialog's cap
  test sat nowhere near the boundary (a `>` loosened to `>=` survived it, which
  is the mutation round 1 fixed on the server side), and the header-precedence
  test only covered an exact-case duplicate while a differently-cased one would
  have been COMBINED onto the wire rather than replaced. Both fixed, and the
  merge now dedupes case-insensitively because header names are.

Also: the upload dialog's unstubbed usage query was firing a real request from
jsdom in a neighbouring suite and leaking a delayed "connection lost" toast
across test boundaries; `GET /api/downloads/usage` had no test at all though the
whole client-side check hangs off it; and the comment claiming the edge 413 is
readable now says what was measured -- readable to 200 MiB over loopback, a bare
`onerror` through a proxy -- so the pre-send check is named as the mechanism and
that branch as the backstop it is.

Every test added or changed here was mutation-checked. One mutation SURVIVED and
is recorded rather than papered over: the `tooBig` term inside `submit()` cannot
be reached, because `ui/Button` swallows the activation itself, so it is marked
as belt and braces instead of being described as the guard.
This commit is contained in:
peter
2026-09-22 23:31:55 +02:00
parent efd7b217ca
commit 7c0647f87a
21 changed files with 566 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tests/test_body_limit.py
+125 -24
View File
@@ -214,9 +214,9 @@ def test_the_default_ceiling_clears_the_largest_body_a_default_route_takes():
# site the drift guard could not see (deep-review round 1). The names come from `_common.py` itself
# below, so a THIRD sibling cannot be added without this list noticing either.
CAPPED_BODY_SITES = {
("diagnostics.py", "capped_body", "BODY_MAX_BYTES"),
("debug_sessions.py", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
("downloads.py", "capped_stream_to", "cap"),
("diagnostics.py", "_parsed_events", "capped_body", "BODY_MAX_BYTES"),
("debug_sessions.py", "_image_body", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
("downloads.py", "spool_upload", "capped_stream_to", "cap"),
}
@@ -228,43 +228,91 @@ def _body_readers() -> set[str]:
return {
node.name
for node in tree.body
if isinstance(node, ast.AsyncFunctionDef) and node.name.startswith("capped_")
if isinstance(node, (ast.AsyncFunctionDef, ast.FunctionDef))
and node.name.startswith("capped_")
}
def test_the_registry_covers_every_body_reading_helper():
"""The mutation guard for the test below: if a helper were renamed or added, the site list would
still match itself while the new one went uncovered."""
"""The mutation guard for the tests below: if a helper were renamed or added, the site list
would still match itself while the new one went uncovered."""
assert _body_readers() == {"capped_body", "capped_stream_to"}, (
"routes/_common.py grew or renamed a body-reading helper — add its call sites to "
"CAPPED_BODY_SITES and this expectation."
)
def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
"""Parsed, not grepped. A regex over the call text has to guess how the arguments are spelled,
and the shapes it guesses wrong — a bare `4 * 1024 * 1024`, a parameter not named `request` —
are the natural ways the NEXT site gets written, so a miss reads as "no new sites" rather than
as a failure. `ast` sees every call whatever it looks like."""
def _body_reading_functions() -> dict[tuple[str, str], dict]:
"""Every function under `app/routes/` that reads a request body itself, keyed by
(module, function).
Parsed, not grepped. A regex over the call text has to guess how the arguments are spelled, and
the shapes it guesses wrong — a bare `4 * 1024 * 1024`, a parameter not named `request` — are
the natural ways the NEXT site gets written, so a miss reads as "no new sites" rather than as a
failure. `ast` sees every call whatever it looks like.
Keyed by the ENCLOSING FUNCTION, not just the module: a set keyed by module and limit
expression collapses two call sites that spell their limit the same way, which is exactly what
a second route copied from the first would do.
"""
readers = _body_readers()
routes_dir = pathlib.Path(__file__).resolve().parents[1] / "app" / "routes"
found: set[tuple[str, str, str]] = set()
for py in routes_dir.rglob("*.py"):
found: dict[tuple[str, str], dict] = {}
for py in sorted(routes_dir.rglob("*.py")):
tree = ast.parse(py.read_text("utf-8"), filename=str(py))
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
for fn in ast.walk(tree):
if not isinstance(fn, (ast.AsyncFunctionDef, ast.FunctionDef)):
continue
fn = node.func
name = fn.attr if isinstance(fn, ast.Attribute) else getattr(fn, "id", None)
if name not in readers:
calls = [
node
for node in ast.walk(fn)
if isinstance(node, ast.Call)
and (
node.func.attr
if isinstance(node.func, ast.Attribute)
else getattr(node.func, "id", None)
)
in readers
]
if not calls:
continue
limit = next(
(kw.value for kw in node.keywords if kw.arg == "limit"),
node.args[2] if name == "capped_stream_to" and len(node.args) > 2 else None,
names = [
node.func.attr
if isinstance(node.func, ast.Attribute)
else getattr(node.func, "id", None)
for node in calls
]
limits = []
for node, name in zip(calls, names):
limit = next((kw.value for kw in node.keywords if kw.arg == "limit"), None)
if limit is None:
pos = 2 if name == "capped_stream_to" else 1
limit = node.args[pos] if len(node.args) > pos else None
limits.append(ast.unparse(limit) if limit is not None else "<none>")
releases = any(
isinstance(node, ast.Call)
and (
node.func.attr
if isinstance(node.func, ast.Attribute)
else getattr(node.func, "id", None)
)
== "release_read_connection"
for node in ast.walk(fn)
)
if limit is None and name != "capped_stream_to":
limit = node.args[1] if len(node.args) > 1 else None
found.add((py.name, name, ast.unparse(limit) if limit is not None else "<none>"))
found[(py.name, fn.name)] = {
"helpers": names,
"limits": limits,
"releases": releases,
}
return found
def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
found = {
(module, fn, site["helpers"][i], site["limits"][i])
for (module, fn), site in _body_reading_functions().items()
for i in range(len(site["helpers"]))
}
assert found == CAPPED_BODY_SITES, (
f"a body-reading call site appeared, moved or changed shape: {found ^ CAPPED_BODY_SITES}. "
"Each one declares a body size; check it against bodylimit.DEFAULT_MAX_BODY_BYTES and, if "
@@ -279,6 +327,59 @@ def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
assert ("POST", "/api/downloads/upload") in bodylimit.ROUTE_MAX_BODY_BYTES
def test_release_read_connection_ends_the_transaction_it_was_called_for(db):
"""The helper itself: a SELECT has autobegun, and afterwards it has not."""
from sqlalchemy import select
from app.db import release_read_connection
db.execute(select(SystemConfig).limit(1))
assert db.in_transaction(), "nothing autobegan, so this test would pass vacuously"
release_read_connection(db)
assert not db.in_transaction()
def test_release_read_connection_refuses_to_discard_pending_work(db):
"""The precondition, ENFORCED rather than described.
The session belongs to the whole request, not to the dependency calling this — so if something
ahead of it ever stages a write, ending the transaction would throw that write away with
nothing in the log. `service.enqueue` already learned that a whole-session rollback takes the
caller's pending work with it, which is why it uses a SAVEPOINT. A comment cannot stop the next
dependency from being added in the wrong place; this can.
"""
from app.db import release_read_connection
db.add(SystemConfig(key="probe_pending_write", value="1"))
with pytest.raises(RuntimeError, match="pending work"):
release_read_connection(db)
db.rollback() # the staged row is this test's own
def test_every_body_reading_site_hands_its_pooled_connection_back_first():
"""Reading the body yourself is what makes a request SLOW inside its own database session.
The auth (or ownership) dependency that must resolve first is a SELECT, so SQLAlchemy has
autobegun and the session is holding one of the pool's connections; awaiting the body then pins
it for as long as the client keeps sending — one connection per in-flight upload, `idle in
transaction`, with nothing to time it out. `db.release_read_connection` ends that transaction,
and every one of these sites needs it.
Held generically because the three sites arrived one at a time and only the newest was fixed:
a fourth body-reading route gets the same requirement automatically.
"""
offenders = [
f"{module}:{fn}"
for (module, fn), site in _body_reading_functions().items()
if not site["releases"]
]
assert not offenders, (
"a route reads its own request body while still inside its read transaction, pinning a "
f"pooled connection for the whole transfer: {offenders}. Call "
"db.release_read_connection(db) before the await — see routes/downloads.py:spool_upload."
)
def test_the_upload_ceiling_and_the_admin_knob_cannot_drift_apart():
"""The upload route's in-handler cap is a `sysconfig` value an admin can change; the edge is a
constant. If the knob could be set above the edge, an admin raising it would silently start
backend/tests/test_upload.py
+134 -10
View File
@@ -12,8 +12,10 @@ parsing (image → file, cover-art skip, dims, duration) is covered directly by
below, which stubs one level down (`run_ffprobe_json`) and exercises the real `probe_media`.
"""
import ast
import asyncio
import io
import pathlib
from pathlib import Path
import pytest
@@ -23,10 +25,10 @@ from app.config import settings
from app.routes._common import capped_stream_to
from sqlalchemy import select
from app.downloads import mediakind, quota, service, storage
from app.downloads import mediakind, quota, storage
from app.downloads import probe as probemod
from app.downloads import upload as uploadmod
from app.models import DownloadJob, DownloadQuota, MediaAsset
from app.models import DISPLAY_TEXT_MAX, DownloadJob, DownloadQuota, MediaAsset
# --- pure: the kind invariant ---------------------------------------------------------------
@@ -422,12 +424,99 @@ def test_a_non_ascii_filename_survives_the_header_round_trip(
assert stored.startswith("nyaralas_osszel"), stored
def test_a_long_display_name_is_truncated_on_EVERY_route_that_takes_one(
db, as_user, make_user, _root, monkeypatch, upload_headers
):
"""The sweep, held end to end rather than at one site.
`display_text` was introduced for the upload path and applied to three of the five writers of
this varchar(255) column; `POST /api/downloads` and `POST /api/downloads/edit` kept the old
`(x or "").strip() or None` and would still have answered a bare 500 (a psycopg DataError,
which main.py re-raises for everything but the NUL case) to a name a user simply pasted. Found
by four independent readers in deep-review round 2 — a sweep is exactly the kind of thing that
gets done four-fifths of the way, so this holds all of it at once.
"""
_probe(monkeypatch, ("h264", "aac", 640, 480, 60))
monkeypatch.setattr(uploadmod.editmod, "ensure_poster", lambda _a, _root: None)
long_name = "n" * 300
bound = "n" * DISPLAY_TEXT_MAX
c = as_user(make_user())
# 1. the upload route — the name arrives in a request header.
up = c.post(
"/api/downloads/upload", content=b"video", headers=upload_headers("a.mp4", long_name)
)
assert up.status_code == 200, up.text
assert up.json()["display_name"] == bound
# 2. the enqueue route — a plain JSON field, from the Add-download dialog. An inline `spec`
# rather than a profile id, so the test does not have to seed the builtin profile table.
enq = c.post(
"/api/downloads",
json={"source": "dQw4w9WgXcQ", "spec": {"kind": "video"}, "display_name": long_name},
)
assert enq.status_code == 200, enq.text
assert enq.json()["display_name"] == bound
# 3. the edit route — the same field, from the editor.
edit = c.post(
"/api/downloads/edit",
json={
"source_job_id": up.json()["id"],
"edit_spec": {"trim": {"start_s": 0, "end_s": 1}},
"display_name": long_name,
},
)
assert edit.status_code == 200, edit.text
assert edit.json()["display_name"] == bound
# 4 + 5. the metadata PATCH's two fields.
patched = c.patch(
f"/api/downloads/{up.json()['id']}",
json={"display_name": long_name, "display_uploader": long_name},
)
assert patched.status_code == 200, patched.text
assert patched.json()["display_name"] == bound
assert patched.json()["display_uploader"] == bound
def test_no_route_assigns_a_display_field_without_the_shared_helper():
"""The drift guard for the test above: it holds five routes, this holds the rule.
Parsed, not grepped. Every `display_name = ...` / `display_uploader = ...` assignment under
`app/routes/` takes a string a user typed and puts it on its way to a varchar(255) column, so
every one of them must go through `display_text`. A sixth route added tomorrow fails here
instead of 500ing in production.
"""
routes_dir = pathlib.Path(__file__).resolve().parents[1] / "app" / "routes"
offenders: list[str] = []
for py in sorted(routes_dir.rglob("*.py")):
tree = ast.parse(py.read_text("utf-8"), filename=str(py))
for node in ast.walk(tree):
if not isinstance(node, ast.Assign):
continue
names = {
t.id if isinstance(t, ast.Name) else t.attr
for t in node.targets
if isinstance(t, (ast.Name, ast.Attribute))
}
if not names & {"display_name", "display_uploader"}:
continue
value = ast.unparse(node.value)
if "display_text" not in value:
offenders.append(f"{py.name}:{node.lineno}: {value}")
assert not offenders, (
"a display field is assigned without models.display_text, so a name longer "
f"than the column is a 500 rather than a truncation: {offenders}"
)
def test_the_display_text_bound_matches_the_columns_it_guards():
"""A number living in two places is how the `cancelled`/`canceled` bug happened. `display_text`
truncates to `DISPLAY_TEXT_MAX`; the columns it feeds are `String(255)`. Held against the model
itself, so widening one without the other fails here rather than as a DataError in production."""
for name in ("display_name", "display_uploader"):
assert DownloadJob.__table__.c[name].type.length == service.DISPLAY_TEXT_MAX
assert DownloadJob.__table__.c[name].type.length == DISPLAY_TEXT_MAX
def test_the_display_name_header_reaches_the_job_and_is_bounded(
@@ -456,7 +545,7 @@ def test_the_display_name_header_reaches_the_job_and_is_bounded(
"/api/downloads/upload", content=b"data2", headers=upload_headers("clip2.bin", long_name)
)
assert bounded.status_code == 200, bounded.text
assert bounded.json()["display_name"] == "n" * service.DISPLAY_TEXT_MAX
assert bounded.json()["display_name"] == "n" * DISPLAY_TEXT_MAX
def test_the_upload_holds_no_db_transaction_while_the_body_streams(
@@ -477,22 +566,57 @@ def test_the_upload_holds_no_db_transaction_while_the_body_streams(
_probe(monkeypatch, (None, None, None, None, None))
seen: dict[str, bool] = {}
real = downloads_routes.capped_stream_to
real_stream = downloads_routes.capped_stream_to
real_release = downloads_routes.release_read_connection
async def spy(request, out, limit, refusal):
seen["in_transaction"] = bool(db.in_transaction())
return await real(request, out, limit, refusal)
def spy_release(session):
# The PRECONDITION, so this test cannot pass vacuously. Asserting only "no transaction
# while streaming" would stay green if the auth/cap reads stopped opening one at all —
# and under `as_user` the auth SELECT is already overridden away, so that is a live risk
# rather than a theoretical one.
seen["before"] = bool(session.in_transaction())
real_release(session)
seen["after_release"] = bool(session.in_transaction())
monkeypatch.setattr(downloads_routes, "capped_stream_to", spy)
async def spy_stream(request, out, limit, refusal):
seen["while_streaming"] = bool(db.in_transaction())
return await real_stream(request, out, limit, refusal)
monkeypatch.setattr(downloads_routes, "release_read_connection", spy_release)
monkeypatch.setattr(downloads_routes, "capped_stream_to", spy_stream)
c = as_user(make_user())
r = c.post("/api/downloads/upload", content=b"data", headers=upload_headers("held.bin"))
assert r.status_code == 200, r.text
assert seen["in_transaction"] is False, (
assert seen["before"] is True, (
"nothing had opened a transaction before the release, so this test proves nothing — the "
"reads it is meant to follow must have moved or been stubbed away"
)
assert seen["after_release"] is False
assert seen["while_streaming"] is False, (
"the request's session was inside a transaction when the transfer began, so a pooled "
"connection is pinned for the whole upload"
)
def test_the_usage_route_reports_the_per_file_upload_cap(as_user, make_user, db):
"""The dialog refuses an over-cap file off this number, and it is the only layer that produces
a refusal carrying the limit on a default install — so the field is load-bearing, not
decorative. Nothing held it: the route had no test at all, and the frontend's `DownloadUsage`
is an unvalidated cast, so dropping the key would have been green everywhere (round 2).
"""
from app import sysconfig
c = as_user(make_user())
r = c.get("/api/downloads/usage")
assert r.status_code == 200, r.text
assert r.json()["upload_max_bytes"] == sysconfig.get_int(db, "download_upload_max_bytes")
# And it is the KNOB, not a constant: a lowered override must show through.
sysconfig.set_value(db, "download_upload_max_bytes", 4096)
db.commit()
assert c.get("/api/downloads/usage").json()["upload_max_bytes"] == 4096
def test_a_refused_upload_leaves_nothing_in_the_staging_directory(
as_user, make_user, db, _root, monkeypatch, upload_headers
):