fix(uploads): make the drift guards fail for the reasons they claim
Deep-review round 3, targeted at the surfaces round 2's fix commit changed. Five of the eight findings are the same shape: the guards written to stop half-done sweeps were themselves half-done. test_upload.py's display-field guard walked only `ast.Assign`, so the one writer with an unbounded REQUEST HEADER behind it — `spool_upload`'s `display_name=display_text(...)` keyword — was invisible to the guard meant to protect it. It also stopped at `app/routes/`, missing the four writers in `downloads/` that actually construct the row, and asserted `not offenders`, which enforces nothing once a refactor moves the writers out of its view. It now walks assignments AND keywords across `app/`, and holds exact set equality against a registry of all twelve write sites with each one's value. test_body_limit.py's release guard used `any(... ast.walk(fn))`, which accepts a `release_read_connection` that runs AFTER the await it exists to precede — satisfying the letter of the rule while pinning a connection for exactly the transfer it was supposed to free. Position is checked now, and a late release is its own, louder failure. The registry keyed sites by file basename while scanning recursively, and `admin.py`, `discover.py` and `playlists.py` each exist both at the top level and under `plex/` — the same entry-collapse the registry was widened to close. Keyed by relative path now. And all three guards recognised a body read only by the `capped_` naming convention, so a route spelling `await request.body()` inline declared no ceiling, released no connection, and was invisible to every one of them. `release_read_connection`'s precondition claimed the caller "must have NOTHING PENDING, and that is checked rather than described" while checking only UNFLUSHED ORM work. Measured: after `add`+`flush` all three collections are empty and the rollback loses the row, and Core DML never enters them at all — both of them this codebase's canonical write shapes. It now also asks the transaction directly via `pg_current_xact_id_if_assigned()`. `errors.tooLarge` was reworded to "try a smaller file", but that key answers every route's body cap: the issue composer's over-size JSON draft got advice about a file it does not have. Back to payload-neutral; the file-specific sentence keeps its own key. The upload dialog's new danger border outranked `dragOver`, so a user replacing an over-cap file got no sign the drop zone was live — and with a file picked the label shows its name rather than "drop it here", so the border was the only feedback left. Four further findings were refuted 3/3 and are not fixed: `aria-invalid` on the drop zone (measured in Chromium — the AX node does carry it on a native button), a bidi-override spoof through `display_text` (the served filename already strips those categories, and the proposed filter would have broken every ZWJ emoji), and `capped_body` not draining (the screenshot route's edge ceiling equals its own cap, so that refusal is unreachable, and the log sink is sent by beacon with nobody reading the answer).
This commit is contained in:
@@ -480,34 +480,112 @@ def test_a_long_display_name_is_truncated_on_EVERY_route_that_takes_one(
|
||||
assert patched.json()["display_uploader"] == bound
|
||||
|
||||
|
||||
def test_no_route_assigns_a_display_field_without_the_shared_helper():
|
||||
"""The drift guard for the test above: it holds five routes, this holds the rule.
|
||||
def _display_write_sites() -> dict[tuple[str, str], set[str]]:
|
||||
"""Every place under `backend/app/` that puts a value into a display field, as
|
||||
(module relative to `app/`, enclosing function) -> the set of value expressions written there.
|
||||
|
||||
Parsed, not grepped. Every `display_name = ...` / `display_uploader = ...` assignment under
|
||||
`app/routes/` takes a string a user typed and puts it on its way to a varchar(255) column, so
|
||||
every one of them must go through `display_text`. A sixth route added tomorrow fails here
|
||||
instead of 500ing in production.
|
||||
TWO SHAPES, because the first version of this guard walked only `ast.Assign` and was therefore
|
||||
blind to the shape the upload route itself uses — `SpooledUpload(..., display_name=...)` is an
|
||||
`ast.keyword`, not an assignment, so the one site with an unbounded REQUEST HEADER behind it was
|
||||
the site the guard could not see (deep-review round 3).
|
||||
|
||||
ANNOTATED assignments are excluded on purpose: `display_name: Mapped[str | None] =
|
||||
mapped_column(...)` in `models.py` and `display_name: str | None = None` in a Pydantic model are
|
||||
declarations of the field, not writes into it.
|
||||
|
||||
The whole of `app/`, not just `app/routes/`: a job row is constructed in `downloads/service.py`
|
||||
and `downloads/upload.py`, so a walk stopping at the HTTP boundary misses the writers that
|
||||
actually reach the column.
|
||||
"""
|
||||
routes_dir = pathlib.Path(__file__).resolve().parents[1] / "app" / "routes"
|
||||
offenders: list[str] = []
|
||||
for py in sorted(routes_dir.rglob("*.py")):
|
||||
app_dir = pathlib.Path(__file__).resolve().parents[1] / "app"
|
||||
found: dict[tuple[str, str], set[str]] = {}
|
||||
for py in sorted(app_dir.rglob("*.py")):
|
||||
module = py.relative_to(app_dir).as_posix()
|
||||
tree = ast.parse(py.read_text("utf-8"), filename=str(py))
|
||||
enclosing: dict[int, str] = {}
|
||||
for fn in ast.walk(tree):
|
||||
if isinstance(fn, (ast.AsyncFunctionDef, ast.FunctionDef)):
|
||||
for node in ast.walk(fn):
|
||||
enclosing.setdefault(id(node), fn.name)
|
||||
for node in ast.walk(tree):
|
||||
if not isinstance(node, ast.Assign):
|
||||
if isinstance(node, ast.Assign):
|
||||
names = {
|
||||
t.id if isinstance(t, ast.Name) else t.attr
|
||||
for t in node.targets
|
||||
if isinstance(t, (ast.Name, ast.Attribute))
|
||||
}
|
||||
value = node.value
|
||||
elif isinstance(node, ast.keyword):
|
||||
names = {node.arg} if node.arg else set()
|
||||
value = node.value
|
||||
else:
|
||||
continue
|
||||
names = {
|
||||
t.id if isinstance(t, ast.Name) else t.attr
|
||||
for t in node.targets
|
||||
if isinstance(t, (ast.Name, ast.Attribute))
|
||||
}
|
||||
if not names & {"display_name", "display_uploader"}:
|
||||
continue
|
||||
value = ast.unparse(node.value)
|
||||
if "display_text" not in value:
|
||||
offenders.append(f"{py.name}:{node.lineno}: {value}")
|
||||
assert not offenders, (
|
||||
"a display field is assigned without models.display_text, so a name longer "
|
||||
f"than the column is a 500 rather than a truncation: {offenders}"
|
||||
key = (module, enclosing.get(id(node), "<module>"))
|
||||
found.setdefault(key, set()).add(ast.unparse(value))
|
||||
return found
|
||||
|
||||
|
||||
# Every write into a display field, with the expression each one writes. A varchar(255) column with
|
||||
# a request header behind it turns a too-long name into a psycopg DataError — a 500 AFTER the whole
|
||||
# file has transferred — so each of these has to be bounded, and each one here has been read.
|
||||
#
|
||||
# Bounded by `display_text` unless noted:
|
||||
# auth.get_or_create_demo_user a literal.
|
||||
# service.redownload CLEARS the field so the fresh metadata wins — `None` is the
|
||||
# one value that needs no bound, and a clear is still a write.
|
||||
# service.enqueue / enqueue_edit `display_name` is already bounded by its caller; the fallback
|
||||
# slices the YouTube title to the same width.
|
||||
# downloads.upload_download passes through `spool_upload`'s value, bounded there.
|
||||
#
|
||||
# Held as EXACT EQUALITY rather than as a list of offenders. The offender form could only fail
|
||||
# upward: refactoring the writers into a shape the walker did not recognise left it asserting
|
||||
# `not []` and enforcing nothing, which is how a drift guard dies quietly (round 3). This one fails
|
||||
# when a site appears, when one disappears, and when one changes what it writes.
|
||||
DISPLAY_WRITE_SITES = {
|
||||
("auth.py", "get_or_create_demo_user"): {"'Demo'"},
|
||||
("downloads/service.py", "redownload"): {"None"},
|
||||
("downloads/service.py", "enqueue"): {
|
||||
"display_name or (asset.title[:DISPLAY_TEXT_MAX] "
|
||||
"if asset.status == 'ready' and asset.title else None)"
|
||||
},
|
||||
("downloads/service.py", "enqueue_edit"): {"display_name or asset.title[:DISPLAY_TEXT_MAX]"},
|
||||
("downloads/upload.py", "ingest_upload"): {"display_text(display_name)"},
|
||||
("routes/auth.py", "callback"): {"display_text(userinfo.get('name'))"},
|
||||
("routes/auth.py", "_complete_link"): {"display_text(userinfo.get('name'))"},
|
||||
("routes/downloads.py", "enqueue_download"): {"display_text(body.display_name)"},
|
||||
("routes/downloads.py", "enqueue_edit"): {"display_text(body.display_name)"},
|
||||
("routes/downloads.py", "update_download_meta"): {
|
||||
"display_text(body.display_name)",
|
||||
"display_text(body.display_uploader)",
|
||||
},
|
||||
("routes/downloads.py", "upload_download"): {"upload.display_name"},
|
||||
("routes/downloads.py", "spool_upload"): {
|
||||
"display_text(_header_text(request, UPLOAD_DISPLAY_NAME_HEADER))"
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def test_no_code_path_assigns_a_display_field_without_the_shared_helper():
|
||||
"""The drift guard for the test above: it holds five routes, this holds the rule.
|
||||
|
||||
Parsed, not grepped. Every value that reaches a display field takes a string a user typed (or
|
||||
YouTube supplied) and puts it on its way to a varchar(255) column, so every one of them must be
|
||||
bounded. A writer added tomorrow fails here instead of 500ing in production.
|
||||
"""
|
||||
found = _display_write_sites()
|
||||
moved = set(found) ^ set(DISPLAY_WRITE_SITES)
|
||||
changed = {
|
||||
key: (found[key], DISPLAY_WRITE_SITES[key])
|
||||
for key in set(found) & set(DISPLAY_WRITE_SITES)
|
||||
if found[key] != DISPLAY_WRITE_SITES[key]
|
||||
}
|
||||
assert found == DISPLAY_WRITE_SITES, (
|
||||
"a display-field write appeared, moved or changed shape. Every value reaching "
|
||||
"display_name/display_uploader must be bounded to models.DISPLAY_TEXT_MAX — normally by "
|
||||
"models.display_text — or a name longer than the column is a 500 rather than a truncation. "
|
||||
f"Sites that appeared or vanished: {moved}. Sites that changed: {changed}."
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user