fix(i18n): write each refusal where it is sent, and read only that form
A refusal's detail now stands where it is sent (HTTPException's detail, a JSONResponse's "detail") or is returned by a factory that only ever is, with any spread before its code and reason, so no spread can replace them. The scan follows no variable and no kwargs any more; what it cannot read in a reason (a lambda's or comprehension's name, a nonlocal or global, a local that reaches itself, a constant or map changed elsewhere, an app import it cannot resolve) is reported. The detail guard reads the same slots and fails on anything but a code dict or a factory's call.
This commit is contained in:
+239
-270
@@ -2,12 +2,19 @@
|
||||
with every reason its code can be sent with.
|
||||
|
||||
The frontend's contract (`i18n/backendIds.contract.test.ts`) runs this and requires a sentence for each
|
||||
code and reason in both languages. A regex over the source read only the shapes it was written for, and a
|
||||
scan that tries to follow every shape opens a silent gap with each one it learns. So this one reads a
|
||||
CLOSED list, the shapes the app writes, and reports everything else, never guesses:
|
||||
code and reason in both languages. A scan that tries to follow every shape opens a silent gap with each one
|
||||
it learns, and following how a value flows through Python (variables, scopes, aliases, a dict changed in
|
||||
place) never closes. So the CODE keeps to a narrow form, and this reads a CLOSED list of it, reporting
|
||||
everything else, never guessing:
|
||||
|
||||
- where a detail stands: where it is sent — `HTTPException`'s second argument or `detail=`, the `"detail"`
|
||||
value of a `JSONResponse`'s content dict — through the arms of `a if c else b`, or as what a module-level
|
||||
factory returns when every return of it is a detail and every reference to it is a call sent as one. A
|
||||
detail held in a variable is not followed;
|
||||
- the code: one literal. A `**` spread, or a key that is not a literal, stands BEFORE a literal `code` and
|
||||
`reason`, both of which the detail then carries: the later key wins, so no spread can change them;
|
||||
- a reason: a string literal (`None` sends none: the site counts as reasonless); both arms of
|
||||
`a if c else b`; a local variable bound only by plain `x = …`; a module constant reached as `mod.NAME`;
|
||||
`a if c else b`; a function's local bound only by plain `x = …`; a module constant reached as `mod.NAME`;
|
||||
a call's `Literal` return annotation;
|
||||
- a parameter typed with a `Literal` (basedpyright holds every value passed in), or a `str` one of a
|
||||
module-level helper (`refused`, `not_found`), followed to every call, each of which must pass it by
|
||||
@@ -15,17 +22,15 @@ CLOSED list, the shapes the app writes, and reports everything else, never guess
|
||||
- an attribute: `e.x` of the exception an `except C as e` catches, or of a parameter typed `C`, when a
|
||||
method of `C` declares `self.x: <Literal>`;
|
||||
- a map lookup (`_SETTING_REASONS[exc.reason]`): the map's values, and its keys must be exactly what the
|
||||
index can be;
|
||||
- the code: one literal, never changed after. A `**` spread in the detail is the `**kwargs` of a helper or
|
||||
of an exception's `__init__` that takes `reason` by name, used for nothing but being spread, and no call
|
||||
of it passes `code=` or a `**` that is not such a spread itself. The detail's variable is changed only by
|
||||
`|= {…}` or `["reason"] = …`, and goes nowhere but into an `HTTPException` / `JSONResponse`.
|
||||
index can be.
|
||||
|
||||
A name bound twice at module level, inside a module-level `if`/`try`/loop, or imported from two places is
|
||||
not looked up. A reference named like a followed helper that the scan cannot resolve (`import app.x`, a
|
||||
relative import) is unread. Anything else is UNREAD, or an unread code, and the contract fails on it unless
|
||||
no screen shows that code. Stdlib only: it runs on the host, beside the frontend's tests
|
||||
(`python -I backend/tests/refusal_sites.py` → JSON).
|
||||
A name bound twice at module level, inside a module-level `if`/`try`/loop, named by a `global`, or imported
|
||||
from two places is not looked up; nor is a local a lambda or a comprehension binds, or one a `nonlocal` or
|
||||
`global` names. A module constant or map that anything stores into, deletes from or (a map) uses but by
|
||||
`map[k]` or a read method is unread. A reference named like a followed helper that the scan cannot resolve
|
||||
(`import app.x`, a relative import, an app module that does not bind it) is unread. Anything else is UNREAD,
|
||||
or an unread code, and the contract fails on it unless no screen shows that code. Stdlib only: it runs on
|
||||
the host, beside the frontend's tests (`python -I backend/tests/refusal_sites.py` → JSON).
|
||||
"""
|
||||
|
||||
import ast
|
||||
@@ -43,13 +48,13 @@ NOT_REFUSALS = {
|
||||
"plex/tmdb.py": "a production country's ISO code",
|
||||
}
|
||||
|
||||
# What a detail held in a variable may be handed to: the response it is sent in, nothing that could change it.
|
||||
SENDERS = {"HTTPException", "JSONResponse"}
|
||||
READS = {"get", "items", "keys", "values", "copy"}
|
||||
# What a map may be used by besides `map[k]`: a method that only reads it.
|
||||
READS = {"get", "items", "keys", "values"}
|
||||
|
||||
Node = ast.AST
|
||||
FUNCS = (ast.FunctionDef, ast.AsyncFunctionDef)
|
||||
SCOPES = (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda, ast.ClassDef)
|
||||
COMPREHENSIONS = (ast.ListComp, ast.SetComp, ast.DictComp, ast.GeneratorExp)
|
||||
COMPOUND = (
|
||||
ast.If,
|
||||
ast.Try,
|
||||
@@ -82,7 +87,10 @@ class Module:
|
||||
# local name → (module, symbol or None for the module)
|
||||
imports: dict[str, tuple[str, str | None]]
|
||||
top: dict[str, Node] # module-level bindings: def, class, or the assignment
|
||||
ambiguous: set[str] # bound more than once, conditionally, or imported from two places
|
||||
ambiguous: set[
|
||||
str
|
||||
] # bound more than once, conditionally, by a `global`, or imported from two places
|
||||
slots: set[int] # the ids of the nodes sent as a detail
|
||||
|
||||
|
||||
def _own_scope(node: Node) -> Iterator[Node]:
|
||||
@@ -120,6 +128,39 @@ def _binds(node: Node) -> set[str]:
|
||||
return set()
|
||||
|
||||
|
||||
def _sender(call: ast.Call) -> str | None:
|
||||
return getattr(call.func, "id", None) or getattr(call.func, "attr", None)
|
||||
|
||||
|
||||
def detail_slots(tree: Node) -> Iterator[tuple[ast.Call, Node | None]]:
|
||||
"""Every place a detail is sent from, with what it holds: `HTTPException`'s second argument or
|
||||
`detail=` (None: sent without one), and the `"detail"` value of a `JSONResponse`'s content dict (a
|
||||
content without one is an answer, not a refusal)."""
|
||||
for node in ast.walk(tree):
|
||||
if not isinstance(node, ast.Call):
|
||||
continue
|
||||
kw = {k.arg: k.value for k in node.keywords}
|
||||
if _sender(node) == "HTTPException":
|
||||
yield node, kw.get("detail", node.args[1] if len(node.args) > 1 else None)
|
||||
elif _sender(node) == "JSONResponse":
|
||||
content = kw.get("content", node.args[0] if node.args else None)
|
||||
if isinstance(content, ast.Dict):
|
||||
for k, v in zip(content.keys, content.values, strict=True):
|
||||
if isinstance(k, ast.Constant) and k.value == "detail":
|
||||
yield node, v
|
||||
|
||||
|
||||
def _code_dict(node: Node) -> bool:
|
||||
"""A `{"code": …}` dict, or a `dict(code=…)` call."""
|
||||
if isinstance(node, ast.Dict):
|
||||
return any(isinstance(k, ast.Constant) and k.value == "code" for k in node.keys)
|
||||
return (
|
||||
isinstance(node, ast.Call)
|
||||
and getattr(node.func, "id", None) == "dict"
|
||||
and any(k.arg == "code" for k in node.keywords)
|
||||
)
|
||||
|
||||
|
||||
def _module_name(path: str) -> str:
|
||||
parts = path.removesuffix(".py").split("/")
|
||||
if parts[-1] == "__init__":
|
||||
@@ -133,6 +174,7 @@ def _load(sources: dict[str, str]) -> dict[str, Module]:
|
||||
tree = ast.parse(src)
|
||||
parents = {child: node for node in ast.walk(tree) for child in ast.iter_child_nodes(node)}
|
||||
origins: dict[str, set[tuple[str, str | None]]] = {}
|
||||
ambiguous: set[str] = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
|
||||
for a in node.names:
|
||||
@@ -141,9 +183,11 @@ def _load(sources: dict[str, str]) -> dict[str, Module]:
|
||||
for a in node.names:
|
||||
if a.asname:
|
||||
origins.setdefault(a.asname, set()).add((a.name, None))
|
||||
elif isinstance(node, ast.Global):
|
||||
ambiguous |= set(node.names)
|
||||
top: dict[str, Node] = {}
|
||||
count: dict[str, int] = {}
|
||||
ambiguous = {name for name, o in origins.items() if len(o) > 1}
|
||||
ambiguous |= {name for name, o in origins.items() if len(o) > 1}
|
||||
for stmt in tree.body:
|
||||
if isinstance(stmt, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
top[stmt.name] = stmt
|
||||
@@ -160,8 +204,9 @@ def _load(sources: dict[str, str]) -> dict[str, Module]:
|
||||
ambiguous.add(name)
|
||||
ambiguous |= {name for name, c in count.items() if c > 1}
|
||||
imports = {name: next(iter(o)) for name, o in origins.items() if name not in ambiguous}
|
||||
slots = {id(d) for _, d in detail_slots(tree) if d is not None}
|
||||
name = _module_name(path)
|
||||
mods[name] = Module(path, name, tree, parents, imports, top, ambiguous)
|
||||
mods[name] = Module(path, name, tree, parents, imports, top, ambiguous, slots)
|
||||
return mods
|
||||
|
||||
|
||||
@@ -171,7 +216,9 @@ class Scan:
|
||||
self.errors: list[str] = []
|
||||
self._following: set[tuple[str, str, str]] = set()
|
||||
self._followed: dict[tuple[str, str, str], Res] = {}
|
||||
self._clean: dict[int, bool] = {}
|
||||
self._resolving: set[int] = set()
|
||||
self._factory: dict[int, bool] = {}
|
||||
self._changed: dict[int, bool] = {}
|
||||
|
||||
# --- names ---------------------------------------------------------------------------------------
|
||||
|
||||
@@ -195,26 +242,31 @@ class Scan:
|
||||
|
||||
def lookup_expr(self, m: Module, e: Node) -> tuple[Module, Node] | tuple[str, None] | None:
|
||||
if isinstance(e, ast.Name):
|
||||
return None if self.local(m, e) else self.lookup(m, e.id)
|
||||
return None if self.binder(m, e) is not None else self.lookup(m, e.id)
|
||||
if isinstance(e, ast.Attribute):
|
||||
base = self.lookup_expr(m, e.value)
|
||||
if base and base[1] is None and isinstance(base[0], str):
|
||||
return self.lookup(self.mods[base[0]], e.attr)
|
||||
return None
|
||||
|
||||
def local(self, m: Module, e: ast.Name) -> bool:
|
||||
"""`e` names a parameter or a variable of a function it sits in."""
|
||||
fn = self.enclosing(m, e, FUNCS)
|
||||
while fn is not None:
|
||||
a = fn.args
|
||||
if e.id in {
|
||||
p.arg for p in [*a.posonlyargs, *a.args, *a.kwonlyargs, a.vararg, a.kwarg] if p
|
||||
}:
|
||||
return True
|
||||
if self.assignments(fn, e.id):
|
||||
return True
|
||||
fn = self.enclosing(m, fn, FUNCS)
|
||||
return False
|
||||
def binder(self, m: Module, e: ast.Name) -> Node | None:
|
||||
"""The innermost scope `e` is bound in: a function (a parameter, or a binding in its own body), a
|
||||
lambda (a parameter), a comprehension (its target); None: a module-level name."""
|
||||
node = m.parents.get(e)
|
||||
while node is not None:
|
||||
if isinstance(node, (*FUNCS, ast.Lambda)):
|
||||
a = node.args
|
||||
params = [*a.posonlyargs, *a.args, *a.kwonlyargs, a.vararg, a.kwarg]
|
||||
if e.id in {p.arg for p in params if p}:
|
||||
return node
|
||||
if isinstance(node, FUNCS) and self.assignments(node, e.id):
|
||||
return node
|
||||
elif isinstance(node, COMPREHENSIONS) and any(
|
||||
e.id in _stored(g.target) for g in node.generators
|
||||
):
|
||||
return node
|
||||
node = m.parents.get(node)
|
||||
return None
|
||||
|
||||
def at(self, m: Module, e: Node) -> str:
|
||||
return f"{m.path}:{getattr(e, 'lineno', '?')}: {ast.unparse(e)}"
|
||||
@@ -269,17 +321,56 @@ class Scan:
|
||||
root = node
|
||||
while isinstance(root, ast.Attribute):
|
||||
root = root.value
|
||||
if not isinstance(root, ast.Name) or self.local(m, root):
|
||||
if not isinstance(root, ast.Name) or self.binder(m, root) is not None:
|
||||
return False
|
||||
if root.id in m.ambiguous or (root.id not in m.top and root.id not in m.imports):
|
||||
return True
|
||||
if root is node or root.id in m.top:
|
||||
if root.id in m.top:
|
||||
return False
|
||||
if root is node: # imported from an app module that does not bind it where the scan can see
|
||||
mod = m.imports[root.id][0]
|
||||
return (mod == "app" or mod.startswith("app.")) and self.lookup(m, root.id) is None
|
||||
found = self.lookup(m, root.id)
|
||||
return (
|
||||
found is not None and found[1] is None
|
||||
) # an app module whose dotted path went nowhere
|
||||
|
||||
def changed(self, m: Module, stmt: Node, name: str, *, only_read: bool) -> bool:
|
||||
"""Something in the app stores into, deletes or rebinds the module-level `name` that `stmt` binds,
|
||||
or (`only_read`, a map) uses it but by `map[k]` or a method that only reads it."""
|
||||
key = id(stmt) * 2 + only_read
|
||||
if key in self._changed:
|
||||
return self._changed[key]
|
||||
own = stmt.targets if isinstance(stmt, ast.Assign) else [getattr(stmt, "target", None)]
|
||||
out = False
|
||||
for cm in self.mods.values():
|
||||
for node in ast.walk(cm.tree):
|
||||
if out:
|
||||
break
|
||||
if not (
|
||||
isinstance(node, ast.Name)
|
||||
and node.id == name
|
||||
or isinstance(node, ast.Attribute)
|
||||
and node.attr == name
|
||||
) or (cm is m and node in own):
|
||||
continue
|
||||
found = self.lookup_expr(cm, node)
|
||||
if not (found and found[0] is m and found[1] is stmt):
|
||||
continue
|
||||
if not isinstance(node.ctx, ast.Load):
|
||||
out = True
|
||||
elif only_read:
|
||||
p = cm.parents.get(node)
|
||||
out = not (
|
||||
isinstance(p, ast.Subscript)
|
||||
and p.value is node
|
||||
and isinstance(p.ctx, ast.Load)
|
||||
or isinstance(p, ast.Attribute)
|
||||
and p.attr in READS
|
||||
)
|
||||
self._changed[key] = out
|
||||
return out
|
||||
|
||||
# --- types ---------------------------------------------------------------------------------------
|
||||
|
||||
def ann_values(self, m: Module, ann: Node | None, depth: int = 0) -> set[str] | None:
|
||||
@@ -352,30 +443,33 @@ class Scan:
|
||||
return Res(unread=[self.at(m, e)])
|
||||
|
||||
def resolve_name(self, m: Module, e: ast.Name) -> Res:
|
||||
fn = self.enclosing(m, e, FUNCS)
|
||||
while fn is not None:
|
||||
assert isinstance(fn, FUNCS)
|
||||
if self.caught_as(m, e, e.id) is not None:
|
||||
return Res(unread=[self.at(m, e)]) # the exception itself, not a reason
|
||||
a = fn.args
|
||||
params = {p.arg: p for p in [*a.posonlyargs, *a.args, *a.kwonlyargs]}
|
||||
assigned = self.assignments(fn, e.id)
|
||||
if e.id in params:
|
||||
if assigned:
|
||||
return Res(unread=[f"{self.at(m, e)} (a parameter its body binds again)"])
|
||||
return self.resolve_param(m, fn, params[e.id], e)
|
||||
if self.caught_as(m, e, e.id) is not None:
|
||||
return Res(unread=[self.at(m, e)]) # the exception itself, not a reason
|
||||
fn = self.binder(m, e)
|
||||
if fn is None:
|
||||
return Res(unread=[self.at(m, e)])
|
||||
if not isinstance(fn, FUNCS):
|
||||
return Res(unread=[f"{self.at(m, e)} (bound by a lambda or a comprehension)"])
|
||||
if any(isinstance(n, (ast.Nonlocal, ast.Global)) and e.id in n.names for n in ast.walk(fn)):
|
||||
return Res(unread=[f"{self.at(m, e)} (named by a nonlocal or a global)"])
|
||||
a = fn.args
|
||||
params = {p.arg: p for p in [*a.posonlyargs, *a.args, *a.kwonlyargs]}
|
||||
assigned = self.assignments(fn, e.id)
|
||||
if e.id in params:
|
||||
if assigned:
|
||||
out = Res()
|
||||
for s in assigned:
|
||||
if isinstance(s, ast.Assign) and all(
|
||||
isinstance(t, ast.Name) for t in s.targets
|
||||
):
|
||||
out = out | self.resolve(m, s.value)
|
||||
else:
|
||||
out = out | Res(unread=[self.at(m, e)])
|
||||
return out
|
||||
fn = self.enclosing(m, fn, FUNCS)
|
||||
return Res(unread=[self.at(m, e)])
|
||||
return Res(unread=[f"{self.at(m, e)} (a parameter its body binds again)"])
|
||||
return self.resolve_param(m, fn, params[e.id], e)
|
||||
out = Res(unread=[] if assigned else [self.at(m, e)]) # `*args`, `**kwargs`
|
||||
for s in assigned:
|
||||
if not (isinstance(s, ast.Assign) and all(isinstance(t, ast.Name) for t in s.targets)):
|
||||
out = out | Res(unread=[self.at(m, e)])
|
||||
elif id(s) in self._resolving:
|
||||
out = out | Res(unread=[f"{self.at(m, e)} (reached again while it is resolved)"])
|
||||
else:
|
||||
self._resolving.add(id(s))
|
||||
out = out | self.resolve(m, s.value)
|
||||
self._resolving.discard(id(s))
|
||||
return out
|
||||
|
||||
def resolve_param(self, m: Module, fn: Node, param: ast.arg, e: Node) -> Res:
|
||||
v = self.ann_values(m, param.annotation)
|
||||
@@ -443,10 +537,11 @@ class Scan:
|
||||
found
|
||||
and isinstance(found[0], Module)
|
||||
and isinstance(found[1], (ast.Assign, ast.AnnAssign))
|
||||
and isinstance(found[1].value, ast.Constant)
|
||||
):
|
||||
stmt = found[1]
|
||||
if isinstance(stmt.value, ast.Constant):
|
||||
return self.resolve(found[0], stmt.value)
|
||||
if self.changed(found[0], found[1], e.attr, only_read=False):
|
||||
return Res(unread=[f"{self.at(m, e)} (bound again or deleted elsewhere)"])
|
||||
return self.resolve(found[0], found[1].value)
|
||||
return Res(unread=[self.at(m, e)])
|
||||
|
||||
def resolve_lookup(self, m: Module, e: ast.Subscript) -> Res:
|
||||
@@ -462,6 +557,9 @@ class Scan:
|
||||
isinstance(k, ast.Constant) and isinstance(k.value, str) for k in table.keys
|
||||
):
|
||||
return Res(unread=[self.at(m, e)])
|
||||
name = e.value.attr if isinstance(e.value, ast.Attribute) else getattr(e.value, "id", "")
|
||||
if self.changed(found[0], found[1], name, only_read=True):
|
||||
return Res(unread=[f"{self.at(m, e)} (the map is changed after it is built)"])
|
||||
out = Res()
|
||||
for v in table.values:
|
||||
out = out | self.resolve(found[0], v)
|
||||
@@ -474,119 +572,57 @@ class Scan:
|
||||
)
|
||||
return out
|
||||
|
||||
# --- spreads -------------------------------------------------------------------------------------
|
||||
|
||||
def clean_spread(self, m: Module, at: Node, src: Node) -> bool:
|
||||
"""`**src` cannot carry a `reason` or a `code`: it is a function's `**kwargs` (or a caught
|
||||
exception's `e.params` kept from its `__init__`'s), or `{k: … for k, v in that.items()}`: a
|
||||
value may change, a key may not."""
|
||||
if isinstance(src, ast.DictComp):
|
||||
g = src.generators
|
||||
if len(g) != 1 or not isinstance(g[0].target, ast.Tuple) or len(g[0].target.elts) != 2:
|
||||
return False
|
||||
k, v = g[0].target.elts
|
||||
it = g[0].iter
|
||||
return (
|
||||
isinstance(k, ast.Name)
|
||||
and isinstance(v, ast.Name)
|
||||
and getattr(src.key, "id", None) == k.id
|
||||
and isinstance(it, ast.Call)
|
||||
and isinstance(it.func, ast.Attribute)
|
||||
and it.func.attr == "items"
|
||||
and not it.args
|
||||
and self.clean_spread(m, at, it.func.value)
|
||||
)
|
||||
if isinstance(src, ast.Attribute) and isinstance(src.value, ast.Name):
|
||||
typed = self.typed(m, at, src.value.id)
|
||||
return bool(typed) and all(self.kept(cm, cls, src.attr) for cm, cls in typed or [])
|
||||
if isinstance(src, ast.Name):
|
||||
fn = self.enclosing(m, at, FUNCS)
|
||||
return (
|
||||
fn is not None
|
||||
and fn.args.kwarg is not None
|
||||
and src.id == fn.args.kwarg.arg
|
||||
and self.kwargs_clean(m, fn, fn)
|
||||
)
|
||||
return False
|
||||
|
||||
def kept(self, cm: Module, cls: ast.ClassDef, attr: str) -> bool:
|
||||
"""`self.attr` is only ever the `**kwargs` of the class's `__init__`, and that one is clean."""
|
||||
init = next(
|
||||
(s for s in cls.body if isinstance(s, ast.FunctionDef) and s.name == "__init__"), None
|
||||
)
|
||||
if init is None or init.args.kwarg is None:
|
||||
return False
|
||||
stores = [
|
||||
n
|
||||
for n in ast.walk(cls)
|
||||
if isinstance(n, ast.Attribute)
|
||||
and n.attr == attr
|
||||
and getattr(n.value, "id", None) == "self"
|
||||
and not isinstance(n.ctx, ast.Load)
|
||||
]
|
||||
kept = [
|
||||
s
|
||||
for s in stores
|
||||
if isinstance(p := cm.parents.get(s), ast.Assign)
|
||||
and getattr(p.value, "id", None) == init.args.kwarg.arg
|
||||
and self.enclosing(cm, p, FUNCS) is init
|
||||
]
|
||||
return len(stores) == 1 and len(kept) == 1 and self.kwargs_clean(cm, init, cls)
|
||||
|
||||
def kwargs_clean(self, m: Module, fn: Node, target: Node) -> bool:
|
||||
"""`fn`'s `**kwargs` holds no `reason` (`fn` takes it by name) and no `code`: it is used for
|
||||
nothing but `**kwargs`, `kwargs.items()`, `self.x = kwargs` or an f-string, and every call of
|
||||
`target` (the helper, or the class `fn` is the `__init__` of) passes no `code=` and no unclean
|
||||
`**`."""
|
||||
assert isinstance(fn, FUNCS) and fn.args.kwarg is not None
|
||||
if id(fn) in self._clean:
|
||||
return self._clean[id(fn)]
|
||||
self._clean[id(fn)] = False # a call that reaches it again while it is checked is not clean
|
||||
kwarg = fn.args.kwarg.arg
|
||||
clean = "reason" in {a.arg for a in [*fn.args.args, *fn.args.kwonlyargs]} and not (
|
||||
self.assignments(fn, kwarg)
|
||||
)
|
||||
for n in _own_scope(fn):
|
||||
if not clean:
|
||||
break
|
||||
if isinstance(n, ast.Name) and n.id == kwarg:
|
||||
p = m.parents.get(n)
|
||||
clean = (
|
||||
isinstance(p, ast.FormattedValue) # printed in a message
|
||||
or isinstance(p, ast.keyword)
|
||||
and p.arg is None
|
||||
or isinstance(p, ast.Dict)
|
||||
and any(k is None and v is n for k, v in zip(p.keys, p.values, strict=True))
|
||||
or isinstance(p, ast.Attribute)
|
||||
and p.attr == "items"
|
||||
and isinstance(m.parents.get(p), ast.Call)
|
||||
or isinstance(p, ast.Assign)
|
||||
and p.value is n
|
||||
and all(
|
||||
isinstance(t, ast.Attribute) and getattr(t.value, "id", None) == "self"
|
||||
for t in p.targets
|
||||
)
|
||||
)
|
||||
refs = list(self.references(m, target)) if clean else []
|
||||
for cm, node, call, resolved in refs:
|
||||
if call is not None:
|
||||
clean = not any(
|
||||
k.arg == "code" or k.arg is None and not self.clean_spread(cm, call, k.value)
|
||||
for k in call.keywords
|
||||
)
|
||||
elif not resolved or not isinstance(target, ast.ClassDef):
|
||||
clean = False # a helper passed around, or a name the scan cannot resolve
|
||||
else:
|
||||
# `except C`, an annotation: not a construction; a subclass is built some other way
|
||||
cls = self.enclosing(cm, node, ast.ClassDef)
|
||||
clean = not (cls is not None and any(node in ast.walk(b) for b in cls.bases))
|
||||
if not clean:
|
||||
break
|
||||
self._clean[id(fn)] = clean
|
||||
return clean
|
||||
|
||||
# --- sites ---------------------------------------------------------------------------------------
|
||||
|
||||
def is_detail(self, node: Node | None) -> bool:
|
||||
if isinstance(node, ast.IfExp):
|
||||
return self.is_detail(node.body) and self.is_detail(node.orelse)
|
||||
return _code_dict(node)
|
||||
|
||||
def factory(self, m: Module, fn: Node) -> bool:
|
||||
"""A module-level function each return of which is a detail, and each reference to which is a
|
||||
call sent as one (`quota_detail`)."""
|
||||
assert isinstance(fn, FUNCS)
|
||||
if id(fn) in self._factory:
|
||||
return self._factory[id(fn)]
|
||||
returns = [n for n in _own_scope(fn) if isinstance(n, ast.Return)]
|
||||
ok = (
|
||||
m.top.get(fn.name) is fn # a name bound twice: its calls are unresolved below
|
||||
and bool(returns)
|
||||
and all(self.is_detail(r.value) for r in returns)
|
||||
)
|
||||
for cm, _, call, resolved in self.references(m, fn) if ok else []:
|
||||
ok = resolved and id(call) in cm.slots # a bare reference is no call: never sent
|
||||
if not ok:
|
||||
break
|
||||
self._factory[id(fn)] = ok
|
||||
return ok
|
||||
|
||||
def placed(self, m: Module, d: Node) -> bool:
|
||||
"""`d` is sent as a detail: from a sender's slot, through the arms of `a if c else b`, or as what a
|
||||
factory returns."""
|
||||
holder, p = d, m.parents.get(d)
|
||||
while isinstance(p, ast.IfExp) and holder is not p.test:
|
||||
holder, p = p, m.parents.get(p)
|
||||
if id(holder) in m.slots:
|
||||
return True
|
||||
fn = self.enclosing(m, p, FUNCS) if isinstance(p, ast.Return) else None
|
||||
return fn is not None and self.factory(m, fn)
|
||||
|
||||
def sayable(self, m: Module, d: Node | None) -> bool:
|
||||
"""What a slot holds is a detail the client can say: a code dict, or a factory's call."""
|
||||
if isinstance(d, ast.IfExp):
|
||||
return self.sayable(m, d.body) and self.sayable(m, d.orelse)
|
||||
if _code_dict(d):
|
||||
return True
|
||||
found = self.lookup_expr(m, d.func) if isinstance(d, ast.Call) else None
|
||||
return bool(
|
||||
found
|
||||
and isinstance(found[0], Module)
|
||||
and isinstance(found[1], FUNCS)
|
||||
and self.factory(found[0], found[1])
|
||||
)
|
||||
|
||||
def entries(self, d: Node) -> list[tuple[str | None, Node]]:
|
||||
"""A dict's (key, value) pairs: None for a `**` spread, "?" for a key that is not a literal."""
|
||||
if isinstance(d, ast.Dict):
|
||||
@@ -610,14 +646,16 @@ class Scan:
|
||||
for v in (v for k, v in items if k == "code"):
|
||||
code = code | self.resolve(m, v)
|
||||
problems = [
|
||||
f"{self.at(m, v)} (a key the scan cannot read)" for k, v in items if k == "?"
|
||||
] + [
|
||||
f"{self.at(m, v)} (a spread that may carry the code)"
|
||||
f"{self.at(m, v)} (a spread in dict())"
|
||||
for k, v in items
|
||||
if k is None and not self.clean_spread(m, d, v)
|
||||
if k is None and isinstance(d, ast.Call)
|
||||
] + [
|
||||
f"{self.at(m, v)} (a spread or a computed key the code and the reason do not follow)"
|
||||
for i, (k, v) in enumerate(items)
|
||||
if k in (None, "?") and not {"code", "reason"} <= {k2 for k2, _ in items[i + 1 :]}
|
||||
]
|
||||
res, has_reason, changed = self.mutations(m, d)
|
||||
problems += changed
|
||||
if not self.placed(m, d):
|
||||
problems.append(f"{self.at(m, d)} (a detail put where the scan cannot follow it)")
|
||||
if code.unread or code.none or len(code.values) != 1 or problems:
|
||||
out["unreadCodes"] += problems or [self.at(m, d)]
|
||||
return
|
||||
@@ -625,6 +663,7 @@ class Scan:
|
||||
next(iter(code.values)), {"reasons": set(), "unread": [], "sites": 0, "reasonless": []}
|
||||
)
|
||||
entry["sites"] += 1
|
||||
res, has_reason = Res(), False
|
||||
for k, v in items:
|
||||
if k == "reason":
|
||||
res, has_reason = res | self.resolve(m, v), True
|
||||
@@ -633,81 +672,17 @@ class Scan:
|
||||
entry["reasons"] |= res.values
|
||||
entry["unread"] += res.unread
|
||||
|
||||
def mutations(self, m: Module, d: Node) -> tuple[Res, bool, list[str]]:
|
||||
"""A detail put in a variable: what `|= {…}` and `["reason"] = …` add, and every other use that
|
||||
could change it or send it elsewhere."""
|
||||
holder, stmt = d, m.parents.get(d)
|
||||
while isinstance(stmt, ast.IfExp) and holder is not stmt.test:
|
||||
holder, stmt = stmt, m.parents.get(stmt)
|
||||
if not (isinstance(stmt, (ast.Assign, ast.AnnAssign)) and stmt.value is holder):
|
||||
return Res(), False, []
|
||||
targets = stmt.targets if isinstance(stmt, ast.Assign) else [stmt.target]
|
||||
fn = self.enclosing(m, d, FUNCS)
|
||||
if fn is None or len(targets) != 1 or not isinstance(targets[0], ast.Name):
|
||||
return (
|
||||
Res(),
|
||||
False,
|
||||
[f"{self.at(m, stmt)} (a detail put where the scan cannot follow it)"],
|
||||
)
|
||||
name = targets[0].id
|
||||
res, found, problems = Res(), False, []
|
||||
for n in _own_scope(fn):
|
||||
if n is stmt:
|
||||
continue
|
||||
if name in _binds(n):
|
||||
if not (isinstance(n, ast.AugAssign) and isinstance(n.op, ast.BitOr)):
|
||||
problems.append(f"{self.at(m, n)} (binds the detail's variable again)")
|
||||
elif isinstance(n.value, ast.Dict):
|
||||
for k, v in self.entries(n.value):
|
||||
if k == "reason":
|
||||
res, found = res | self.resolve(m, v), True
|
||||
elif k in ("code", "?") or k is None and not self.clean_spread(m, n, v):
|
||||
problems.append(f"{self.at(m, v)} (may change the code)")
|
||||
elif not self.clean_spread(m, n, n.value):
|
||||
problems.append(f"{self.at(m, n)} (may change the code)")
|
||||
continue
|
||||
if not (isinstance(n, ast.Name) and n.id == name and isinstance(n.ctx, ast.Load)):
|
||||
continue
|
||||
p = m.parents.get(n)
|
||||
if isinstance(p, ast.AugAssign) and p.target is n:
|
||||
continue
|
||||
if isinstance(p, ast.Subscript) and p.value is n:
|
||||
key = p.slice.value if isinstance(p.slice, ast.Constant) else None
|
||||
assign = m.parents.get(p)
|
||||
if isinstance(p.ctx, ast.Load):
|
||||
def other_details(self) -> list[tuple[str, int, str | None]]:
|
||||
"""Each slot that sends what the client cannot say — nothing, or anything but a code dict or a
|
||||
factory's call: (its file, its line, the function it is sent from)."""
|
||||
out = []
|
||||
for m in self.mods.values():
|
||||
for call, d in detail_slots(m.tree):
|
||||
if self.sayable(m, d):
|
||||
continue
|
||||
if (
|
||||
key == "reason"
|
||||
and isinstance(p.ctx, ast.Store)
|
||||
and isinstance(assign, ast.Assign)
|
||||
and assign.targets == [p]
|
||||
):
|
||||
res, found = res | self.resolve(m, assign.value), True
|
||||
elif not (
|
||||
isinstance(key, str)
|
||||
and key not in ("code", "reason")
|
||||
and isinstance(p.ctx, ast.Store)
|
||||
):
|
||||
problems.append(f"{self.at(m, p)} (may change the code or drop the reason)")
|
||||
continue
|
||||
if isinstance(p, ast.Attribute) and p.attr in READS:
|
||||
continue
|
||||
sender = (
|
||||
p
|
||||
if isinstance(p, ast.Call)
|
||||
else m.parents.get(p)
|
||||
if isinstance(p, ast.keyword)
|
||||
else None
|
||||
)
|
||||
func = getattr(sender, "func", None)
|
||||
if (
|
||||
isinstance(sender, ast.Call)
|
||||
and (getattr(func, "id", None) or getattr(func, "attr", None)) in SENDERS
|
||||
and (n in sender.args or isinstance(p, ast.keyword) and p.arg == "detail")
|
||||
):
|
||||
continue
|
||||
problems.append(f"{self.at(m, n)} (the detail goes where the scan cannot follow it)")
|
||||
return res, found, problems
|
||||
fn = self.enclosing(m, call, FUNCS)
|
||||
out.append((m.path, call.lineno, getattr(fn, "name", None)))
|
||||
return sorted(out)
|
||||
|
||||
def run(self) -> dict:
|
||||
out: dict = {
|
||||
@@ -727,15 +702,7 @@ class Scan:
|
||||
):
|
||||
out["aliases"].setdefault(m.path, {})[name] = sorted(v)
|
||||
for node in ast.walk(m.tree):
|
||||
is_dict = isinstance(node, ast.Dict) and any(
|
||||
isinstance(k, ast.Constant) and k.value == "code" for k in node.keys
|
||||
)
|
||||
is_call = (
|
||||
isinstance(node, ast.Call)
|
||||
and getattr(node.func, "id", None) == "dict"
|
||||
and any(k.arg == "code" for k in node.keywords)
|
||||
)
|
||||
if not (is_dict or is_call):
|
||||
if not _code_dict(node):
|
||||
continue
|
||||
if m.path in NOT_REFUSALS:
|
||||
out["excluded"][m.path] = out["excluded"].get(m.path, 0) + 1
|
||||
@@ -750,14 +717,16 @@ def scan(sources: dict[str, str]) -> dict:
|
||||
return Scan(sources).run()
|
||||
|
||||
|
||||
def app_sources(app: pathlib.Path = APP) -> dict[str, str]:
|
||||
return {
|
||||
p.relative_to(app).as_posix(): p.read_text(encoding="utf-8")
|
||||
for p in sorted(app.rglob("*.py"))
|
||||
if "__pycache__" not in p.parts
|
||||
}
|
||||
|
||||
|
||||
def scan_app(app: pathlib.Path = APP) -> dict:
|
||||
return scan(
|
||||
{
|
||||
p.relative_to(app).as_posix(): p.read_text(encoding="utf-8")
|
||||
for p in sorted(app.rglob("*.py"))
|
||||
if "__pycache__" not in p.parts
|
||||
}
|
||||
)
|
||||
return scan(app_sources(app))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in New Issue
Block a user