fix(activity): runs keep their true outcome and spend, the page reaches every kept row, a closing dialog leaves the server log alone
Backend - A run's outcome write sits outside the job's own try: a database blip after a run that succeeded no longer records it as failed (the row waits for the next start's sweep). - An interrupted run keeps what it spent (the sweep sums its quota rows, as finish does). - The Scheduler's ad-hoc row says running while any backfill runs, not only the newest. - recent_runs reads each job's last runs off its index instead of ranking the whole table. - An interval change is stored with its audit row in one transaction, then rescheduled. - One API instance per database is the stated contract (.env.example, jobruns). - Turning full history on for a channel already complete records no run. - Migration 0109 round-trips interrupted and ad-hoc runs; the run words' DB CHECKs are test-held to the enums; the account filter is one helper for the four activity reads. - An audit row says how many of its request's log lines are still kept. Frontend - An "All" period reaches every row still kept (runs and admin actions outlive 90 days). - Back/Forward is followed in place by the server log: a closing dialog no longer turns Live off; a pivot still opens a fresh log. - Runs moved over from the audit log read their one-liner in the reader's language. - Filter choices or accounts that failed to load say so beside the filters, with a retry. - An old server-log link keeps its account, text and range; a remembered retired page opens its tab. - A request whose lines are gone says so instead of linking to an empty list. - The log lines and the history rows share one row frame, and the server log one list frame with the other tabs; an empty list is announced; each day's quota is readable.
This commit is contained in:
+42
-15
@@ -5,13 +5,14 @@ OWN short session, committed at once, for the reason `quota.record_usage` gives:
|
||||
may roll back (its `except: db.rollback()`), and the history of what happened must not go with it —
|
||||
nor may this bookkeeping commit the job's half-done session.
|
||||
|
||||
Only the API process runs jobs (the worker never starts the scheduler), so at its startup nothing can
|
||||
still be running: `interrupt_leftovers` closes what a stopped process left open.
|
||||
Only the API process runs jobs (the worker never starts the scheduler), and one API process serves a
|
||||
database (`.env.example`, the Scheduler section: the single-run guard is in its memory too), so at its
|
||||
startup nothing can still be running: `interrupt_leftovers` closes what a stopped process left open.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import delete, func, select, update
|
||||
from sqlalchemy import delete, func, select, union_all, update
|
||||
from sqlalchemy.engine import CursorResult
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -70,30 +71,46 @@ def finish(
|
||||
|
||||
|
||||
def interrupt_leftovers() -> int:
|
||||
"""Every row a previous process left `running` → `interrupted`. Called once at startup, before
|
||||
the scheduler can open a new run. Returns how many."""
|
||||
"""Every row a previous process left `running` → `interrupted`, with what it spent until then
|
||||
(its quota rows outlive it, as `finish` sums them). Called once at startup, before the scheduler
|
||||
can open a new run. Returns how many."""
|
||||
spent = (
|
||||
select(func.coalesce(func.sum(QuotaEvent.units), 0))
|
||||
.where(QuotaEvent.job_run_id == JobRun.id)
|
||||
.scalar_subquery()
|
||||
)
|
||||
with SessionLocal() as s:
|
||||
res = s.execute(
|
||||
update(JobRun)
|
||||
.where(JobRun.status == RunStatus.RUNNING)
|
||||
.values(status=RunStatus.INTERRUPTED, finished_at=datetime.now(timezone.utc))
|
||||
.values(
|
||||
status=RunStatus.INTERRUPTED,
|
||||
finished_at=datetime.now(timezone.utc),
|
||||
quota_units=spent,
|
||||
)
|
||||
)
|
||||
s.commit()
|
||||
return res.rowcount if isinstance(res, CursorResult) else 0
|
||||
|
||||
|
||||
def recent_runs(db: Session, per_job: int) -> dict[str, list[JobRun]]:
|
||||
"""Each job's last `per_job` runs, newest first, by job id — one query (a window per job)."""
|
||||
ranked = select(
|
||||
JobRun.id,
|
||||
func.row_number()
|
||||
.over(partition_by=JobRun.job, order_by=(JobRun.started_at.desc(), JobRun.id.desc()))
|
||||
.label("rank"),
|
||||
def recent_runs(db: Session, jobs: list[str], per_job: int) -> dict[str, list[JobRun]]:
|
||||
"""Each of `jobs`' last `per_job` runs, newest first, by job id — one query, each job's part a
|
||||
short walk of `ix_job_runs_job_started`, so a poll costs the same at any history length (a window
|
||||
over the whole table grew with every run kept)."""
|
||||
if not jobs:
|
||||
return {}
|
||||
newest = union_all(
|
||||
*(
|
||||
select(JobRun.id)
|
||||
.where(JobRun.job == job)
|
||||
.order_by(JobRun.started_at.desc(), JobRun.id.desc())
|
||||
.limit(per_job)
|
||||
for job in jobs
|
||||
)
|
||||
).subquery()
|
||||
rows = db.execute(
|
||||
select(JobRun)
|
||||
.join(ranked, ranked.c.id == JobRun.id)
|
||||
.where(ranked.c.rank <= per_job)
|
||||
.where(JobRun.id.in_(select(newest.c.id)))
|
||||
.order_by(JobRun.job, JobRun.started_at.desc(), JobRun.id.desc())
|
||||
).scalars()
|
||||
out: dict[str, list[JobRun]] = {}
|
||||
@@ -102,6 +119,16 @@ def recent_runs(db: Session, per_job: int) -> dict[str, list[JobRun]]:
|
||||
return out
|
||||
|
||||
|
||||
def running_one(db: Session, job: str) -> JobRun | None:
|
||||
"""The newest run of `job` still going, however many runs ended after it started."""
|
||||
return db.scalars(
|
||||
select(JobRun)
|
||||
.where(JobRun.job == job, JobRun.status == RunStatus.RUNNING)
|
||||
.order_by(JobRun.started_at.desc(), JobRun.id.desc())
|
||||
.limit(1)
|
||||
).first()
|
||||
|
||||
|
||||
def gc(db: Session, retention_days: int) -> int:
|
||||
"""Delete finished runs older than `retention_days` (0 = keep forever). A run still going is
|
||||
never reaped. The caller commits."""
|
||||
|
||||
Reference in New Issue
Block a user