peter/siftlode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(plex): a segment name ffmpeg never spells is refused; teardown no longer holds the registry lock

Browse Source 
The artifact guard parsed the sequence with int(), so a zero-padded or non-ASCII-digit name read
as a segment the session had reached and parked the sync route for the full 20 s wait. It now
also requires the canonical spelling segment_name() produces.

Stopping, replacing, culling or reaping a session waited up to 3 s for ffmpeg and deleted a whole
segmented film under _lock, which every segment request and checkpoint takes. Teardown now signals
the process and moves its directory aside under the lock, and waits/deletes after it. A new session
on the same path still waits for the dying ffmpeg first (it writes by path), and the orphan sweep
recognises a moved-aside directory left by a crash.
This commit is contained in:
peter
2026-10-28 18:48:21 +01:00
parent 4084dd2f92
commit e96a562f14
7 changed files with 289 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/plex/stream.py
+92 -19
View File
@@ -66,7 +66,9 @@ _SESSION_IDLE_S = 600 # reap a session with no access for this long
# restart leaves them behind like any other. `sweep_orphans` deletes ONLY names
# matching one of those two shapes: `_HLS_ROOT` is admin-configurable (PLEX_HLS_DIR), so it may
# legitimately point at a shared scratch path we do not own, and a blanket "delete every directory
# under the root" would take the neighbours with it.
# under the root" would take the neighbours with it. A torn-down session's directory is renamed to
# its own name + `_TOMB` + 8 hex digits before it is deleted (see `_detach`), so a crash between the
# two leaves a name this sweep still recognises.
#
# The KEY is anchored to digits (a Plex rating_key is a number) — that is what keeps foreign names
# out. A `.+` there let `snapshot_2024_backup_+1.00` through, and pairing `.+` with an optional tail
@@ -74,8 +76,10 @@ _SESSION_IDLE_S = 600 # reap a session with no access for this long
# non-numeric rating_key would merely leave its directory unswept, which is the safe direction to
# fail. Covered by test_plex_stream.py — this regex is the only thing standing between the sweep
# and someone else's data.
_TOMB = ".dead-"
_SESSION_DIR_RE = re.compile(
r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{2}$"
r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$"
rf"|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{{2}}({re.escape(_TOMB)}[0-9a-f]{{8}})?$"
)
# When this process started. `sweep_orphans` only deletes directories older than this: everything
# newer belongs to a session THIS run created, and since `start_session` reuses a directory path
@@ -93,7 +97,7 @@ _MTIME_GRACE_S = 2.0
_lock = threading.Lock()
# Keyed by VIEWER and item, (user_id, rating_key). Keyed by the item alone, a second account opening
# the same film `_kill`ed the first one's session: the displaced viewer got a 404, reclaimed at its
# the same film tore down the first one's session: the displaced viewer got a 404, reclaimed at its
# playhead, and so evicted the other in turn — two people could not watch one film at once. (One
# account on two devices still shares a session; that case ends in the player's bounded reclaim.)
_sessions: dict[tuple[int, str], "HlsSession"] = {}
@@ -641,29 +645,82 @@ def _probe_first_pts(seg: Path) -> float | None:
return None
def _kill(s: HlsSession) -> None:
# Teardown is split in two, because every segment request and checkpoint takes `_lock`: waiting up to
# 3 s for an ffmpeg to exit and deleting a whole segmented film under it stalled every other
# viewer's playback for one viewer's seek or one reap pass. `_detach` (under the lock) signals the
# process and renames its directory aside — so the deletion can never hit a NEW session that reuses
# the path — and `_finish` (after the lock is released) waits and deletes.
#
# What the rename cannot cover is the dying ffmpeg itself: it writes by path, so until it has exited
# it can still put a segment, or its final ENDLIST playlist, into a new session's directory on the
# same path. `_dying` remembers it by that path, and `_spawn` waits it out — under the lock, on
# purpose — before it creates the directory again.
_dying: dict[Path, subprocess.Popen] = {}
def _await_exit(key: str, proc: subprocess.Popen) -> None:
try:
s.proc.terminate()
try:
s.proc.wait(timeout=3)
proc.wait(timeout=3)
# silent-ok: the escalation IS the handling. A transcode that ignores TERM for three
# seconds gets KILL, which is exactly what this branch does.
except subprocess.TimeoutExpired:
s.proc.kill()
proc.kill()
except Exception as exc: # noqa: BLE001 — teardown finishes whatever the process does
# Not nothing: what survives this is an ffmpeg nobody owns any more, holding a VAAPI
# handle and a core, and the transcode cap that is supposed to bound them counts rows in
# `_sessions` rather than processes. One line is the only way that ever surfaces.
log.info(
"could not stop the transcode process",
extra=logs.event("plex.stream.kill_failed", key=key, reason=type(exc).__name__),
)
def _detach(s: HlsSession) -> tuple[HlsSession, Path | None]:
"""Caller holds _lock and has taken `s` out of `_sessions`. Hand the result to `_finish` once the
lock is released."""
try:
s.proc.terminate()
except Exception as exc: # noqa: BLE001 — teardown finishes whatever the process does
log.info(
"could not stop the transcode process",
extra=logs.event("plex.stream.kill_failed", key=s.key, reason=type(exc).__name__),
)
shutil.rmtree(s.dir, ignore_errors=True)
_dying[s.dir] = s.proc
tomb = s.dir.with_name(f"{s.dir.name}{_TOMB}{secrets.token_hex(4)}")
try:
s.dir.rename(tomb)
# silent-ok: nothing on disk (ffmpeg died before writing, or a sweep took it) — nothing to delete.
except FileNotFoundError:
return s, None
except OSError as exc:
# Cannot be moved aside, so it cannot be deleted outside the lock either: the path may be a
# new session's a moment from now. Deleted here instead, the old way.
log.info(
"a session directory could not be moved aside; deleted under the lock",
extra=logs.event("plex.stream.tomb_failed", key=s.key, reason=type(exc).__name__),
)
shutil.rmtree(s.dir, ignore_errors=True)
return s, None
return s, tomb
def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
"""Caller holds _lock. Three bounds, applied after a session is registered, each dropping the
least recently accessed sessions over it:
def _finish(doomed: "list[tuple[HlsSession, Path | None]]") -> None:
"""The slow half of a teardown, called WITHOUT `_lock`."""
for s, tomb in doomed:
_await_exit(s.key, s.proc)
with _lock:
if _dying.get(s.dir) is s.proc:
del _dying[s.dir]
if tomb is not None:
shutil.rmtree(tomb, ignore_errors=True)
def _enforce_caps(
user_id: int, max_transcodes: int, max_sessions: int
) -> "list[tuple[HlsSession, Path | None]]":
"""Caller holds _lock, and `_finish`es what this returns once it has released it. Three bounds,
applied after a session is registered, each dropping the least recently accessed sessions over it:
* this account's own sessions (`_MAX_SESSIONS_PER_USER`) — first, so a single account filling the
box evicts itself before anyone else;
@@ -676,6 +733,8 @@ def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
and the player restarts at the playhead, so from the server nothing failed — and "my film keeps
rebuffering" has no other explanation anywhere, hence the line."""
doomed: list[tuple[HlsSession, Path | None]] = []
def cull(pool: list[tuple[tuple[int, str], HlsSession]], cap: int, bound: str) -> None:
if len(pool) <= cap:
return
@@ -690,12 +749,13 @@ def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
sessions=len(pool),
),
)
_kill(sess)
_sessions.pop(reg, None)
doomed.append(_detach(sess))
cull([kv for kv in _sessions.items() if kv[0][0] == user_id], _MAX_SESSIONS_PER_USER, "user")
cull([kv for kv in _sessions.items() if kv[1].transcode], max_transcodes, "transcode")
cull(list(_sessions.items()), max_sessions, "total")
return doomed
def start_session(
@@ -759,12 +819,18 @@ def start_session(
# can't starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a
# platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
doomed: list[tuple[HlsSession, Path | None]] = []
with _lock:
if replace_only is not None and _sessions.get(reg) is not replace_only:
return None # superseded/culled while we waited out of lock — don't clobber the newer one
old = _sessions.pop(reg, None)
if old is not None:
_kill(old)
doomed.append(_detach(old))
# An ffmpeg still exiting from THIS path (the old session, or one stopped a moment ago)
# would write into the directory about to be made; see `_dying`.
prev = _dying.pop(directory, None)
if prev is not None:
_await_exit(key, prev)
shutil.rmtree(directory, ignore_errors=True)
directory.mkdir(parents=True, exist_ok=True)
proc = subprocess.Popen(
@@ -816,11 +882,12 @@ def start_session(
)
# Admin-configurable caps (Configuration → Plex); at least 1 each so playback can't be
# capped to zero.
_enforce_caps(
doomed += _enforce_caps(
user_id,
max(1, sysconfig.get_int(db, "plex_max_transcodes")),
max(1, sysconfig.get_int(db, "plex_max_sessions")),
)
_finish(doomed)
return s
s = _spawn(*_select_encode(db, item, transcode, src))
@@ -974,6 +1041,11 @@ def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
# session, absent for a single-audio one.
if variant not in (s.variants or (None,)):
return False
# And the name must be the one ffmpeg SPELLS for that sequence: `int()` also reads `seg_007.ts`
# and a non-ASCII digit run as a sequence the session may have reached, and neither file is
# ever written — a yes there parked the request for the whole wait.
if filename != segment_name(seq, variant):
return False
if seq == 0:
return True
try:
@@ -1108,16 +1180,16 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
`reap_abandoned` bought the same safety by keeping every finished session for ten minutes — each
one a whole segmented copy of the film on a volume shared with the download center."""
now = time.time()
dropped = 0
doomed: list[tuple[HlsSession, Path | None]] = []
with _lock:
for reg, s in list(_sessions.items()):
done = s.proc.poll() is not None
idle = now - s.last_access
if idle > _SESSION_IDLE_S or (done and idle > finished_idle_s):
_kill(s)
_sessions.pop(reg, None)
dropped += 1
return dropped
doomed.append(_detach(s))
_finish(doomed)
return len(doomed)
def stop_session(user_id: int, key: str, session_id: str) -> bool:
@@ -1132,8 +1204,9 @@ def stop_session(user_id: int, key: str, session_id: str) -> bool:
s = _sessions.get((user_id, key))
if s is None or s.id != session_id:
return False
_kill(s)
_sessions.pop((user_id, key), None)
doomed = [_detach(s)]
_finish(doomed)
return True
backend/app/routes/plex/playback.py
+4 -3
View File
@@ -632,9 +632,10 @@ def stream_hls_file(
try:
content = path.read_bytes()
except OSError:
# The session was torn down between the wait and the read — `_kill` rmtrees the whole
# directory, and nothing holds `_lock` across those two steps. It has to present as the same
# 404 a gone session does: the player's recovery is keyed on the STATUS (`code === 404`), so
# The session was torn down between the wait and the read — teardown moves the whole
# directory aside and deletes it, and nothing holds `_lock` across those two steps. It has
# to present as the same 404 a gone session does: the player's recovery is keyed on the
# STATUS (`code === 404`), so
# a 500 here skipped the reclaim and ended playback with "couldn't play" instead of
# restarting at the playhead. Ordinary trigger: `loadSession` awaits the new session — which
# kills and rmtrees the old one — before destroying the old hls instance, so the outgoing