fix(downloads): stop a ReDoS title and a fatal thumbnail convert from wedging/killing downloads
Three prod-observed download bugs, root-caused from a frozen prod queue with the
worker pinned at 100% CPU:
- ReDoS in normalize_title: _TRAILING_HASHTAGS had two overlapping unbounded
quantifiers around the required letter, so a Facebook title with a long hashtag
cluster that is NOT at the end ("... #a #b #c | Page") backtracked catastrophically.
It runs inside the yt-dlp progress hook holding the GIL, so one such download froze
the whole worker (and could stall feed enrichment too - sync/videos.py calls it).
Rewrite with a zero-width "has a letter" lookahead + a possessive tag body: one
unambiguous split, nothing to backtrack. 40-tag stress case: ~0.1ms.
- Reddit download killed by a cosmetic thumbnail convert: yt-dlp's
FFmpegThumbnailsConvertor runs "-bsf:v mjpeg2jpeg" on a JPEG-served-under-a-.png-name
(Reddit) and fails; being a postprocessor, it failed the WHOLE download even though
the media was fine. Drop it and produce the <base>.jpg poster/og:image ourselves via
editmod.thumbnail_to_jpg (plain ffmpeg decode->mjpeg, best-effort -> ensure_poster
cuts a frame on failure). The FB Messenger / MS Teams share preview keeps working
(posters verified as real JPEGs).
- classify: map "authentication is required" (Reddit's auth wall) to login_required
instead of unknown, so the UI explains it rather than offering a pointless retry.
Tests: ReDoS-timing + numeric-marker regression, the Reddit classify case, and
thumbnail_to_jpg (real JPEG out / None on bad input, never raises).
This commit is contained in:
@@ -489,6 +489,28 @@ def build_poster_cmd(src: Path, dest: Path, at_s: float) -> list[str]:
|
||||
]
|
||||
|
||||
|
||||
def thumbnail_to_jpg(src: Path | None, dest: Path, timeout_s: int = 30) -> Path | None:
|
||||
"""Convert a downloaded thumbnail (webp / png / jpeg — whatever the source shipped) into a real
|
||||
baseline JPEG for the `<name>.jpg` poster + og:image share sidecar.
|
||||
|
||||
Uses a plain ffmpeg decode→mjpeg, deliberately NOT yt-dlp's `FFmpegThumbnailsConvertor` (whose
|
||||
`-bsf:v mjpeg2jpeg` path crashes on some already-JPEG thumbnails, e.g. Reddit's, and used to
|
||||
fail the entire download). Best-effort: returns None on any failure, so the caller falls back to
|
||||
a cut video frame via `ensure_poster` and the download still succeeds."""
|
||||
if src is None or not src.exists():
|
||||
return None
|
||||
try:
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-nostdin",
|
||||
"-i", str(src), "-frames:v", "1", "-update", "1", str(dest)],
|
||||
capture_output=True, timeout=timeout_s, check=True,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError) as exc:
|
||||
log.info("thumbnail->jpg conversion failed (%s); poster falls back to a cut frame", str(exc)[:120])
|
||||
return None
|
||||
return dest if dest.exists() else None
|
||||
|
||||
|
||||
def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
|
||||
"""Generate a poster frame for a ready asset that has no thumbnail (e.g. a direct media URL
|
||||
like a reddit HLS manifest, which yt-dlp can't get a thumbnail for). Written as the media's
|
||||
|
||||
Reference in New Issue
Block a user