fix(downloads): stop a ReDoS title and a fatal thumbnail convert from wedging/killing downloads

Three prod-observed download bugs, root-caused from a frozen prod queue with the
worker pinned at 100% CPU:

- ReDoS in normalize_title: _TRAILING_HASHTAGS had two overlapping unbounded
  quantifiers around the required letter, so a Facebook title with a long hashtag
  cluster that is NOT at the end ("... #a #b #c | Page") backtracked catastrophically.
  It runs inside the yt-dlp progress hook holding the GIL, so one such download froze
  the whole worker (and could stall feed enrichment too - sync/videos.py calls it).
  Rewrite with a zero-width "has a letter" lookahead + a possessive tag body: one
  unambiguous split, nothing to backtrack. 40-tag stress case: ~0.1ms.

- Reddit download killed by a cosmetic thumbnail convert: yt-dlp's
  FFmpegThumbnailsConvertor runs "-bsf:v mjpeg2jpeg" on a JPEG-served-under-a-.png-name
  (Reddit) and fails; being a postprocessor, it failed the WHOLE download even though
  the media was fine. Drop it and produce the <base>.jpg poster/og:image ourselves via
  editmod.thumbnail_to_jpg (plain ffmpeg decode->mjpeg, best-effort -> ensure_poster
  cuts a frame on failure). The FB Messenger / MS Teams share preview keeps working
  (posters verified as real JPEGs).

- classify: map "authentication is required" (Reddit's auth wall) to login_required
  instead of unknown, so the UI explains it rather than offering a pointless retry.

Tests: ReDoS-timing + numeric-marker regression, the Reddit classify case, and
thumbnail_to_jpg (real JPEG out / None on bad input, never raises).
This commit is contained in:
2026-08-12 18:27:35 +02:00
parent 6883fedc1b
commit 37cca723fc
8 changed files with 117 additions and 5 deletions
+7 -3
View File
@@ -211,9 +211,13 @@ def build_ydl_opts(
opts["postprocessors"].append(
{"key": "FFmpegMetadata", "add_chapters": True, "add_metadata": True}
)
# Normalize the written thumbnail to jpg so we always have a `<name>.jpg` poster sidecar.
# `already_have_thumbnail` keeps that file on disk after embedding (default deletes it).
opts["postprocessors"].append({"key": "FFmpegThumbnailsConvertor", "format": "jpg"})
# NOTE: we deliberately do NOT add yt-dlp's `FFmpegThumbnailsConvertor` here. Its jpg conversion
# runs `ffmpeg -bsf:v mjpeg2jpeg` when the written thumbnail is already JPEG bytes (Reddit serves
# a JPEG under a `.png` name), and that bitstream filter *fails* on some such images — and since a
# postprocessor error is fatal, it used to kill the WHOLE download even though the media was fine.
# The `<name>.jpg` poster sidecar is instead produced robustly by our own pipeline (worker
# `_download` → `editmod.thumbnail_to_jpg`, with `ensure_poster` cutting a video frame as a
# fallback), so the og:image share preview (Facebook Messenger / MS Teams) keeps working.
if s["embed_thumbnail"]:
opts["postprocessors"].append(
{"key": "EmbedThumbnail", "already_have_thumbnail": True}