fix(downloads): stop a ReDoS title and a fatal thumbnail convert from wedging/killing downloads

Three prod-observed download bugs, root-caused from a frozen prod queue with the
worker pinned at 100% CPU:

- ReDoS in normalize_title: _TRAILING_HASHTAGS had two overlapping unbounded
  quantifiers around the required letter, so a Facebook title with a long hashtag
  cluster that is NOT at the end ("... #a #b #c | Page") backtracked catastrophically.
  It runs inside the yt-dlp progress hook holding the GIL, so one such download froze
  the whole worker (and could stall feed enrichment too - sync/videos.py calls it).
  Rewrite with a zero-width "has a letter" lookahead + a possessive tag body: one
  unambiguous split, nothing to backtrack. 40-tag stress case: ~0.1ms.

- Reddit download killed by a cosmetic thumbnail convert: yt-dlp's
  FFmpegThumbnailsConvertor runs "-bsf:v mjpeg2jpeg" on a JPEG-served-under-a-.png-name
  (Reddit) and fails; being a postprocessor, it failed the WHOLE download even though
  the media was fine. Drop it and produce the <base>.jpg poster/og:image ourselves via
  editmod.thumbnail_to_jpg (plain ffmpeg decode->mjpeg, best-effort -> ensure_poster
  cuts a frame on failure). The FB Messenger / MS Teams share preview keeps working
  (posters verified as real JPEGs).

- classify: map "authentication is required" (Reddit's auth wall) to login_required
  instead of unknown, so the UI explains it rather than offering a pointless retry.

Tests: ReDoS-timing + numeric-marker regression, the Reddit classify case, and
thumbnail_to_jpg (real JPEG out / None on bad input, never raises).
This commit is contained in:
2026-08-12 18:27:35 +02:00
parent 6883fedc1b
commit 37cca723fc
8 changed files with 117 additions and 5 deletions
+23
View File
@@ -24,6 +24,29 @@ def test_trailing_hashtags_are_stripped():
assert normalize_title("Great cooking video #food #yummy") == "Great cooking video"
def test_numeric_episode_marker_is_not_treated_as_a_hashtag():
# A bare "#3" is an episode marker, not an SEO hashtag (no letters) → it must survive.
assert normalize_title("My show #3") == "My show #3"
def test_midstring_hashtags_are_fast_and_kept_redos_regression():
# Facebook prepends an engagement prefix and appends a page name, leaving a long hashtag cluster
# in the MIDDLE. That is not a trailing SEO cluster, so the tags stay — and, critically, this
# exact shape used to trigger catastrophic backtracking (ReDoS) in `_TRAILING_HASHTAGS` that
# pinned a download-worker thread at 100% CPU forever inside the yt-dlp progress hook.
import time
raw = (
"398K views · 6.1K reactions | The money should follow 🙄 #reel #reels #meme #memes "
"#fun #funny #work #worklife #office #viral #trending #fyp #foryou #foryoupage | Memebonanza4U"
)
t = time.perf_counter()
out = normalize_title(raw) or ""
assert time.perf_counter() - t < 1.0 # linear now; the old regex ran for minutes / never returned
assert "#foryoupage" in out and "#reel" in out # mid-string hashtags preserved (not trailing)
assert "views" not in out # the engagement-count prefix was still stripped
def test_emoji_and_symbols_are_dropped():
assert "🔥" not in (normalize_title("Fire recipe 🔥🔥") or "")