fix(downloads): stop a ReDoS title and a fatal thumbnail convert from wedging/killing downloads
Three prod-observed download bugs, root-caused from a frozen prod queue with the
worker pinned at 100% CPU:
- ReDoS in normalize_title: _TRAILING_HASHTAGS had two overlapping unbounded
quantifiers around the required letter, so a Facebook title with a long hashtag
cluster that is NOT at the end ("... #a #b #c | Page") backtracked catastrophically.
It runs inside the yt-dlp progress hook holding the GIL, so one such download froze
the whole worker (and could stall feed enrichment too - sync/videos.py calls it).
Rewrite with a zero-width "has a letter" lookahead + a possessive tag body: one
unambiguous split, nothing to backtrack. 40-tag stress case: ~0.1ms.
- Reddit download killed by a cosmetic thumbnail convert: yt-dlp's
FFmpegThumbnailsConvertor runs "-bsf:v mjpeg2jpeg" on a JPEG-served-under-a-.png-name
(Reddit) and fails; being a postprocessor, it failed the WHOLE download even though
the media was fine. Drop it and produce the <base>.jpg poster/og:image ourselves via
editmod.thumbnail_to_jpg (plain ffmpeg decode->mjpeg, best-effort -> ensure_poster
cuts a frame on failure). The FB Messenger / MS Teams share preview keeps working
(posters verified as real JPEGs).
- classify: map "authentication is required" (Reddit's auth wall) to login_required
instead of unknown, so the UI explains it rather than offering a pointless retry.
Tests: ReDoS-timing + numeric-marker regression, the Reddit classify case, and
thumbnail_to_jpg (real JPEG out / None on bad input, never raises).
This commit is contained in:
@@ -34,7 +34,15 @@ _ENGAGEMENT_PREFIX = re.compile(
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LETTER_RUN = re.compile(r"[^\W\d_]+", re.UNICODE)
|
||||
_TRAILING_HASHTAGS = re.compile(r"(?:\s+#[^\s#]*[^\W\d\s_][^\s#]*)+\s*$", re.UNICODE)
|
||||
# Trailing SEO hashtag cluster: a run of whitespace-separated `#tags` at the very end, where each
|
||||
# tag contains at least one letter (a bare numeric "#3" episode marker is NOT a hashtag and
|
||||
# survives). The "has a letter" test is a zero-width lookahead and the tag body is matched
|
||||
# possessively (`*+`), so there is exactly ONE way to split each tag and nothing to backtrack over.
|
||||
# The previous form `[^\s#]*[letter][^\s#]*` had TWO overlapping unbounded quantifiers around the
|
||||
# required letter: on a long hashtag run that ends with trailing non-tag text (a Facebook
|
||||
# "… #a #b #c | Page" title), the `$` anchor fails and the engine backtracks catastrophically
|
||||
# (ReDoS) — it pinned a worker thread at 100% CPU forever inside the download progress hook.
|
||||
_TRAILING_HASHTAGS = re.compile(r"(?:\s+#(?=[^\s#]*[^\W\d\s_])[^\s#]*+)+\s*$", re.UNICODE)
|
||||
_REPEAT_PUNCT = re.compile(r"([!?.,\-])\1{1,}")
|
||||
|
||||
# Short function words → lowercased when the title is de-shouted (proper title-case style).
|
||||
|
||||
Reference in New Issue
Block a user