peter
37cca723fc
Three prod-observed download bugs, root-caused from a frozen prod queue with the
worker pinned at 100% CPU:
- ReDoS in normalize_title: _TRAILING_HASHTAGS had two overlapping unbounded
quantifiers around the required letter, so a Facebook title with a long hashtag
cluster that is NOT at the end ("... #a #b #c | Page") backtracked catastrophically.
It runs inside the yt-dlp progress hook holding the GIL, so one such download froze
the whole worker (and could stall feed enrichment too - sync/videos.py calls it).
Rewrite with a zero-width "has a letter" lookahead + a possessive tag body: one
unambiguous split, nothing to backtrack. 40-tag stress case: ~0.1ms.
- Reddit download killed by a cosmetic thumbnail convert: yt-dlp's
FFmpegThumbnailsConvertor runs "-bsf:v mjpeg2jpeg" on a JPEG-served-under-a-.png-name
(Reddit) and fails; being a postprocessor, it failed the WHOLE download even though
the media was fine. Drop it and produce the <base>.jpg poster/og:image ourselves via
editmod.thumbnail_to_jpg (plain ffmpeg decode->mjpeg, best-effort -> ensure_poster
cuts a frame on failure). The FB Messenger / MS Teams share preview keeps working
(posters verified as real JPEGs).
- classify: map "authentication is required" (Reddit's auth wall) to login_required
instead of unknown, so the UI explains it rather than offering a pointless retry.
Tests: ReDoS-timing + numeric-marker regression, the Reddit classify case, and
thumbnail_to_jpg (real JPEG out / None on bad input, never raises).
49 lines
2.6 KiB
Python
49 lines
2.6 KiB
Python
"""Category classification of raw yt-dlp / ffmpeg download-failure strings (S2)."""
|
|
from app.downloads.errors import classify
|
|
|
|
|
|
def test_real_ytdlp_messages_map_to_categories():
|
|
# Verbatim-shaped yt-dlp / ffmpeg output → the category the UI explains.
|
|
cases = {
|
|
"ERROR: [youtube] xxxxxxxxxxx: Video unavailable": "unavailable",
|
|
"ERROR: [youtube] abc: Private video. Sign in if you've been granted access": "unavailable",
|
|
"ERROR: [youtube] abc: This video is no longer available because the uploader has closed their "
|
|
"account": "unavailable",
|
|
"ERROR: [youtube] abc: Sign in to confirm your age. This video may be inappropriate": "login_required",
|
|
"ERROR: [youtube] abc: Sign in to confirm you're not a bot": "login_required",
|
|
"ERROR: [youtube] abc: Join this channel to get access to members-only content": "login_required",
|
|
"ERROR: [Reddit] 1vi1x56: Account authentication is required.": "login_required",
|
|
"ERROR: [youtube] abc: The uploader has not made this video available in your country": "geo_blocked",
|
|
"ERROR: [youtube] abc: Requested format is not available": "format_unavailable",
|
|
"ERROR: [youtube] abc: Only images are available for download": "format_unavailable",
|
|
"ERROR: [generic] This video is DRM protected": "drm",
|
|
"ffmpeg failed (1): Conversion failed!": "postprocess",
|
|
"The source download is no longer available.": "source_missing",
|
|
"The source file is missing.": "source_missing",
|
|
"ERROR: unable to download video data: HTTP Error 403: Forbidden": "network",
|
|
"ERROR: Unable to download webpage: <urlopen error timed out>": "network",
|
|
}
|
|
for msg, expected in cases.items():
|
|
assert classify(msg) == expected, f"{msg!r} → {classify(msg)!r}, expected {expected!r}"
|
|
|
|
|
|
def test_unmatched_and_empty_are_unknown():
|
|
assert classify("some totally novel failure text") == "unknown"
|
|
assert classify("") == "unknown"
|
|
assert classify(None) == "unknown"
|
|
|
|
|
|
def test_matching_is_case_insensitive():
|
|
assert classify("VIDEO UNAVAILABLE") == "unavailable"
|
|
assert classify("drm protected") == "drm"
|
|
|
|
|
|
def test_login_beats_generic_unavailable_when_both_present():
|
|
# A members-only video also reads as "unavailable" to a signed-out client; the sign-in cause is
|
|
# the more useful, more specific one, so it must win (rule order guards this).
|
|
msg = (
|
|
"ERROR: [youtube] abc: Join this channel to get access to members-only content. "
|
|
"This video is unavailable"
|
|
)
|
|
assert classify(msg) == "login_required"
|