peter
37cca723fc
Three prod-observed download bugs, root-caused from a frozen prod queue with the
worker pinned at 100% CPU:
- ReDoS in normalize_title: _TRAILING_HASHTAGS had two overlapping unbounded
quantifiers around the required letter, so a Facebook title with a long hashtag
cluster that is NOT at the end ("... #a #b #c | Page") backtracked catastrophically.
It runs inside the yt-dlp progress hook holding the GIL, so one such download froze
the whole worker (and could stall feed enrichment too - sync/videos.py calls it).
Rewrite with a zero-width "has a letter" lookahead + a possessive tag body: one
unambiguous split, nothing to backtrack. 40-tag stress case: ~0.1ms.
- Reddit download killed by a cosmetic thumbnail convert: yt-dlp's
FFmpegThumbnailsConvertor runs "-bsf:v mjpeg2jpeg" on a JPEG-served-under-a-.png-name
(Reddit) and fails; being a postprocessor, it failed the WHOLE download even though
the media was fine. Drop it and produce the <base>.jpg poster/og:image ourselves via
editmod.thumbnail_to_jpg (plain ffmpeg decode->mjpeg, best-effort -> ensure_poster
cuts a frame on failure). The FB Messenger / MS Teams share preview keeps working
(posters verified as real JPEGs).
- classify: map "authentication is required" (Reddit's auth wall) to login_required
instead of unknown, so the UI explains it rather than offering a pointless retry.
Tests: ReDoS-timing + numeric-marker regression, the Reddit classify case, and
thumbnail_to_jpg (real JPEG out / None on bad input, never raises).
47 lines
1.7 KiB
Python
47 lines
1.7 KiB
Python
"""`thumbnail_to_jpg`: robustly turn a downloaded thumbnail into a real JPEG poster/og:image.
|
|
|
|
This replaced yt-dlp's `FFmpegThumbnailsConvertor`, whose `-bsf:v mjpeg2jpeg` path crashed on some
|
|
already-JPEG thumbnails (Reddit serves a JPEG under a `.png` name) — and because a postprocessor
|
|
error is fatal, it used to fail the WHOLE download. The key regression property here is that a bad
|
|
thumbnail returns None (poster falls back to a cut frame) instead of raising.
|
|
"""
|
|
import shutil
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from app.downloads import edit
|
|
|
|
_ffmpeg_missing = shutil.which("ffmpeg") is None
|
|
needs_ffmpeg = pytest.mark.skipif(_ffmpeg_missing, reason="ffmpeg not available")
|
|
|
|
|
|
def _is_jpeg(p: Path) -> bool:
|
|
return p.exists() and p.read_bytes()[:3] == b"\xff\xd8\xff"
|
|
|
|
|
|
@needs_ffmpeg
|
|
def test_converts_a_png_to_a_real_jpeg(tmp_path):
|
|
src = tmp_path / "thumb.png"
|
|
subprocess.run(
|
|
["ffmpeg", "-y", "-loglevel", "error", "-f", "lavfi",
|
|
"-i", "color=c=red:s=64x64", "-frames:v", "1", str(src)],
|
|
check=True,
|
|
)
|
|
out = edit.thumbnail_to_jpg(src, tmp_path / "poster.jpg")
|
|
assert out is not None and _is_jpeg(out)
|
|
|
|
|
|
def test_returns_none_for_a_missing_source(tmp_path):
|
|
assert edit.thumbnail_to_jpg(None, tmp_path / "p.jpg") is None
|
|
assert edit.thumbnail_to_jpg(tmp_path / "nope.png", tmp_path / "p.jpg") is None
|
|
|
|
|
|
@needs_ffmpeg
|
|
def test_returns_none_on_garbage_input_instead_of_raising(tmp_path):
|
|
# The whole point: a thumbnail ffmpeg can't decode must NOT crash the download.
|
|
bad = tmp_path / "bad.png"
|
|
bad.write_bytes(b"this is definitely not an image")
|
|
assert edit.thumbnail_to_jpg(bad, tmp_path / "p.jpg") is None
|