feat(downloads): ASCII-slug names, universal browser playability, force re-download
Mini-epic "Downloads correctness & UX" (S1–S3), on top of the iOS-AV1 + 720p fixes.
S1 — naming & titles:
- ASCII-fold on-disk dir/file names AND the served (Content-Disposition) filename: no
spaces (underscore-joined), no accents (á→a, ő→o, ß→ss), pure ASCII — portable and free
of the `?` mojibake accented names show in non-UTF-8 tools. (Forward-only; existing files
clean up via re-download.)
- Strip a leading social engagement prefix ("1.6M views · 66K reactions | …") from titles.
S2 — universal browser playability (generalizes the iOS re-encode):
- ensure_browser_playable guarantees the stored file is H.264 + AAC/MP3 in mp4 — the only
combo every browser decodes. Re-encode just the offending stream(s) (video for AV1/VP9/HEVC,
audio for Opus/etc), or remux when only the container is wrong. Gated on the mp4 compat
profile (an explicit vcodec / non-mp4 custom profile opts out).
S3 — force re-download (preserves share links):
- POST /downloads/{job_id}/redownload keeps the job row so its public DownloadLink survives
and resolves to the freshly-downloaded file; deletes the old file and re-fetches under the
current rules. A failed re-download leaves the job in error (the intended broken-link case).
- Library "Re-download" action (icon + confirm), api method, hu/en strings.
Tests: naming/title-strip (test_naming), browser_transcode_flags matrix; existing
download_filename expectation updated for the underscore policy.
This commit is contained in:
@@ -20,13 +20,35 @@ from xml.sax.saxutils import escape
|
||||
|
||||
# Reserve room for the " [id].ext" suffix + parent dirs within the 255-byte component limit.
|
||||
_MAX_TITLE = 120
|
||||
# User-facing (Content-Disposition) names can be longer than an on-disk component.
|
||||
_MAX_DISPLAY = 150
|
||||
# Filesystem-illegal on Windows/Plex + control chars.
|
||||
_ILLEGAL = re.compile(r'[<>:"/\\|?*\x00-\x1f]')
|
||||
# Unicode categories we drop entirely: emoji/pictographs (So), modifier symbols/skin-tones
|
||||
# (Sk), and every control/format/private/surrogate/unassigned char (C*). Letters (incl.
|
||||
# accented á/ö/ü/ß), digits, and ordinary punctuation are KEPT — Siftlode is trilingual, so
|
||||
# ASCII-folding would mangle Hungarian/German titles.
|
||||
# (Sk), and every control/format/private/surrogate/unassigned char (C*).
|
||||
_DROP_CATEGORIES = {"So", "Sk", "Cc", "Cf", "Cs", "Co", "Cn"}
|
||||
# Latin letters NFKD can't decompose to an ASCII base — fold them explicitly (trilingual HU/DE
|
||||
# plus common European). Everything else with a diacritic (á/é/ö/ő/ü/ű, ç, ñ…) NFKD-decomposes
|
||||
# into base + a combining mark we then drop, and any remaining non-ASCII is stripped, so every
|
||||
# generated name is pure ASCII: no spaces, no accents — portable and free of the `?` mojibake
|
||||
# that accented names show in non-UTF-8 tools.
|
||||
_FOLD_MAP = str.maketrans(
|
||||
{
|
||||
"ß": "ss", "ẞ": "SS", "ø": "o", "Ø": "O", "ł": "l", "Ł": "L", "đ": "d", "Đ": "D",
|
||||
"ð": "d", "Ð": "D", "þ": "th", "Þ": "Th", "æ": "ae", "Æ": "AE", "œ": "oe", "Œ": "OE",
|
||||
"ı": "i", "İ": "I", "·": " ", "–": "-", "—": "-",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _ascii_fold(text: str) -> str:
|
||||
"""Transliterate to pure ASCII: apply the explicit map, NFKD-split accented letters and drop
|
||||
the combining marks (á→a, ő→o), then strip any leftover non-ASCII (non-Latin scripts that have
|
||||
no ASCII base)."""
|
||||
text = text.translate(_FOLD_MAP)
|
||||
text = unicodedata.normalize("NFKD", text)
|
||||
text = "".join(ch for ch in text if unicodedata.category(ch) != "Mn")
|
||||
return text.encode("ascii", "ignore").decode("ascii")
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -41,16 +63,17 @@ class MediaMeta:
|
||||
|
||||
|
||||
def sanitize(name: str, limit: int = _MAX_TITLE) -> str:
|
||||
"""Turn an arbitrary (emoji-laden, clickbait) title into a clean, space-free path token.
|
||||
"""Turn an arbitrary (emoji-laden, clickbait) title into a clean, space-free ASCII path token.
|
||||
|
||||
Steps: NFKC-normalize (fold fancy width/compat forms) → drop emoji/symbols/control chars →
|
||||
strip filesystem-illegal chars → collapse whitespace and runs of the same punctuation →
|
||||
join words with underscores → trim junk punctuation from the ends → length-cap. Accented
|
||||
letters and digits survive; the result never contains spaces or path separators."""
|
||||
ASCII-fold accents (á→a, ő→o, ß→ss) → strip filesystem-illegal chars → collapse whitespace and
|
||||
runs of the same punctuation → join words with underscores → trim junk punctuation from the ends
|
||||
→ length-cap. The result is pure ASCII with no spaces or path separators (digits survive)."""
|
||||
text = unicodedata.normalize("NFKC", name or "")
|
||||
text = "".join(
|
||||
ch for ch in text if unicodedata.category(ch) not in _DROP_CATEGORIES
|
||||
)
|
||||
text = _ascii_fold(text)
|
||||
text = _ILLEGAL.sub("", text)
|
||||
# Collapse repeated punctuation ("!!!", "???", "-----") to a single mark.
|
||||
text = re.sub(r"([^\w\s])\1{1,}", r"\1", text)
|
||||
@@ -68,13 +91,19 @@ def sanitize(name: str, limit: int = _MAX_TITLE) -> str:
|
||||
|
||||
|
||||
def display_filename(name: str) -> str:
|
||||
"""User-facing download filename (Content-Disposition): strip emoji/symbols/control +
|
||||
filesystem-illegal chars, but KEEP spaces, accents, and ordinary punctuation — unlike the
|
||||
underscore-joined on-disk name, this is the readable name the user saves to their device."""
|
||||
"""User-facing download filename (Content-Disposition) base: the same ASCII-slug policy as the
|
||||
on-disk name — emoji/symbols/control + illegal chars dropped, accents folded, whitespace joined
|
||||
with underscores — so a saved file is portable (no spaces, no accents). Only the length cap
|
||||
differs (a served name may be longer than an on-disk path component)."""
|
||||
text = unicodedata.normalize("NFKC", name or "")
|
||||
text = "".join(ch for ch in text if unicodedata.category(ch) not in _DROP_CATEGORIES)
|
||||
text = _ascii_fold(text)
|
||||
text = _ILLEGAL.sub("", text)
|
||||
text = re.sub(r"\s+", " ", text).strip(" .")
|
||||
text = re.sub(r"([^\w\s])\1{1,}", r"\1", text)
|
||||
text = re.sub(r"\s+", "_", text.strip())
|
||||
text = re.sub(r"_{2,}", "_", text).strip("_.-·—–|,;:!?ّ ")
|
||||
if len(text) > _MAX_DISPLAY:
|
||||
text = text[:_MAX_DISPLAY].rstrip("_.-")
|
||||
return text or "download"
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user