docs(sync): scope normalize_description to channels + lock line-ending behavior
Review follow-up. The docstring said "channel/video description" but the helper is wired only to the channel store sites; clarify it is deliberately channel-only (video descriptions are user-facing prose where paragraph breaks carry meaning, unlike a channel's metadata block) so it doesn't read as an asymmetric fix. Also document that str.splitlines() folds CR/CRLF and Unicode line separators to "\n", and lock it with a test.
This commit is contained in:
@@ -39,11 +39,14 @@ def to_tsquery_str(q: str) -> str | None:
|
||||
|
||||
|
||||
def normalize_description(text: str | None) -> str | None:
|
||||
"""Strip whitespace-only lines from a channel/video description before storing. YouTube
|
||||
descriptions pad contact/link blocks with runs of blank lines; keeping them wastes storage,
|
||||
"""Strip whitespace-only lines from a CHANNEL description before storing. YouTube channel
|
||||
descriptions pad their contact/link blocks with runs of blank lines; keeping them wastes storage,
|
||||
bloats the About popover, and adds nothing to the search document. Drops purely-empty lines and
|
||||
right-strips each kept line. None stays None; all-blank collapses to "". Applied at the
|
||||
sync/store site so the DB (and the derived search_vector) holds the clean text."""
|
||||
right-strips each kept line; `str.splitlines()` also folds CR / CRLF and Unicode line separators
|
||||
to "\\n" (a benign line-ending normalization). None stays None; all-blank collapses to "".
|
||||
|
||||
Deliberately CHANNEL-only — video descriptions are left with their original formatting: they are
|
||||
user-facing prose where paragraph breaks carry meaning, unlike a channel's metadata block."""
|
||||
if text is None:
|
||||
return None
|
||||
kept = [line.rstrip() for line in text.splitlines() if line.strip()]
|
||||
|
||||
@@ -50,3 +50,8 @@ def test_normalize_description_all_blank_collapses_to_empty():
|
||||
|
||||
def test_normalize_description_single_line_unchanged():
|
||||
assert normalize_description("just one line") == "just one line"
|
||||
|
||||
|
||||
def test_normalize_description_folds_crlf_line_endings():
|
||||
# splitlines() folds CR/CRLF to \n — a documented, benign line-ending normalization.
|
||||
assert normalize_description("a\r\nb\r\n\r\nc") == "a\nb\nc"
|
||||
|
||||
Reference in New Issue
Block a user