docs(sync): scope normalize_description to channels + lock line-ending behavior
Review follow-up. The docstring said "channel/video description" but the helper is wired only to the channel store sites; clarify it is deliberately channel-only (video descriptions are user-facing prose where paragraph breaks carry meaning, unlike a channel's metadata block) so it doesn't read as an asymmetric fix. Also document that str.splitlines() folds CR/CRLF and Unicode line separators to "\n", and lock it with a test.
This commit is contained in:
@@ -39,11 +39,14 @@ def to_tsquery_str(q: str) -> str | None:
|
|||||||
|
|
||||||
|
|
||||||
def normalize_description(text: str | None) -> str | None:
|
def normalize_description(text: str | None) -> str | None:
|
||||||
"""Strip whitespace-only lines from a channel/video description before storing. YouTube
|
"""Strip whitespace-only lines from a CHANNEL description before storing. YouTube channel
|
||||||
descriptions pad contact/link blocks with runs of blank lines; keeping them wastes storage,
|
descriptions pad their contact/link blocks with runs of blank lines; keeping them wastes storage,
|
||||||
bloats the About popover, and adds nothing to the search document. Drops purely-empty lines and
|
bloats the About popover, and adds nothing to the search document. Drops purely-empty lines and
|
||||||
right-strips each kept line. None stays None; all-blank collapses to "". Applied at the
|
right-strips each kept line; `str.splitlines()` also folds CR / CRLF and Unicode line separators
|
||||||
sync/store site so the DB (and the derived search_vector) holds the clean text."""
|
to "\\n" (a benign line-ending normalization). None stays None; all-blank collapses to "".
|
||||||
|
|
||||||
|
Deliberately CHANNEL-only — video descriptions are left with their original formatting: they are
|
||||||
|
user-facing prose where paragraph breaks carry meaning, unlike a channel's metadata block."""
|
||||||
if text is None:
|
if text is None:
|
||||||
return None
|
return None
|
||||||
kept = [line.rstrip() for line in text.splitlines() if line.strip()]
|
kept = [line.rstrip() for line in text.splitlines() if line.strip()]
|
||||||
|
|||||||
@@ -50,3 +50,8 @@ def test_normalize_description_all_blank_collapses_to_empty():
|
|||||||
|
|
||||||
def test_normalize_description_single_line_unchanged():
|
def test_normalize_description_single_line_unchanged():
|
||||||
assert normalize_description("just one line") == "just one line"
|
assert normalize_description("just one line") == "just one line"
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_description_folds_crlf_line_endings():
|
||||||
|
# splitlines() folds CR/CRLF to \n — a documented, benign line-ending normalization.
|
||||||
|
assert normalize_description("a\r\nb\r\n\r\nc") == "a\nb\nc"
|
||||||
|
|||||||
Reference in New Issue
Block a user