fix(downloads,sync): bound what one user can make the server fetch, download and spend
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches it from the WORKER, so any approved user could aim the server at the compose network, the LAN or a cloud metadata endpoint — and read the result off job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a port scanner. downloads/urlguard.py refuses a URL whose host resolves to a loopback/private/link-local/reserved address, at enqueue AND again in the worker (DNS can move in between, and the worker is where the socket opens). A network-class failure on a `url` job now reports one fixed sentence instead of the raw transport text; the error CODE is still classified from the original, so the localized explanation is unchanged. DNS rebinding is explicitly NOT closed — that needs the connection gated inside yt-dlp — and the module says so rather than implying more than it does. SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video, so a URL that is purely a playlist or channel still expanded into every entry — one enqueue, unbounded downloads. `playlist_items` pins it to one. Size was unbounded too: `check_enqueue` compares the CURRENT footprint against the cap, and a download's size is unknown until it lands, so one job could overshoot by any amount; the owner's remaining budget is now handed to yt-dlp as `max_filesize`, applied AFTER the adapter merge so no source adapter can raise a limit the quota layer set. A join's cut-list is capped at 100: an accurate join emits two filters per segment into one ffmpeg filter_complex. SEC-14: the YouTube budget is instance-wide, so one user holding down a manual sync spent the day for everyone. Per-user (not per-IP — the cost belongs to the account) 3-per-10-minutes limiters on the two bulk pulls, subscriptions and playlists, with separate budgets so one does not close the other. The review also listed /rss, /backfill and /enrich; E3 deleted them, and search stays unlimited on purpose — it is interactive, not a bulk pull. Gate: pytest 896 -> 924 (none skipped), vitest 671.
This commit is contained in:
@@ -7,12 +7,20 @@ from app.audit import AuditAction
|
||||
from app.auth import admin_user, current_user, has_read_scope, require_human
|
||||
from app.db import get_db
|
||||
from app.models import Channel, Subscription, User, Video
|
||||
from app.ratelimit import RateLimiter
|
||||
from app.sync.runner import estimate_deep_backfill
|
||||
from app.sync.subscriptions import import_subscriptions
|
||||
from app.scheduler import running_job_ids
|
||||
|
||||
router = APIRouter(prefix="/api/sync", tags=["sync"])
|
||||
|
||||
# SEC-14: the YouTube API budget is SHARED across the whole instance, and a subscription import
|
||||
# walks every page of the caller's subscriptions — so one user holding down a manual sync button can
|
||||
# spend the day's units for everyone. Keyed by user id, not client IP: the cost belongs to the
|
||||
# account, and the same person on a second device is the same drain. 3 per 10 minutes leaves manual
|
||||
# re-syncing entirely usable (the scheduler does the routine work anyway) while removing the loop.
|
||||
_sync_limiter = RateLimiter(max_events=3, window_seconds=600)
|
||||
|
||||
|
||||
@router.post("/subscriptions")
|
||||
def sync_subscriptions(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
|
||||
@@ -22,6 +30,8 @@ def sync_subscriptions(user: User = Depends(require_human), db: Session = Depend
|
||||
status_code=403,
|
||||
detail="Connect your YouTube account (read access) to import subscriptions.",
|
||||
)
|
||||
if not _sync_limiter.allow(str(user.id)):
|
||||
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
|
||||
with quota.measured(db, user.id, quota.QuotaAction.SUBSCRIPTIONS_PULL) as m:
|
||||
result = import_subscriptions(db, user)
|
||||
result["quota_used_estimate"] = m.used
|
||||
|
||||
Reference in New Issue
Block a user