fix(files): review round 1, the server — the tray's window, the GC's locks, a file's origin, the upgrade path
- The queue keeps a done job by when it became done (ready_at), not by its last write: renaming, keeping or moving an old file, or the GC's expiry warning, no longer brings it back as just finished. - GC expiry and eviction re-read each file under the asset→job locks the user routes take, and commit one file at a time: a delete, a re-download or a keep made while the GC runs is honoured, never a second release that frees a file another user still holds. - A file records the origin it was placed under (download_jobs.source_key, migration 0105): an edit whose parent expired still goes back to its folder, not to Uploads. - A folder a file is being put in is held (FOR SHARE) until it lands; a delete of it waits and takes that file home too, instead of failing on the foreign key. - The hash route reads at most two files at once (429 past that) and holds no connection meanwhile. - 0104's backfill dates old files by their request date only: no notice prod wrote names its job. - The old DOWNLOAD_ROOT still finds the files; a files root that is not a mounted volume stops the worker and is logged by the API; the volume-copy recipe looks the project-prefixed names up. - Stale comments about the /downloads mount and the Downloads page.
This commit is contained in:
@@ -68,9 +68,11 @@ def site_key(url: str) -> str:
|
||||
def source_key_for(db: Session, job: DownloadJob) -> str:
|
||||
"""The origin a job's file came from — the source folder it belongs in.
|
||||
|
||||
youtube → YouTube; a clip → the clip's source (YouTube or Plex); an upload → Uploads; a URL →
|
||||
its site; an edit → wherever its parent's file came from (its parent job, or, once that is
|
||||
deleted, the parent's asset)."""
|
||||
The one recorded when it was placed, else: youtube → YouTube; a clip → the clip's source
|
||||
(YouTube or Plex); an upload → Uploads; a URL → its site; an edit → wherever its parent's file
|
||||
came from (its parent job, or, once that is deleted, the parent's asset)."""
|
||||
if job.source_key:
|
||||
return job.source_key
|
||||
kind = job.source_kind
|
||||
if kind == "youtube":
|
||||
return SourceKey.YOUTUBE
|
||||
@@ -122,9 +124,11 @@ def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
|
||||
|
||||
|
||||
def place(db: Session, job: DownloadJob) -> None:
|
||||
"""Put a new job's file in its source folder. Called wherever a job is created; the job must
|
||||
carry its `user_id`, `source_kind` and source refs already."""
|
||||
job.folder_id = source_folder(db, job.user_id, source_key_for(db, job)).id
|
||||
"""Put a new job's file in its source folder, and record that origin on it. Called wherever a
|
||||
job is created; the job must carry its `user_id`, `source_kind` and source refs already."""
|
||||
key = source_key_for(db, job)
|
||||
job.source_key = key
|
||||
job.folder_id = source_folder(db, job.user_id, key).id
|
||||
|
||||
|
||||
def done_expiry(db: Session, folder_id: int | None, kept: bool) -> datetime | None:
|
||||
@@ -151,12 +155,25 @@ class FolderError(Exception):
|
||||
self.reason = reason
|
||||
|
||||
|
||||
def upload_target(db: Session, user_id: int, folder_id: int) -> FileFolder:
|
||||
"""The folder an upload asked to land in, if it may: one of the user's own folders, or their
|
||||
Uploads folder. A file put in a site's folder would claim an origin it does not have."""
|
||||
folder = db.get(FileFolder, folder_id)
|
||||
def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
|
||||
"""The user's folder `folder_id` that a file is about to be put in — held (FOR SHARE) until the
|
||||
caller's transaction ends, so a delete of the folder waits for the file and then takes it home
|
||||
with the rest (`delete_own` locks it FOR UPDATE), never failing on its foreign key. `hold=False`
|
||||
only looks (a check ahead of a long wait, which must not hold a lock through it). Another user's
|
||||
folder answers like a missing one."""
|
||||
stmt = select(FileFolder).where(FileFolder.id == folder_id)
|
||||
if hold:
|
||||
stmt = stmt.with_for_update(read=True)
|
||||
folder = db.execute(stmt.execution_options(populate_existing=True)).scalar_one_or_none()
|
||||
if folder is None or folder.user_id != user_id:
|
||||
raise FolderError("file_folder_gone")
|
||||
return folder
|
||||
|
||||
|
||||
def upload_target(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
|
||||
"""The folder an upload asked to land in, if it may: one of the user's own folders, or their
|
||||
Uploads folder. A file put in a site's folder would claim an origin it does not have."""
|
||||
folder = own_folder(db, user_id, folder_id, hold=hold)
|
||||
if folder.kind == FolderKind.SOURCE and folder.source_key != SourceKey.UPLOAD:
|
||||
raise FolderError("file_folder_not_its_source")
|
||||
return folder
|
||||
@@ -219,6 +236,8 @@ def delete_own(db: Session, folder: FileFolder) -> int:
|
||||
folder it came from, its clock starting there now. Returns how many moved."""
|
||||
if folder.kind != FolderKind.OWN:
|
||||
raise FolderError("file_folder_not_own")
|
||||
# Wait out a file being put in it right now (`own_folder`), then take that one home too.
|
||||
db.execute(select(FileFolder.id).where(FileFolder.id == folder.id).with_for_update())
|
||||
jobs = db.execute(select(DownloadJob).where(DownloadJob.folder_id == folder.id)).scalars().all()
|
||||
for job in jobs:
|
||||
job.folder_id = source_folder(db, job.user_id, source_key_for(db, job)).id
|
||||
|
||||
Reference in New Issue
Block a user