peter/siftlode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

release: 0.81.0 (improvement/e12-security-ops)

Browse Source
This commit is contained in:
peter
2026-09-05 04:54:26 +02:00
parent e23128b4a1 6a0efba477
commit 1d750977b3
63 changed files with 5069 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+1789
View File
File diff suppressed because it is too large Load Diff
LICENSE
+661
View File
@@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.
README.md
+18 -3
View File
@@ -94,9 +94,18 @@ docker compose -f docker-compose.selfhost.yml pull # or: docker compose pull
docker compose -f docker-compose.selfhost.yml up -d
```
Database migrations run automatically on startup. Your data (accounts, subscriptions, playlists, the
video catalog) lives in a Postgres volume — back it up with `scripts/backup.sh` (or `backup.ps1` on
Windows) and restore with `scripts/restore.sh`.
That follows `latest`, which moves with every release. Set `IMAGE_TAG=0.80.0` in `.env` to pin a
version and upgrade on your own schedule; released versions are listed in [`CHANGELOG.md`](CHANGELOG.md)
and in the app's Release Notes. The registry keeps only the newest few of those, so a pin decides
*when* you take the next hop rather than letting you stay put forever — see
[docs/self-hosting.md](docs/self-hosting.md#upgrading).
**Back up before upgrading.** Migrations run automatically on startup and are not undone by pulling
the old image back. A backup is the database dump **plus your `.env`** — `TOKEN_ENCRYPTION_KEY` is
what decrypts the Google, SMTP and Plex secrets stored in that database, so a dump restored without it
is only half an instance. Use `scripts/backup.sh` (or `backup.ps1` on Windows) and `scripts/restore.sh`;
the equivalent one-liners, the pinning recipe and the rollback steps are in
[docs/self-hosting.md](docs/self-hosting.md#backups).
## How it works
@@ -111,6 +120,12 @@ Windows) and restore with `scripts/restore.sh`.
FastAPI + PostgreSQL (SQLAlchemy, Alembic) backend; React + Vite + Tailwind + TanStack Query
frontend; packaged as a single Docker image with Docker Compose.
## License
[GNU AGPL-3.0](LICENSE). You can run, study, modify and share Siftlode; if you modify it and let other
people use it over a network, they are entitled to your modified source. Copyright © 2026 Siftlode
contributors.
## Note
This project is developed with AI assistance.
VERSION
+1 -1
View File
@@ -1 +1 @@
0.80.0
0.81.0
backend/app/audit.py
+1
View File
@@ -31,6 +31,7 @@ class AuditAction:
# recorded because both are the admin acting ON someone's access, not merely reading a list.
USER_ACTIVATE = "user_activate"
USER_VERIFY_RESEND = "user_verify_resend"
USER_SETPW_LINK = "user_setpw_link"
INVITE_APPROVE = "invite_approve"
INVITE_DENY = "invite_deny"
INVITE_ADD = "invite_add"
backend/app/downloads/edit.py
+10
View File
@@ -54,6 +54,10 @@ class EditAborted(Exception):
# --- edit spec ------------------------------------------------------------------------------
# SEC-13: ceiling on a join's cut-list. An accurate join emits two filters per segment into a single
# ffmpeg `filter_complex`, so "however many the caller sent" is an unbounded command line.
_MAX_SEGMENTS = 100
def normalize_edit_spec(spec: dict | None) -> dict:
"""Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
@@ -88,6 +92,12 @@ def normalize_edit_spec(spec: dict | None) -> dict:
if en > st:
norm.append({"start_s": round(st, 3), "end_s": round(en, 3)})
norm.sort(key=lambda x: x["start_s"])
# SEC-13: an accurate join builds one ffmpeg `filter_complex` with two filters per segment,
# so an unbounded list becomes an unbounded command line and an ffmpeg that never finishes.
# The cap is far above any real cut-list (the UI's own splitter produces a handful) and the
# excess is DROPPED rather than raising: the surplus segments are the attack, the first
# _MAX_SEGMENTS are a legitimate edit.
del norm[_MAX_SEGMENTS:]
if norm:
out["segments"] = norm
# A join always chooses a codec path: accurate=filter-concat re-encode,
backend/app/downloads/errors.py
+36
View File
@@ -8,9 +8,28 @@ explanation (and decides whether a retry makes sense). The raw text stays availa
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
"""
# Two worker-authored strings that this module owns because both are coupled to `_RULES`.
#
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
# rule drifting apart is exactly how a category silently becomes unreachable.
#
# NETWORK_REDACTED_MESSAGE is the opposite direction: it REPLACES a message after classification, so
# it deliberately has no rule. Nothing re-classifies a stored `job.error` today; if anything ever
# does, it must carry the original code rather than re-deriving one from this text.
#
# Other worker-raised texts are still inline literals at their raise sites. The ffmpeg ones and
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
# classifies as `unknown` — it is a refusal, not a diagnosis, and no category would say more.
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
# case-insensitive on a substring, against the raw yt-dlp/ffmpeg message.
_RULES: tuple[tuple[str, str], ...] = (
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
# first — it is our wording, so it can never be a false match on an upstream message.
("download storage is full", "quota_exceeded"),
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
("sign in to confirm your age", "login_required"),
("confirm your age", "login_required"),
@@ -91,3 +110,20 @@ def classify(msg: str | None) -> str:
if marker in low:
return category
return "unknown"
def user_error_text(source_kind: str, msg: str, code: str) -> str:
"""The failure text to STORE on the job, given the raw message and its category.
SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection
refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner with
extra steps, since the raw text is shown to the user behind the details disclosure. `urlguard`
stops the fetch aiming at our own network; this stops the remaining readout.
Only the network class, and only for ``url`` jobs: a YouTube job has no user-chosen host, and
every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis. A separate
function rather than an inline branch in the worker so it can be pinned by a test — the raw
text leaking back is a security regression that would otherwise pass a green suite."""
if source_kind == "url" and code == "network":
return NETWORK_REDACTED_MESSAGE
return msg
backend/app/downloads/formats.py
+20 -1
View File
@@ -116,6 +116,7 @@ def build_ydl_opts(
progress_hook,
extra: dict | None = None,
cookiefile: str | None = None,
max_bytes: int | None = None,
) -> dict:
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
@@ -125,12 +126,20 @@ def build_ydl_opts(
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
merge so a source adapter can never clobber it."""
merge so a source adapter can never clobber it.
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp refuses a format larger
than it rather than filling the disk and only then being caught by the quota accounting."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
"outtmpl": outtmpl,
"noplaylist": True,
# SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to the single video. A
# URL that is PURELY a playlist or channel has no such video to fall back to, so it still
# expands into every entry — one enqueue, an unbounded number of downloads. One job means
# one item; a user who wants the whole list enqueues it item by item, under their quota.
"playlist_items": "1",
"quiet": True,
"no_warnings": True,
"noprogress": True,
@@ -154,6 +163,16 @@ def build_ydl_opts(
if extra:
opts.update(extra)
# SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
# `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set.
#
# A non-positive budget is left unset because 0 is not an expressible yt-dlp filter, NOT because
# nothing needs to happen: the worker refuses that job outright before extraction (`_OwnerRefused`
# → `quota_exceeded`). Enqueue cannot make that call — it ran while the user was still under the
# cap. Delete the worker's refusal and this branch silently uncaps the one user already over it.
if max_bytes and max_bytes > 0:
opts["max_filesize"] = max_bytes
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
# be able to override the authenticated session the worker resolved for this specific user.
if cookiefile:
backend/app/downloads/pipeline.py
+68 -4
View File
@@ -15,7 +15,8 @@ from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import errors as dlerrors
from app.downloads import adapters, credentials, formats, mediakind, probe, service, storage
from app.downloads import adapters, credentials, formats, mediakind, probe, quota, service, storage
from app.downloads import urlguard
from app.downloads.jobdb import (
_edit_staging_dir,
_job_status,
@@ -36,6 +37,18 @@ class _Aborted(Exception):
"""Raised inside the progress hook when the job was paused/canceled mid-download."""
class _OwnerRefused(Exception):
"""A refusal that belongs to THIS job's owner, not to the source.
MediaAssets are shared: two users asking for the same video and format get one row (see
`service.get_or_create_asset`), and the generic failure handler writes the message onto it —
correctly, because an upstream failure really is everyone's. A per-user refusal is not. Written
to the shared row it would fail a second user's queued job for the same video with the first
user's private reason (`worker._process_job` reads `asset.error` verbatim), telling someone with
an empty library to free up space. So this is caught separately: the job fails, the asset goes
back to `pending` for whoever else is holding it."""
# Both progress sources call back FAR faster than the database should be touched: yt-dlp fires its
# hook per chunk, and ffmpeg's `-progress pipe:1` emits a block of ~12 lines every ~0.5s while
# run_ffmpeg calls its callbacks once per LINE. Every callback here is a round-trip (`_job_status`
@@ -305,13 +318,17 @@ def _media_info(info: dict) -> dict:
return {**info, **chosen}
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter, cookiefile=None):
def _extract_with_fallback(
job_id, asset_id, url, spec, staging, adapter, cookiefile=None, max_bytes=None
):
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
error); other sources make a single attempt. Returns the info dict of the successful attempt.
`cookiefile` (S6) is the acting user's per-host cookies.txt, already gated by
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated."""
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated.
`max_bytes` (SEC-13) is the owner's remaining storage budget, applied to every attempt."""
client_sets = adapter.client_sets()
outtmpl = str(staging / "%(id)s.%(ext)s")
last_exc: Exception | None = None
@@ -322,6 +339,7 @@ def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter, cookie
_make_progress_hook(job_id, asset_id),
extra=adapter.ydl_opts_extra(clients),
cookiefile=cookiefile,
max_bytes=max_bytes,
)
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
_touch_job(job_id) # stall-watchdog heartbeat: extraction writes no progress, so a slow /
@@ -415,8 +433,17 @@ def _download(
_set_job(job_id, phase="downloading", progress=0)
adapter = adapters.resolve_adapter(source_kind, source_ref)
url = service.source_url(source_kind, source_ref)
# SEC-12, second half: the enqueue guard ran when the job was created, which may have been
# hours ago and against a different DNS answer. This is where the socket actually gets
# opened, so the check is repeated here and the job fails rather than fetching.
if source_kind == "url":
try:
urlguard.check_url(url)
except urlguard.BlockedUrl as e:
raise RuntimeError(f"Refusing to download from this address ({e.reason}).") from e
# S6: attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
# this asset is that user's private, salted identity (the double gate lives in credentials).
remaining_bytes: int | None = None
with SessionLocal() as cdb:
asset_sig = cdb.execute(
select(MediaAsset.format_sig).where(MediaAsset.id == asset_id)
@@ -425,6 +452,19 @@ def _download(
cookiefile = credentials.resolve_cookiefile(
cdb, user_id, url, asset_sig, spec, staging
)
# SEC-13: what this user has left. `check_enqueue` only compared the footprint as it
# stood when the job was QUEUED — a download's size is unknown until it finishes, so a
# single job could previously blow through the cap by any amount. Handing the remainder
# to yt-dlp as `max_filesize` makes the format selector refuse an oversized file up
# front instead.
remaining_bytes = quota.remaining_bytes(cdb, user_id)
# …and zero is a REFUSAL, not "no limit". Earlier jobs of this user may have filled the cap
# while this one sat in the queue, and 0 is not an expressible yt-dlp `max_filesize`, so
# passing it down would leave the ceiling unset — the one user already at their cap would be
# the only one downloading unbounded. Fail the job here instead. Not retryable until they
# free space, which is what the `quota_exceeded` category tells them.
if remaining_bytes == 0:
raise _OwnerRefused(dlerrors.QUOTA_FULL_MESSAGE)
info = _media_info(
_extract_with_fallback(
job_id,
@@ -434,6 +474,7 @@ def _download(
staging,
adapter,
cookiefile=str(cookiefile) if cookiefile else None,
max_bytes=remaining_bytes,
)
)
# Re-resolve now that yt-dlp's info reveals the true source (a bare/redirected url only
@@ -525,8 +566,31 @@ def _download(
# job's status (the route set it to paused/canceled). ref_count is adjusted by the route.
_reset_asset_pending(asset_id)
log.info("job %s aborted (pause/cancel)", job_id)
except _OwnerRefused as exc:
# Fail this job only. The asset is never marked `error`; it goes back to `pending` (below),
# so another user's job for the same video is claimed and downloaded normally instead of
# inheriting a reason that is about someone else's account.
msg = str(exc)[:500]
with SessionLocal() as db:
service.finalize_if_running(
db,
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
phase=None,
)
db.commit()
# Released AFTER the job is terminal: the moment the asset is `pending` another job can claim
# it, and this one must already be out of the way. (The staging tree is still torn down in
# `finally` below — the same narrow window the pause/cancel path above has always had.)
_reset_asset_pending(asset_id)
log.info("job %s refused for its owner: %s", job_id, msg)
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
msg = str(exc)[:500]
code = dlerrors.classify(msg)
# SEC-12: redact a transport-level readout for user-supplied URLs (see user_error_text).
msg = dlerrors.user_error_text(source_kind, msg, code)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
@@ -539,7 +603,7 @@ def _download(
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
error_code=code,
phase=None,
)
db.commit()
backend/app/downloads/quota.py
+22
View File
@@ -7,6 +7,13 @@ Limits come from a per-user DownloadQuota row if the admin set one, else the sys
Enforcement split: max_jobs + max_bytes are checked at enqueue (they bound holdings); per-user
max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads).
DOWNLOAD jobs additionally consult `remaining_bytes` in the worker, because enqueue can only compare
the footprint as it stood when the job was created and earlier jobs may have filled the cap since.
That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user each read the same
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
is unknown until it finishes. Edit jobs do not consult it at all (tracked as SEC-19).
"""
from dataclasses import dataclass
@@ -122,6 +129,21 @@ def check_enqueue(db: Session, user_id: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, used)
def remaining_bytes(db: Session, user_id: int) -> int | None:
"""Storage this user may still spend on ONE download: ``None`` for an unlimited user, else the
cap minus the current footprint, floored at 0.
Zero is a real answer, not "no limit" — it means the cap is already met and the caller must
REFUSE rather than proceed unbounded. `check_enqueue` cannot make that refusal on the caller's
behalf: it ran when the job was queued, while the user was still under the cap, and the jobs
that filled it have landed since. Keeping the arithmetic here (rather than inline in the worker)
is what keeps every max_bytes decision in this module."""
lim = resolve(db, user_id)
if lim.unlimited:
return None
return max(0, lim.max_bytes - footprint(db, user_id))
def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
"""Like check_enqueue, but for an upload whose size is KNOWN at ingest: also reject when adding
this file would push the footprint over max_bytes. A download can't pre-check its size (unknown
backend/app/downloads/urlguard.py
+108
View File
@@ -0,0 +1,108 @@
"""Refuse a user-supplied download URL that points back into our own network (SEC-12).
The Download Center deliberately accepts an arbitrary ``http(s)`` URL — that is the feature — and
hands it to yt-dlp running on the WORKER. Without a guard, any approved user can make the server
fetch anything the server can reach: other containers on the compose network, the LAN, a cloud
metadata endpoint. The failure text then comes back to them in ``job.error``, so "connection
refused" vs "HTTP Error 401" vs "timed out" turns the download box into an internal port scanner.
What this closes: a URL whose host resolves to a loopback / private / link-local / reserved address
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
differently between the two moments, and the worker is where the socket is actually opened.
What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
with an internal address.
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
kept). Tracked as SEC-17.
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
hostname into the box), which is the reachable one.
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
user-controlled destination to guard.
"""
import ipaddress
import socket
from urllib.parse import urlparse
# Reason codes travel to the client inside the structured 422 detail, so they are part of the API
# vocabulary (frontend `localizeDetail`) — keep them stable.
REASON_SCHEME = "scheme" # not http(s)
REASON_NO_HOST = "no_host" # URL carries no hostname
REASON_MALFORMED = "malformed" # unparseable authority or port; urlparse raises late, so guard it
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
class BlockedUrl(Exception):
"""A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
REASON_* codes above."""
def __init__(self, reason: str):
super().__init__(reason)
self.reason = reason
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
"""Every address family we must not open a connection to. Checked explicitly rather than
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
mapped = getattr(ip, "ipv4_mapped", None)
if mapped is not None:
ip = mapped
return bool(
ip.is_private
or ip.is_loopback
or ip.is_link_local
or ip.is_multicast
or ip.is_reserved
or ip.is_unspecified
)
def _resolve(host: str, port: int | None) -> list[str]:
"""Every address ``host`` resolves to. Raises BlockedUrl(REASON_UNRESOLVABLE) when DNS has no
answer — reported as its own outcome, because "we could not look that up" and "we refuse to
fetch that" are different things to tell a user."""
try:
infos = socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
except socket.gaierror as exc:
raise BlockedUrl(REASON_UNRESOLVABLE) from exc
return [info[4][0] for info in infos]
def check_url(raw: str) -> None:
"""Raise BlockedUrl unless ``raw`` is an http(s) URL whose host resolves only to addresses the
server is allowed to fetch from. A host with even ONE forbidden address is refused: a
round-robin answer must not become a coin flip."""
# `urlparse` itself raises ValueError on a malformed authority ("http://[::1/x" — an unclosed
# bracket), and `.port` raises on ":99999" / ":abc" when READ rather than at parse time. Both
# escape the caller's `except BlockedUrl` as a 500 unless caught here, which is the opposite of
# what this module exists to do: turn a bad URL into a structured, localizable refusal.
try:
u = urlparse((raw or "").strip())
port = u.port
except ValueError as exc:
raise BlockedUrl(REASON_MALFORMED) from exc
if u.scheme not in ("http", "https"):
raise BlockedUrl(REASON_SCHEME)
host = u.hostname
if not host:
raise BlockedUrl(REASON_NO_HOST)
for addr in _resolve(host, port):
try:
ip = ipaddress.ip_address(addr)
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
raise BlockedUrl(REASON_PRIVATE) from None
if _is_forbidden(ip):
raise BlockedUrl(REASON_PRIVATE)
backend/app/routes/admin.py
+73 -7
View File
@@ -32,7 +32,12 @@ from app.models import (
User,
)
from app.routes._common import get_or_404
from app.routes.auth import APPROVAL_VERIFY_TTL, issue_verification_link, send_verification_link
from app.routes.auth import (
APPROVAL_VERIFY_TTL,
issue_set_password_link,
issue_verification_link,
send_verification_link,
)
from app.utils import canonical_email, iso, valid_email
router = APIRouter(prefix="/api/admin", tags=["admin"])
@@ -168,9 +173,7 @@ def _activate_user_by_email(db: Session, email: str) -> None:
def _decide(db: Session, invite_id: int, admin: User, approved: bool) -> Invite:
inv = db.get(Invite, invite_id)
if inv is None:
raise HTTPException(status_code=404, detail="No such invite")
inv = get_or_404(db, Invite, invite_id, detail="No such invite")
# Idempotency guard: only a still-pending invite can be decided. Without this a repeat/stale
# approve or deny re-runs its side effects — re-activating a user, or (now that deny emails)
# sending a duplicate decision mail, even flipping an already-approved invite to "denied" and
@@ -361,9 +364,7 @@ def delete_invite(
falls back to the env ALLOWED_EMAILS / ADMIN_EMAILS bootstrap lists once no invite exists, so for
an address on those the deletion leaves access intact (and an ADMIN_EMAILS address is still
promoted to admin on its next Google sign-in). The admin has to hear that, not guess it."""
inv = db.get(Invite, invite_id)
if inv is None:
raise HTTPException(status_code=404, detail="No such invite")
inv = get_or_404(db, Invite, invite_id, detail="No such invite")
if _accounts_for(db, [inv.email]):
raise HTTPException(
status_code=409,
@@ -397,7 +398,15 @@ def _serialize_user(u: User) -> dict:
"email_verified": u.email_verified,
"is_demo": u.is_demo,
"has_password": u.password_hash is not None,
# Distinguishes "this account never had a password" (Google-only) from "SEC-10 cleared it" —
# the same distinction _send_reset_if_eligible needs, and the one an admin has to see before
# deciding what kind of help someone needs.
"password_revoked_at": iso(u.password_revoked_at),
"has_google": u.google_sub is not None,
# Only when it DIFFERS from the account address (E9.5 S3): "connected" is not the same
# statement as "connected as someone else", and the second one is the one that explains a
# confusing sign-in.
"google_email": u.google_email if u.google_email != u.email else None,
"created_at": iso(u.created_at),
}
@@ -612,6 +621,63 @@ def list_pending_access(_: User = Depends(admin_user), db: Session = Depends(get
return out
@router.get("/users/{user_id}/access")
def user_access(
user_id: int,
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Everything the account panel shows about why this person can or cannot get in (E12 S5).
Its own endpoint rather than extra columns on /users: `is_allowed` and `can_activate` each run a
query, and the panel opens for ONE user at a time — folding them into the list would pay that cost
per row on every load to answer a question nobody asked yet."""
target = get_or_404(db, User, user_id, detail="No such user")
return _serialize_user(target) | {
"is_allowed": is_allowed(db, target.email),
"can_activate": can_activate(db, target),
"email_configured": email_mod.email_enabled(db),
}
@router.post("/users/{user_id}/set-password-link")
def create_set_password_link(
user_id: int,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Mint a one-time link that lets this account set a password, with no email involved (E12 S5).
The answer to SEC-15, SEC-16 and "no SMTP and someone forgot their password" at once. The admin
hands the link over by whatever channel they trust; opening it sets a password and signs out
whoever held the old one.
Deliberately NOT gated on the account's state. It is useful for a pre-empted address, for a
revoked password, for a Google-only account that wants one, and for a plain forgotten password on
an instance with no mail — narrowing it to one of those would only make the admin's real case the
one it refuses. The demo account is the single exception: it is shared, so handing anyone a way to
change its password is handing them everyone else's session."""
target = get_or_404(db, User, user_id, detail="No such user")
if target.is_demo:
raise HTTPException(
status_code=400, detail={"code": "setpw", "reason": "demo", "email": target.email}
)
url, expires_at = issue_set_password_link(db, target)
# The URL itself is NOT recorded: the audit log is read by humans and kept long, and a live
# credential-setting link is the last thing that belongs in it. That the link was issued, by whom,
# for whom, is the auditable fact.
audit.record(
db,
admin.id,
AuditAction.USER_SETPW_LINK,
target_type="user",
target_id=target.id,
summary=target.email,
)
db.commit()
return {"ok": True, "url": url, "expires_at": iso(expires_at), "email": target.email}
@router.post("/users/{user_id}/activate")
def activate_user(
user_id: int,
backend/app/routes/auth.py
+50 -4
View File
@@ -52,6 +52,16 @@ VERIFY_TTL = timedelta(hours=24)
# in and sets no password.
APPROVAL_VERIFY_TTL = timedelta(days=7)
RESET_TTL = timedelta(hours=1)
# An admin-issued set-password link (E12 S5) travels out of band — the admin has to actually reach the
# person and hand it over — so the one-hour reset window is far too short. A day matches VERIFY_TTL and
# the affordance the admin was shown.
ADMIN_SETPW_TTL = timedelta(hours=24)
# Both kinds set a password from a link, and the SIGNED-OUT card consumes them through the same
# `#reset=` entry, so the user-facing flow is one flow. They stay separate kinds for one reason: a
# `reset` token proves control of the MAILBOX (it was mailed there) and therefore confirms the address;
# an admin-issued one proves nothing about the mailbox — the admin could hand it to anyone — so it must
# not silently satisfy the email gate. See password_reset_confirm.
_PASSWORD_TOKEN_KINDS = ("reset", "setpw")
_DECOY_PASSWORD_HASH = hash_password(secrets.token_urlsafe(16))
@@ -773,6 +783,35 @@ def _consume_token(db: Session, raw: str | None, kind: str) -> User | None:
return db.get(User, row.user_id)
def _consume_password_token(db: Session, raw: str | None) -> tuple[User, str] | None:
"""Spend a password-setting token of either kind; returns (user, kind). The kind is returned, not
swallowed, because the caller has to treat them differently — only a mailed reset attests the
address."""
for kind in _PASSWORD_TOKEN_KINDS:
user = _consume_token(db, raw, kind)
if user is not None:
return user, kind
return None
def issue_set_password_link(db: Session, user: User) -> tuple[str, datetime]:
"""An admin-issued, single-use link that lets `user` set a password without any email (E12 S5).
One mechanism answers three things: SEC-15 (on a no-SMTP instance every registration is
auto-verified, so a pre-empted address cannot be recovered by mail), SEC-16 (a pre-empted address
stays takeable until its owner resets, and password-reset needs working mail), and the everyday
"no SMTP configured and someone forgot their password".
Issuing does NOT revoke the current password: an admin misclick must not lock anyone out, and the
account may be perfectly healthy. USING the link replaces the hash and bumps the session epoch,
which is what evicts a squatter — see password_reset_confirm."""
expires_at = datetime.now(timezone.utc) + ADMIN_SETPW_TTL
raw = _issue_token(db, user, "setpw", ADMIN_SETPW_TTL)
# The same `#reset=` fragment the mailed link uses: the signed-out card already knows how to
# consume it, and a second entry shape would be a second thing to keep correct.
return f"{_app_base()}/#reset={raw}", expires_at
@router.post("/register")
def register(
body: EmailPasswordIn,
@@ -1105,7 +1144,7 @@ def password_reset_check(body: TokenIn, request: Request, db: Session = Depends(
"""
if not _reset_check_limiter.allow(_client_ip(request)):
return {"ok": True}
return {"ok": _peek_token(db, body.token, "reset") is not None}
return {"ok": any(_peek_token(db, body.token, k) is not None for k in _PASSWORD_TOKEN_KINDS)}
@router.post("/password-reset/confirm")
@@ -1118,15 +1157,22 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
status_code=400,
detail={"code": "reset", "reason": "weak_password", "min": PASSWORD_MIN_LEN},
)
user = _consume_token(db, token, "reset")
if user is None:
spent = _consume_password_token(db, token)
if spent is None:
raise HTTPException(status_code=400, detail={"code": "reset", "reason": "invalid"})
user, kind = spent
user.password_hash = hash_password(new_password)
user.password_revoked_at = None # the account has a password again — SEC-10's trace is spent
# A valid reset link proves control of the mailbox, so completing a reset also verifies the
# email — this rescues an account that registered but never clicked the verification link (it
# can then sign in). The separate admin-approval gate (is_active) is deliberately NOT touched
# here: a reset proves email ownership, not that an admin approved the account.
#
# An admin-issued `setpw` link proves NOTHING about the mailbox — the admin hands it over by
# whatever means they choose — so it must not satisfy the email gate. Claiming otherwise would let
# an admin confirm an address nobody ever proved, which is the same false attestation SEC-15 is
# about. The person still confirms their address the normal way.
if kind == "reset":
user.email_verified = True
bump_session_epoch(
user
@@ -1135,7 +1181,7 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
for row in db.execute(
select(AuthToken).where(
AuthToken.user_id == user.id,
AuthToken.kind == "reset",
AuthToken.kind.in_(_PASSWORD_TOKEN_KINDS),
AuthToken.used_at.is_(None),
)
).scalars():
backend/app/routes/downloads.py
+28
View File
@@ -28,6 +28,7 @@ from app.downloads import edit as editmod
from app.downloads import links as linksmod
from app.downloads import quota, service, storage
from app.downloads import upload as uploadmod
from app.downloads import urlguard
from app.models import (
DownloadJob,
DownloadLink,
@@ -65,7 +66,18 @@ def resolve_source(raw: str) -> tuple[str, str]:
if _VIDEO_ID.match(raw):
return "youtube", raw
if raw.startswith(("http://", "https://")):
# `urlparse` raises ValueError on a malformed authority ("http://[::1/x"), and this runs
# BEFORE the urlguard check that would otherwise turn it into a structured refusal — so
# unguarded, that typo escapes as a 500. Answered in urlguard's own shape, because it is the
# same typo class: ":99999" is caught there and localized, and one of the two arriving as a
# bare English string would be exactly the split the canonical-patterns table forbids.
try:
u = urlparse(raw)
except ValueError as exc:
raise HTTPException(
status_code=422,
detail={"code": "url_blocked", "reason": urlguard.REASON_MALFORMED},
) from exc
host = u.netloc.lower()
if "youtube.com" in host:
if u.path.startswith(("/shorts/", "/embed/", "/live/")):
@@ -352,6 +364,17 @@ def enqueue_download(
body: EnqueueDownloadIn, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
source_kind, source_ref = resolve_source(body.source)
# SEC-12: a non-YouTube source is a raw URL the worker will fetch server-side, so refuse the
# ones pointing back into our own network BEFORE a job exists. The worker re-checks (DNS can
# move between here and the download); see downloads/urlguard.py for what that does and does
# not cover.
if source_kind == "url":
try:
urlguard.check_url(source_ref)
except urlguard.BlockedUrl as e:
raise HTTPException(
status_code=422, detail={"code": "url_blocked", "reason": e.reason}
) from e
spec, profile_id = _resolve_spec(db, user, body.profile_id, body.spec)
display_name = (body.display_name or "").strip() or None
try:
@@ -532,7 +555,12 @@ def _clean_url(raw) -> str | None:
return None
if not url.startswith(("http://", "https://")):
return None
# The third site in this module handing user input to `urlparse`, which raises on a malformed
# authority. This one already has a "not a plausible URL" answer, so the typo belongs in it.
try:
parsed = urlparse(url)
except ValueError:
return None
if not parsed.netloc:
return None
return url[:2048]
backend/app/routes/playlists.py
+22
View File
@@ -20,10 +20,30 @@ from app.sync.playlists import (
repull_playlist,
sync_user_playlists,
)
from app.ratelimit import RateLimiter
from app.youtube.client import YouTubeClient, YouTubeError
router = APIRouter(prefix="/api/playlists", tags=["playlists"])
# SEC-14, same reasoning as the sync module's limiter: a playlist pull lists every playlist the
# caller owns and then every item in each, against the instance-wide YouTube budget. Its own
# limiter rather than a shared one — spending the manual subscription-import allowance must not
# also close the playlist button, and vice versa.
#
# It guards EVERY route that spends under QuotaAction.PLAYLISTS_PULL, not just the bulk one: a
# scripted loop over the single-playlist re-pull drains the same shared budget, and limiting one of
# two routes with the same quota action is the asymmetric fix this codebase keeps re-learning.
# Deliberately NOT on the push side (`push-plan` / `push`): those run on every push attempt,
# including the no-op "already up to date" path, so a 3-per-10-minute allowance would refuse
# legitimate work. Their exposure is tracked as SEC-18 instead.
_playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600)
def _check_pull_cooldown(user: User) -> None:
"""Shared gate so the two PLAYLISTS_PULL routes cannot drift apart again."""
if not _playlist_pull_limiter.allow(str(user.id)):
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
def _own_playlist(db: Session, user: User, playlist_id: int) -> Playlist:
pl = db.get(Playlist, playlist_id)
@@ -284,6 +304,7 @@ def sync_youtube(
status_code=403,
detail="Connect YouTube (read access) to sync your playlists.",
)
_check_pull_cooldown(user)
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return sync_user_playlists(db, user)
@@ -304,6 +325,7 @@ def revert_youtube(
raise HTTPException(
status_code=403, detail="Connect YouTube (read access) to sync your playlists."
)
_check_pull_cooldown(user)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return repull_playlist(db, user, pl)
backend/app/routes/setup.py
+8 -1
View File
@@ -16,6 +16,7 @@ from sqlalchemy.orm import Session
from app import email as email_mod
from app import mailtext
from app import state, sysconfig
from app.utils import canonical_email
from app.auth import PASSWORD_MIN_LEN
from app.db import get_db
from app.models import User
@@ -70,7 +71,13 @@ def setup_admin(body: SetupAdminIn, db: Session = Depends(require_setup)) -> dic
raise HTTPException(
status_code=400, detail=f"Password must be at least {PASSWORD_MIN_LEN} characters."
)
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
# Resolve canonically, not by the literal string (DX-17). The wizard can be re-run, and E9.5's
# whole point is that two spellings of one Gmail mailbox are one person — matching literally here
# would let a second run under the other spelling create the DUPLICATE admin row the epic exists
# to prevent. Every other lookup went canonical in S3; this one was outside that diff.
user = db.execute(
select(User).where(User.email_canonical == canonical_email(email))
).scalar_one_or_none()
if user is None:
user = User(email=email, role="admin")
db.add(user)
backend/app/routes/sync.py
+10
View File
@@ -7,12 +7,20 @@ from app.audit import AuditAction
from app.auth import admin_user, current_user, has_read_scope, require_human
from app.db import get_db
from app.models import Channel, Subscription, User, Video
from app.ratelimit import RateLimiter
from app.sync.runner import estimate_deep_backfill
from app.sync.subscriptions import import_subscriptions
from app.scheduler import running_job_ids
router = APIRouter(prefix="/api/sync", tags=["sync"])
# SEC-14: the YouTube API budget is SHARED across the whole instance, and a subscription import
# walks every page of the caller's subscriptions — so one user holding down a manual sync button can
# spend the day's units for everyone. Keyed by user id, not client IP: the cost belongs to the
# account, and the same person on a second device is the same drain. 3 per 10 minutes leaves manual
# re-syncing entirely usable (the scheduler does the routine work anyway) while removing the loop.
_sync_limiter = RateLimiter(max_events=3, window_seconds=600)
@router.post("/subscriptions")
def sync_subscriptions(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
@@ -22,6 +30,8 @@ def sync_subscriptions(user: User = Depends(require_human), db: Session = Depend
status_code=403,
detail="Connect your YouTube account (read access) to import subscriptions.",
)
if not _sync_limiter.allow(str(user.id)):
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
with quota.measured(db, user.id, quota.QuotaAction.SUBSCRIPTIONS_PULL) as m:
result = import_subscriptions(db, user)
result["quota_used_estimate"] = m.used
backend/tests/test_abuse_limits.py
+112
View File
@@ -0,0 +1,112 @@
"""E12 · S3 — the route-level halves of SEC-12 and SEC-14.
`test_urlguard.py` pins the address judgement itself; this pins that the ENQUEUE route actually
consults it and answers in the structured `{code, reason}` shape the frontend localizes, and that a
manual sync trigger stops spending the instance-wide YouTube budget when held down.
DB-backed (the `as_user` fixture needs real rows) — skipped when no database is reachable.
"""
from app.downloads import urlguard
def test_enqueueing_a_loopback_url_is_refused_with_a_structured_detail(
db, as_user, make_user, monkeypatch
):
user = make_user()
db.commit()
monkeypatch.setattr(
urlguard.socket, "getaddrinfo", lambda *a, **kw: [(None, None, None, "", ("127.0.0.1", 0))]
)
r = as_user(user).post("/api/downloads", json={"source": "http://redis.internal:6379/x"})
assert r.status_code == 422
# The shape matters as much as the refusal: a bare English string under a bilingual UI is the
# thing the canonical-patterns table forbids.
assert r.json()["detail"] == {"code": "url_blocked", "reason": urlguard.REASON_PRIVATE}
def test_a_public_url_gets_past_the_guard(db, as_user, make_user, monkeypatch):
# Mutation check for the test above: with a public answer the request must fail LATER (or
# succeed), never with url_blocked — otherwise the guard would be refusing everything and the
# first test would still pass.
user = make_user()
db.commit()
monkeypatch.setattr(
urlguard.socket,
"getaddrinfo",
lambda *a, **kw: [(None, None, None, "", ("93.184.216.34", 0))],
)
r = as_user(user).post("/api/downloads", json={"source": "https://example.com/v"})
detail = r.json().get("detail") if r.status_code >= 400 else None
assert not (isinstance(detail, dict) and detail.get("code") == "url_blocked")
def test_youtube_sources_never_consult_the_guard(db, as_user, make_user, monkeypatch):
# A YouTube job resolves to a fixed public host from an 11-char id, so there is no
# user-controlled destination — resolving one would be pure latency on every enqueue.
def boom(*a, **kw):
raise AssertionError("the URL guard resolved DNS for a YouTube source")
user = make_user()
db.commit()
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
as_user(user).post("/api/downloads", json={"source": "dQw4w9WgXcQ"})
def test_manual_subscription_sync_goes_on_cooldown(db, as_user, make_user, monkeypatch):
from app.routes import sync as syncroutes
user = make_user()
db.commit()
# Neither the YouTube grant nor the import itself is under test — the limiter is.
monkeypatch.setattr(syncroutes, "has_read_scope", lambda u: True)
monkeypatch.setattr(syncroutes, "import_subscriptions", lambda d, u: {"imported": 0})
c = as_user(user)
codes = [c.post("/api/sync/subscriptions").status_code for _ in range(4)]
# 3 per 10 minutes: the first three pass, the fourth is refused.
assert codes[:3] == [200, 200, 200]
assert codes[3] == 429
assert c.post("/api/sync/subscriptions").json()["detail"] == {"code": "sync_cooldown"}
def test_the_cooldown_is_per_user_not_global(db, as_user, make_user, monkeypatch):
# Keyed by user id on purpose: the shared YouTube budget is spent per account, and one user
# exhausting their allowance must not lock everyone else out of their own sync button.
from app.routes import sync as syncroutes
a, b = make_user(), make_user()
db.commit()
monkeypatch.setattr(syncroutes, "has_read_scope", lambda u: True)
monkeypatch.setattr(syncroutes, "import_subscriptions", lambda d, u: {"imported": 0})
ca = as_user(a)
for _ in range(4):
ca.post("/api/sync/subscriptions")
assert ca.post("/api/sync/subscriptions").status_code == 429
assert as_user(b).post("/api/sync/subscriptions").status_code == 200
def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_user, monkeypatch):
# SEC-14, the sweep half: `revert-youtube` spends under the SAME QuotaAction.PLAYLISTS_PULL key
# as the bulk `sync-youtube`, so limiting only one of them left the identical drain open one
# route over. Both must draw on the one allowance.
from app.models import Playlist
from app.routes import playlists as plroutes
user = make_user()
pl = Playlist(user_id=user.id, name="Linked", source="youtube", yt_playlist_id="PL123")
db.add(pl)
db.commit()
monkeypatch.setattr(plroutes, "has_read_scope", lambda u: True)
monkeypatch.setattr(plroutes, "repull_playlist", lambda d, u, p: {"ok": True})
monkeypatch.setattr(plroutes, "sync_user_playlists", lambda d, u: {"ok": True})
c = as_user(user)
codes = [c.post(f"/api/playlists/{pl.id}/revert-youtube").status_code for _ in range(4)]
assert codes[:3] == [200, 200, 200]
assert codes[3] == 429
# One allowance, not one each: the bulk route is now on cooldown too.
r = c.post("/api/playlists/sync-youtube")
assert r.status_code == 429
assert r.json()["detail"] == {"code": "sync_cooldown"}
backend/tests/test_download_errors.py
+20
View File
@@ -1,5 +1,6 @@
"""Category classification of raw yt-dlp / ffmpeg download-failure strings (S2)."""
from app.downloads import errors as dlerrors
from app.downloads.errors import classify
@@ -54,3 +55,22 @@ def test_login_beats_generic_unavailable_when_both_present():
"This video is unavailable"
)
assert classify(msg) == "login_required"
def test_our_own_worker_refusals_classify_as_themselves():
# The worker raises these itself (not yt-dlp), and the rule that catches each one matches only a
# fragment of the text. Pinning message and rule together is what stops an edit to the wording
# from silently orphaning the category — the UI would then fall back to "Download failed".
assert classify(dlerrors.QUOTA_FULL_MESSAGE) == "quota_exceeded"
def test_a_url_jobs_network_failure_is_redacted_but_youtube_keeps_its_text():
# SEC-12: the raw transport text for a user-chosen host:port is a port-scanner readout, so it is
# replaced for `url` jobs. Everything else must survive verbatim, or a real diagnosis is lost.
raw = "ERROR: Unable to download webpage: <urlopen error [Errno 111] Connection refused>"
assert dlerrors.user_error_text("url", raw, "network") == dlerrors.NETWORK_REDACTED_MESSAGE
# Same failure, but the destination was never user-chosen — nothing to leak, keep the detail.
assert dlerrors.user_error_text("youtube", raw, "network") == raw
# A user-chosen host, but a category that says something real about the CONTENT, not the network.
login = "ERROR: [facebook] 123: You must be logged in to see this post"
assert dlerrors.user_error_text("url", login, "login_required") == login
backend/tests/test_download_limits.py
+109
View File
@@ -10,6 +10,8 @@ Uses the `db` fixture (real Postgres) — skipped when no database is reachable.
its OWN session, so every test COMMITS its setup before calling it.
"""
from app.config import settings
from app.downloads import pipeline, quota
from app.models import DownloadJob, DownloadQuota, MediaAsset, User
from app.worker import _claim_job
@@ -169,3 +171,110 @@ def test_list_downloads_is_capped(db, monkeypatch):
monkeypatch.setattr(dl, "_MAX_LIST", 2)
rows = dl.list_downloads(user=u, db=db)
assert len(rows) == 2 # bounded by the cap, not the 3 rows on disk
# --- E12 · S3 · the worker-side max_bytes remainder ------------------------------------------
def _sized_asset(db, size_bytes, ref):
a = MediaAsset(
source_kind="url",
source_ref=ref,
format_sig="s",
status="ready",
ref_count=1,
size_bytes=size_bytes,
)
db.add(a)
db.flush()
return a
def test_remaining_bytes_is_none_for_an_unlimited_user(db):
u = User(email="unlimited@example.com")
db.add(u)
db.flush()
_quota(db, u.id, max_concurrent=2, unlimited=True)
db.flush()
# None, not a number: "no cap" and "no budget left" must never be the same value downstream —
# build_ydl_opts leaves max_filesize unset for both, so only the caller can tell them apart.
assert quota.remaining_bytes(db, u.id) is None
def test_remaining_bytes_is_the_unspent_part_of_the_cap(db):
u = User(email="partial@example.com")
db.add(u)
db.flush()
db.add(
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False)
)
a = _sized_asset(db, 400, "partial-1")
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
db.flush()
assert quota.remaining_bytes(db, u.id) == 600
def test_remaining_bytes_is_zero_once_the_cap_is_met(db):
# THE case this exists for: `check_enqueue` let these jobs in while the user was under the cap,
# and an earlier one has filled it since. Zero is what makes the worker refuse; if this ever
# returned the raw negative or None instead, the user already over quota would be the only one
# downloading with no size ceiling at all.
u = User(email="atcap@example.com")
db.add(u)
db.flush()
db.add(
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False)
)
a = _sized_asset(db, 1500, "atcap-1") # overshot: one job finished larger than the remainder
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
db.flush()
assert quota.remaining_bytes(db, u.id) == 0
def test_a_refusal_for_one_user_does_not_poison_the_shared_asset(db, monkeypatch, tmp_path):
"""A user at their cap fails their own job, and another user's queued job for the same video
still downloads.
MediaAssets are deduped across users, and the generic failure handler writes the message onto
that shared row — correct for an upstream failure, wrong for "YOUR storage is full", which
`worker._process_job` would then hand verbatim to someone with an empty library.
"""
monkeypatch.setattr(settings, "download_root", str(tmp_path))
over, other = User(email="over@example.com"), User(email="other@example.com")
db.add_all([over, other])
db.flush()
db.add(
DownloadQuota(
user_id=over.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False
)
)
full = _sized_asset(db, 1500, "already-held")
_job(db, over.id, queue_pos=0, status="done", asset_id=full.id)
# `downloading` is the state the worker leaves it in — `_claim_asset` compare-and-sets it before
# the child runs — and `_reset_asset_pending` only acts on that status. Seeded as `pending` this
# assertion would pass with the release deleted.
shared = MediaAsset(
source_kind="youtube",
source_ref="x" * 11,
format_sig="s",
status="downloading",
ref_count=2,
)
db.add(shared)
db.flush()
mine = _job(db, over.id, queue_pos=1, status="running", asset_id=shared.id)
theirs = _job(db, other.id, queue_pos=2, asset_id=shared.id)
db.commit()
# The refusal happens before extraction, so nothing needs stubbing — if it ever stops happening
# first, this test starts reaching the network, which is its own signal.
pipeline._download(mine.id, shared.id, "youtube", "x" * 11, {}, over.id)
db.expire_all()
assert db.get(DownloadJob, mine.id).status == "error"
assert db.get(DownloadJob, mine.id).error_code == "quota_exceeded"
# The asset is released, NOT marked error — otherwise the next line's job inherits the refusal.
assert db.get(MediaAsset, shared.id).status == "pending"
assert db.get(MediaAsset, shared.id).error is None
assert db.get(DownloadJob, theirs.id).status == "queued"
backend/tests/test_email_canonical.py
+25
View File
@@ -345,3 +345,28 @@ def test_me_reports_the_google_address_only_when_it_differs(client, db, make_use
== 200
)
assert client.get("/api/me").json()["google_email"] == "me@gmail.com"
# --- DX-17 · the setup wizard resolves its admin canonically ---------------------------------
def test_rerunning_the_wizard_with_another_gmail_spelling_updates_the_same_admin(db):
"""The wizard can be re-run, and E9.5's premise is that two spellings of one Gmail mailbox are one
person. Resolving by the literal string here would create a SECOND admin row — exactly the
duplicate the epic exists to prevent. This site was outside S3's sweep (DX-17).
Calls the handler directly: `require_setup` is an instance-state gate, not the behaviour under
test, and the test fixtures present an already-configured instance."""
from app.routes.setup import SetupAdminIn, setup_admin
setup_admin(SetupAdminIn(email="a.b.c@gmail.com", password=_UAT_PW), db)
setup_admin(SetupAdminIn(email="abc@gmail.com", password="another good passphrase"), db)
admins = db.execute(select(User).where(User.role == "admin")).scalars().all()
assert len(admins) == 1, [u.email for u in admins]
# The address stays AS TYPED on the row it matched; only identity is canonical.
assert admins[0].email == "a.b.c@gmail.com"
# And the second run's password is the live one — it updated, it did not fork.
from app.security import verify_password
assert verify_password("another good passphrase", admins[0].password_hash)
backend/tests/test_formats.py
+49
View File
@@ -109,3 +109,52 @@ def test_browser_transcode_flags_universal_playability():
assert F("h264", "aac", False) == (True, False, False)
# A failed probe / missing stream is never re-encoded.
assert F(None, None, True) == (False, False, False)
# --- SEC-13 · one enqueue is one bounded download -------------------------------------------
def test_a_playlist_url_can_only_yield_one_item():
# `noplaylist` alone only demotes a video-inside-a-playlist URL to that video; a URL that is
# purely a playlist or channel has no such video, so without this it expands into every entry.
opts = _opts({})
assert opts["noplaylist"] is True
assert opts["playlist_items"] == "1"
def test_the_remaining_quota_becomes_a_hard_size_ceiling():
opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000)
assert opts["max_filesize"] == 5_000_000
def test_no_budget_means_no_size_filter():
# Unlimited users pass None; a non-positive remainder is the quota layer's refusal to make at
# enqueue, not an impossible yt-dlp filter to express here.
for budget in (None, 0, -1):
assert "max_filesize" not in formats.build_ydl_opts(
{}, "/tmp/x", lambda d: None, max_bytes=budget
)
def test_a_source_adapter_cannot_raise_the_size_ceiling():
# The adapter merge happens after max_filesize is set, so this pins that no adapter option can
# widen a storage cap — the mirror of the cookiefile rule, in the opposite direction.
opts = formats.build_ydl_opts(
{}, "/tmp/x", lambda d: None, extra={"max_filesize": 999_999_999}, max_bytes=5_000_000
)
assert opts["max_filesize"] == 5_000_000
def test_a_join_cut_list_is_capped():
# An accurate join emits two filters per segment into one ffmpeg filter_complex, so an unbounded
# list is an unbounded command line. The surplus is dropped; the legitimate head survives.
segs = [{"start_s": i, "end_s": i + 1} for i in range(500)]
out = editmod.normalize_edit_spec({"segments": segs})
assert len(out["segments"]) == editmod._MAX_SEGMENTS
assert out["segments"][0] == {"start_s": 0.0, "end_s": 1.0}
def test_a_normal_cut_list_is_untouched():
segs = [{"start_s": 0, "end_s": 5}, {"start_s": 10, "end_s": 12}]
out = editmod.normalize_edit_spec({"segments": segs})
assert len(out["segments"]) == 2
backend/tests/test_set_password_link.py
+280
View File
@@ -0,0 +1,280 @@
"""E12 · S5 — the admin-issued set-password link.
One mechanism answering three things: SEC-15 (a no-SMTP instance auto-verifies every registration, so
a pre-empted address cannot be recovered by mail), SEC-16 (a pre-empted address stays takeable until
its owner resets, and reset needs working mail), and the everyday "no SMTP and someone forgot theirs".
The property that must not regress is the one that makes it honest: an admin-issued link proves
NOTHING about the mailbox — the admin hands it over by whatever means they like — so it must not
confirm the address the way a mailed reset does.
"""
from app.models import AuthToken
def _link(client_admin, user_id: int):
r = client_admin.post(f"/api/admin/users/{user_id}/set-password-link")
assert r.status_code == 200, r.text
return r.json()["url"].split("#reset=", 1)[1]
def test_the_link_sets_a_password_without_confirming_the_address(db, as_user, make_user, client):
"""The whole point of a separate token kind. A mailed reset arrives IN the mailbox, so completing
it proves ownership and confirms the address. An admin-issued link does not, and claiming
otherwise would let an admin confirm an address nobody ever proved."""
admin = make_user(role="admin")
target = make_user(email_verified=False)
db.commit()
token = _link(as_user(admin), target.id)
r = client.post(
"/auth/password-reset/confirm", json={"token": token, "password": "a brand new passphrase"}
)
assert r.status_code == 200, r.text
db.refresh(target)
assert target.password_hash is not None
assert target.email_verified is False # the address is still unproven — and says so
def test_a_mailed_reset_still_confirms_the_address(db, as_user, make_user, client):
"""The mutation check for the test above: if `email_verified` were simply never set, this would
fail — and the rescue path for "registered but never clicked the link" would be gone."""
from datetime import timedelta
from app.routes.auth import _issue_token
user = make_user(email_verified=False)
db.commit()
token = _issue_token(db, user, "reset", timedelta(hours=1))
r = client.post(
"/auth/password-reset/confirm", json={"token": token, "password": "a brand new passphrase"}
)
assert r.status_code == 200, r.text
db.refresh(user)
assert user.email_verified is True
def test_using_the_link_evicts_whoever_held_the_old_password(db, as_user, make_user, client):
"""SEC-16's actual remedy: the squatter's session dies the moment the real owner sets a password.
The epoch bump is what does it, and it is the reason this reuses the reset confirm path rather
than writing a second one that could forget to bump."""
target = make_user()
db.commit()
before = target.session_epoch
admin = make_user(role="admin")
db.commit()
token = _link(as_user(admin), target.id)
client.post(
"/auth/password-reset/confirm", json={"token": token, "password": "the owner's own choice"}
)
db.refresh(target)
assert target.session_epoch != before
def test_issuing_does_not_touch_the_current_password(db, as_user, make_user):
"""An admin misclick must not lock anyone out. Issuing is an offer; only USING it changes
anything — which is also why it is safe to offer on a perfectly healthy account."""
target = make_user()
db.commit()
original = target.password_hash
admin = make_user(role="admin")
db.commit()
_link(as_user(admin), target.id)
db.refresh(target)
assert target.password_hash == original
assert target.password_revoked_at is None
def test_the_link_is_single_use(db, as_user, make_user, client):
target = make_user()
admin = make_user(role="admin")
db.commit()
token = _link(as_user(admin), target.id)
first = client.post(
"/auth/password-reset/confirm", json={"token": token, "password": "first passphrase here"}
)
second = client.post(
"/auth/password-reset/confirm", json={"token": token, "password": "second passphrase here"}
)
assert first.status_code == 200
assert second.status_code == 400
def test_a_new_link_burns_the_outstanding_ones(db, as_user, make_user, client):
"""Spending any password token burns the siblings of BOTH kinds. Without that, a spent reset
would leave a live admin link behind — a second key to a door someone just changed the lock on."""
target = make_user()
admin = make_user(role="admin")
db.commit()
first = _link(as_user(admin), target.id)
second = _link(as_user(admin), target.id)
ok = client.post(
"/auth/password-reset/confirm", json={"token": second, "password": "the newer passphrase"}
)
assert ok.status_code == 200
stale = client.post(
"/auth/password-reset/confirm", json={"token": first, "password": "the older passphrase"}
)
assert stale.status_code == 400
def test_the_probe_recognises_an_admin_link(db, as_user, make_user, client):
"""`/auth/password-reset/check` is what stops the card drawing a form for a dead link. It peeked
only at `reset` tokens, so without teaching it the new kind every admin-issued link would open
onto "this link is spent"."""
target = make_user()
admin = make_user(role="admin")
db.commit()
token = _link(as_user(admin), target.id)
r = client.post("/auth/password-reset/check", json={"token": token})
assert r.json() == {"ok": True}
def test_only_an_admin_can_mint_one(db, as_user, make_user):
plain = make_user()
victim = make_user()
db.commit()
r = as_user(plain).post(f"/api/admin/users/{victim.id}/set-password-link")
assert r.status_code in (401, 403)
assert db.query(AuthToken).filter(AuthToken.kind == "setpw").count() == 0
def test_the_shared_demo_account_is_refused(db, as_user, make_user):
"""The demo account is shared, so handing anyone a way to change its password hands them
everyone else's session on it."""
admin = make_user(role="admin")
demo = make_user(is_demo=True)
db.commit()
r = as_user(admin).post(f"/api/admin/users/{demo.id}/set-password-link")
assert r.status_code == 400
assert r.json()["detail"]["reason"] == "demo"
def test_the_access_panel_reports_the_gates(db, as_user, make_user):
admin = make_user(role="admin")
target = make_user(email_verified=False, is_active=False)
db.commit()
body = as_user(admin).get(f"/api/admin/users/{target.id}/access").json()
assert body["email_verified"] is False
assert body["is_active"] is False
assert set(body) >= {
"is_allowed",
"can_activate",
"email_configured",
"password_revoked_at",
"google_email",
"has_password",
"has_google",
}
def test_the_panel_names_a_google_address_only_when_it_differs(db, as_user, make_user):
""" "Connected" and "connected as someone else" are different statements, and only the second one
explains a confusing sign-in. Repeating the account's own address would be noise on every row."""
admin = make_user(role="admin")
same = make_user(email="same@example.com")
same.google_email = "same@example.com"
other = make_user(email="typed@example.com")
other.google_email = "google-primary@example.com"
db.commit()
c = as_user(admin)
assert c.get(f"/api/admin/users/{same.id}/access").json()["google_email"] is None
assert (
c.get(f"/api/admin/users/{other.id}/access").json()["google_email"]
== "google-primary@example.com"
)
def test_the_link_is_audited_without_recording_the_link(db, as_user, make_user):
"""The audit log is read by humans and kept long; a live credential-setting link is the last
thing that belongs in it. That one was ISSUED, by whom, for whom, is the auditable fact."""
from app.models import AuditLog
admin = make_user(role="admin")
target = make_user()
db.commit()
url = as_user(admin).post(f"/api/admin/users/{target.id}/set-password-link").json()["url"]
secret = url.split("#reset=", 1)[1]
rows = db.query(AuditLog).filter(AuditLog.action == "user_setpw_link").all()
assert len(rows) == 1
assert rows[0].summary == target.email
assert secret not in str(rows[0].summary) + str(rows[0].before) + str(rows[0].after)
def test_a_google_only_account_can_be_given_a_password(db, as_user, make_user, client):
"""Deliberately not gated on the account's state: the admin's real case is whichever one it is,
and narrowing this would only make it refuse exactly then."""
admin = make_user(role="admin")
google_only = make_user(password=None)
db.commit()
assert google_only.password_hash is None
token = _link(as_user(admin), google_only.id)
r = client.post(
"/auth/password-reset/confirm", json={"token": token, "password": "now it has one too"}
)
assert r.status_code == 200
db.refresh(google_only)
assert google_only.password_hash is not None
def test_an_unknown_user_is_a_404(db, as_user, make_user):
admin = make_user(role="admin")
db.commit()
assert as_user(admin).post("/api/admin/users/999999/set-password-link").status_code == 404
assert as_user(admin).get("/api/admin/users/999999/access").status_code == 404
def test_the_issued_token_is_stored_hashed(db, as_user, make_user):
"""Same rule as every other token here: the DB holds a hash, so a database read cannot hand
someone a working link."""
admin = make_user(role="admin")
target = make_user()
db.commit()
raw = _link(as_user(admin), target.id)
rows = db.query(AuthToken).filter(AuthToken.kind == "setpw").all()
assert len(rows) == 1
assert rows[0].token_hash != raw
assert raw not in rows[0].token_hash
def test_the_link_outlives_the_one_hour_reset_window(db, as_user, make_user):
"""It travels out of band — the admin has to actually reach the person — so the mailed reset's
one hour is far too short."""
from datetime import datetime, timedelta, timezone
admin = make_user(role="admin")
target = make_user()
db.commit()
_link(as_user(admin), target.id)
row = db.query(AuthToken).filter(AuthToken.kind == "setpw").one()
assert row.expires_at - datetime.now(timezone.utc) > timedelta(hours=2)
def test_users_listing_still_serializes(db, as_user, make_user):
"""The list gained two fields; this pins that it did not gain the two per-row QUERIES the panel
endpoint exists to keep out of it."""
admin = make_user(role="admin")
db.commit()
rows = as_user(admin).get("/api/admin/users").json()
assert rows and "password_revoked_at" in rows[0] and "google_email" in rows[0]
assert "can_activate" not in rows[0]
backend/tests/test_urlguard.py
+99
View File
@@ -0,0 +1,99 @@
"""SEC-12 · the download URL guard.
The Download Center hands a user-supplied URL to yt-dlp on the worker, so the server fetches
whatever the user typed. These pin that a URL resolving into our own network is refused, that the
refusal reasons stay distinct (an unresolvable host is not the same message as a blocked one), and
that a normal public URL still goes through.
DNS is stubbed rather than live: the point under test is the address JUDGEMENT, and a test that
resolves real names would be slow, offline-fragile, and would silently start passing if the guard
stopped resolving at all.
"""
import pytest
from app.downloads import urlguard
def _stub_dns(monkeypatch, addrs):
"""Make every lookup answer with `addrs` (a list of textual IPs)."""
def fake_getaddrinfo(host, port, *a, **kw):
return [(None, None, None, "", (addr, port or 0)) for addr in addrs]
monkeypatch.setattr(urlguard.socket, "getaddrinfo", fake_getaddrinfo)
def _blocked_reason(url):
with pytest.raises(urlguard.BlockedUrl) as e:
urlguard.check_url(url)
return e.value.reason
@pytest.mark.parametrize(
"addr",
[
"127.0.0.1", # loopback
"10.1.2.3", # RFC1918
"192.168.0.5", # RFC1918
"172.16.9.9", # RFC1918
"169.254.169.254", # link-local — the cloud metadata endpoint
"0.0.0.0", # unspecified
"::1", # IPv6 loopback
"fe80::1", # IPv6 link-local
"fc00::1", # IPv6 unique-local
"::ffff:127.0.0.1", # loopback wearing an IPv6 hat
],
)
def test_addresses_on_our_own_network_are_refused(monkeypatch, addr):
_stub_dns(monkeypatch, [addr])
assert _blocked_reason("http://whatever.example/video") == urlguard.REASON_PRIVATE
def test_one_bad_answer_in_a_round_robin_blocks_the_whole_host(monkeypatch):
# A host answering with both a public and an internal address must not become a coin flip:
# yt-dlp would pick for itself, so the guard refuses if ANY answer is forbidden.
_stub_dns(monkeypatch, ["93.184.216.34", "127.0.0.1"])
assert _blocked_reason("https://mixed.example/v") == urlguard.REASON_PRIVATE
def test_a_public_address_passes(monkeypatch):
_stub_dns(monkeypatch, ["93.184.216.34"])
urlguard.check_url("https://example.com/watch?v=abc") # does not raise
def test_unresolvable_is_its_own_reason(monkeypatch):
import socket as _socket
def boom(*a, **kw):
raise _socket.gaierror("nope")
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
# Distinct from PRIVATE on purpose: "we couldn't look that up" and "we refuse to fetch that"
# are different things to tell a user, and the frontend renders them differently.
assert _blocked_reason("https://nx.example/v") == urlguard.REASON_UNRESOLVABLE
@pytest.mark.parametrize("url", ["file:///etc/passwd", "ftp://host/x", "gopher://host/1"])
def test_non_http_schemes_are_refused_before_any_lookup(monkeypatch, url):
def boom(*a, **kw): # a scheme rejection must not have resolved anything first
raise AssertionError("DNS was consulted for a non-http scheme")
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
assert _blocked_reason(url) == urlguard.REASON_SCHEME
def test_a_url_without_a_host_is_refused():
assert _blocked_reason("http:///just/a/path") == urlguard.REASON_NO_HOST
def test_a_malformed_url_is_a_refusal_not_a_crash(monkeypatch):
# Both shapes raise ValueError LATE — `urlparse` on an unclosed bracket, `.port` only when the
# property is READ — and `.hostname` happily returns first. Unguarded, either turns a user typo
# into a 500 that escapes the caller's `except BlockedUrl` and loses the localized detail this
# module exists to produce.
_stub_dns(monkeypatch, ["93.184.216.34"])
assert _blocked_reason("http://example.com:99999/v.mp4") == urlguard.REASON_MALFORMED
assert _blocked_reason("http://example.com:abc/v.mp4") == urlguard.REASON_MALFORMED
assert _blocked_reason("http://[::1/v.mp4") == urlguard.REASON_MALFORMED
assert _blocked_reason("http://[example.com]/v.mp4") == urlguard.REASON_MALFORMED
docker-compose.selfhost.yml
+12
View File
@@ -53,6 +53,11 @@ services:
# Downloaded media. Defaults to a Docker-managed named volume; set DOWNLOAD_HOST_PATH in .env
# to a host directory (e.g. one your Plex server reads) to keep the Plex-style tree there.
- ${DOWNLOAD_HOST_PATH:-siftlode_downloads}:/downloads
# Plex library, read-only. The Plex module streams files itself, so it needs to SEE them: set
# PLEX_MEDIA_HOST_PATH in .env to the host directory holding the library (the local side of the
# admin "path map", whose default is /data=/plex-media). Left unset it falls back to an empty
# named volume — harmless, and the Plex module is off by default.
- ${PLEX_MEDIA_HOST_PATH:-siftlode_plex_media}:/plex-media:ro
security_opt:
- no-new-privileges:true
cap_drop:
@@ -74,6 +79,9 @@ services:
worker:
image: git.b1fr0st.eu/peter/siftlode:${IMAGE_TAG:-latest}
command: ["python", "-m", "app.worker"]
# tini as PID 1: reaps orphaned ffmpeg/yt-dlp grandchildren when the watchdog hard-kills a
# download child's process group (the worker reaps its own direct children via subprocess.wait).
init: true
env_file:
- .env
environment:
@@ -93,6 +101,8 @@ services:
networks: [internal]
volumes:
- ${DOWNLOAD_HOST_PATH:-siftlode_downloads}:/downloads
# Same read-only Plex library as the API — the worker resolves library paths too.
- ${PLEX_MEDIA_HOST_PATH:-siftlode_plex_media}:/plex-media:ro
security_opt:
- no-new-privileges:true
cap_drop:
@@ -114,6 +124,8 @@ services:
volumes:
siftlode_pgdata:
siftlode_downloads:
# Empty fallback for the Plex media mount when PLEX_MEDIA_HOST_PATH is unset (Plex module off).
siftlode_plex_media:
networks:
internal:
docker-compose.yml
+12
View File
@@ -38,6 +38,11 @@ services:
# Downloaded media. Defaults to a Docker-managed named volume; set DOWNLOAD_HOST_PATH in .env
# to a host directory (e.g. one your Plex server reads) to keep the Plex-style tree there.
- ${DOWNLOAD_HOST_PATH:-downloads}:/downloads
# Plex library, read-only. The Plex module streams files itself, so it needs to SEE them: set
# PLEX_MEDIA_HOST_PATH in .env to the host directory holding the library (the local side of the
# admin "path map", whose default is /data=/plex-media). Left unset it falls back to an empty
# named volume — harmless, and the Plex module is off by default.
- ${PLEX_MEDIA_HOST_PATH:-plex_media}:/plex-media:ro
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; exit(0 if urllib.request.urlopen('http://localhost:8000/healthz').status == 200 else 1)"]
interval: 15s
@@ -57,6 +62,9 @@ services:
GIT_SHA: ${GIT_SHA:-unknown}
BUILD_DATE: ${BUILD_DATE:-}
command: ["python", "-m", "app.worker"]
# tini as PID 1: reaps orphaned ffmpeg/yt-dlp grandchildren when the watchdog hard-kills a
# download child's process group (the worker reaps its own direct children via subprocess.wait).
init: true
env_file: .env
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-siftlode}:${POSTGRES_PASSWORD:-siftlode}@db:5432/${POSTGRES_DB:-siftlode}
@@ -73,6 +81,8 @@ services:
condition: service_started
volumes:
- ${DOWNLOAD_HOST_PATH:-downloads}:/downloads
# Same read-only Plex library as the API — the worker resolves library paths too.
- ${PLEX_MEDIA_HOST_PATH:-plex_media}:/plex-media:ro
restart: unless-stopped
# PO-token provider sidecar: mints YouTube Proof-of-Origin tokens so the worker beats bot-detection
@@ -85,3 +95,5 @@ services:
volumes:
pgdata:
downloads:
# Empty fallback for the Plex media mount when PLEX_MEDIA_HOST_PATH is unset (Plex module off).
plex_media:
docs/self-hosting.md
+188 -7
View File
@@ -115,13 +115,34 @@ DOWNLOAD_HOST_PATH=/mnt/media/youtube
The directory must be writable by the container user (uid `1000`): `sudo chown -R 1000:1000 <dir>`.
## Plex library (optional)
If you run a **Plex** server, Siftlode can browse and play your movies and shows beside your YouTube
feed. Two things have to line up:
1. **Server URL + token**, entered as admin under **Configuration → Plex**. That is how Siftlode reads
your libraries.
2. **The media files themselves**, mounted into the container read-only. Siftlode streams the files
directly rather than proxying Plex, so it has to be able to open them.
Point `PLEX_MEDIA_HOST_PATH` at the directory holding your library and re-run `up -d`:
```bash
PLEX_MEDIA_HOST_PATH=/mnt/media
```
It appears inside the container at `/plex-media`, read-only. Siftlode then rewrites the paths Plex
reports into that mount using the **path map** on the same admin page — its default, `/data=/plex-media`,
means "wherever Plex says `/data/movies/…`, look in `/plex-media/movies/…`". Set the left-hand side to
whatever prefix your Plex server uses, and mount the matching host directory so the rest of the path
still lines up.
Leave `PLEX_MEDIA_HOST_PATH` unset if you don't use Plex — the mount falls back to an empty volume and
the module stays off.
## Day-to-day
```bash
# Update to the latest release
docker compose -f docker-compose.selfhost.yml pull
docker compose -f docker-compose.selfhost.yml up -d
# Logs / status (api = web, worker = downloads/edits)
docker compose -f docker-compose.selfhost.yml logs -f api
docker compose -f docker-compose.selfhost.yml logs -f worker
@@ -131,6 +152,166 @@ docker compose -f docker-compose.selfhost.yml ps
docker compose -f docker-compose.selfhost.yml down
```
Your data (accounts, subscriptions, playlists, the video catalog) lives in the `siftlode_pgdata`
Docker volume — back that up to keep your instance's state. Database migrations run automatically
when the app starts, so updating is just pull + up.
## Backups
**A backup is three things, not one.** Copying only the database gets you an instance that starts and
then cannot read its own secrets.
1. **The database** — accounts, subscriptions, playlists, the video catalog, and every setting you
made in the admin pages. It lives in the `siftlode_pgdata` Docker volume; dump it with the command
below.
2. **`.env`** — in particular `TOKEN_ENCRYPTION_KEY`. Your Google client secret, SMTP password, Plex
token and every stored refresh token are encrypted with it *inside* the database. Restore a dump
against a different key and those values are lost — you would have to clear the affected rows from
`system_config` and re-enter every secret by hand. Keep `.env` beside the dump, and just as private.
3. **Downloaded media** *(only if you use the Download Center)* — the `siftlode_downloads` volume, or
whatever `DOWNLOAD_HOST_PATH` points at. Not needed to bring the instance back; the jobs and their
metadata are in the database, the files themselves are not.
Dump and restore the database with these two commands. They also exist as scripts
(`scripts/backup.sh` + `scripts/restore.sh`, or `backup.ps1` / `restore.ps1` on Windows) if you have
the source checkout:
If you customised `POSTGRES_USER` / `POSTGRES_DB` in `.env`, substitute them for `siftlode` below —
the scripts read them from the environment, these one-liners cannot.
```bash
# Back up. Write to a .part name and rename only on success: redirecting straight into the final
# name creates the file BEFORE pg_dump runs, so a failed dump leaves a plausible-looking 0-byte
# file in backups/ that you will not discover until the day you need it.
mkdir -p backups && out="backups/siftlode-$(date +%Y%m%d-%H%M%S).dump" && \
docker compose -f docker-compose.selfhost.yml exec -T db \
pg_dump -U siftlode -Fc siftlode > "$out.part" \
&& [ -s "$out.part" ] && mv "$out.part" "$out" && echo "wrote $out" \
|| { rm -f "$out.part"; echo "BACKUP FAILED" >&2; false; }
# Restore. Stop the app first: pg_restore --clean DROPs each object in its own transaction, and a
# running api holds locks that make those DROPs queue — abort the wait and you are left with a
# half-dropped database. Restoring with the app down also means nothing writes into a schema that
# is only partly back.
docker compose -f docker-compose.selfhost.yml stop api worker
docker compose -f docker-compose.selfhost.yml exec -T db \
pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
# Start it again only once the restore reported no errors — and, if you are rolling a release back,
# only AFTER pinning the older IMAGE_TAG. Every start runs `alembic upgrade head`, so bringing the
# newer image back here would re-apply the migrations you just restored away. Use `up -d`, not
# `start`: `start` relaunches the existing container, so it ignores the tag you just pinned and
# comes back on the image you are rolling back FROM.
docker compose -f docker-compose.selfhost.yml up -d
```
On Windows, do **not** pipe the dump through PowerShell — it corrupts binary data. Write it inside the
container and copy it out instead. Check the exit code between the two steps and use a timestamped
name: the dump inside the container keeps its name, so a `cp` that runs after a *failed* dump would
copy out the previous run's file and quietly overwrite your only backup with a stale one.
Paste it as ONE block. `throw` only aborts the statement it is in, so as separate pasted lines the
copy would still run after a failed dump — and copy out the previous run's file:
```powershell
& {
# Inside the block, because a scriptblock inherits the caller's preference — under a profile that
# sets SilentlyContinue, every `throw` below would be swallowed and the block would run on.
$ErrorActionPreference = "Stop"
$compose = "docker-compose.selfhost.yml"
New-Item -ItemType Directory -Force -Path backups | Out-Null
$out = "backups\siftlode-$(Get-Date -Format yyyyMMdd-HHmmss).dump"
docker compose -f $compose exec -T db pg_dump -U siftlode -Fc siftlode -f /tmp/s.dump
$code = $LASTEXITCODE
# Remove the in-container dump on the failure path too: leaving it there is how a LATER failed run
# copies out this run's file and overwrites a good backup with a stale one.
if ($code -ne 0) { docker compose -f $compose exec -T db rm -f /tmp/s.dump; throw "BACKUP FAILED - pg_dump returned $code" }
docker compose -f $compose cp db:/tmp/s.dump $out
$code = $LASTEXITCODE
docker compose -f $compose exec -T db rm -f /tmp/s.dump
# Remove the partial file too: `cp` streams, so a mid-stream failure leaves a truncated dump
# under the final name — the plausible-looking bad backup the .part dance above avoids.
if ($code -ne 0) { Remove-Item $out -ErrorAction SilentlyContinue; throw "BACKUP FAILED - could not copy the dump out (exit $code)" }
if ((Get-Item $out).Length -eq 0) { Remove-Item $out; throw "BACKUP FAILED - the dump came out empty" }
"wrote $out"
}
```
## Upgrading
**Take a backup first.** Every start runs any new database migrations automatically, and a migration
is not reversible by pulling the old image back — see rollback below.
```bash
docker compose -f docker-compose.selfhost.yml pull
docker compose -f docker-compose.selfhost.yml up -d
```
That tracks `latest`, which moves whenever a release is cut. **To decide for yourself when you move,
pin a version** by putting it in `.env`:
```bash
IMAGE_TAG=0.80.0
```
Then `pull` + `up -d` stays on that version until you change the line. Released versions and what
changed in them are listed in the app itself, under **Release Notes** in the account menu, and in
[`CHANGELOG.md`](../CHANGELOG.md).
**The registry keeps only the newest few releases, not every version the changelog lists.** A pin is
"decide when you take the next hop", not "stay here forever": once you are several releases behind,
your pinned tag can no longer be pulled — which matters the day you rebuild the host, prune your
Docker images, or move to a new machine. A container that is already running is unaffected (Docker
keeps the image locally), and you can always go back to `latest` or build the image yourself from the
repository. If you want to sit on one version long term, keep your own copy:
`docker save git.b1fr0st.eu/peter/siftlode:0.80.0 -o siftlode-0.80.0.tar`.
### Rollback
If a release misbehaves, set `IMAGE_TAG` back to the previous version and `up -d`. That is enough
**as long as the release did not change the database schema**. If it did, the older image will refuse
to start against the newer schema.
**Restoring your pre-upgrade dump is the recommended way out of that** — it puts the database back
exactly where the old image expects it, in one step, with no revision arithmetic:
```bash
# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. restore the dump you took before upgrading (see Backups above). Leave the app stopped:
# starting it here would run the NEWER image's migrations against the database you just restored.
docker compose -f docker-compose.selfhost.yml exec -T db \
pg_restore -U siftlode -d siftlode --clean --if-exists < backups/siftlode-<timestamp>.dump
# 3. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d
```
If you have no usable dump, you can step the schema back instead — but **not with `alembic downgrade
-1`**. One release routinely ships several migrations (0.80.0 shipped four), so `-1` undoes only the
newest one and leaves the schema at a revision the older image *also* cannot resolve: you lose that
migration's data and still get a crash loop. Downgrade to the revision the target version expects,
which that version can tell you itself:
```bash
# 1. stop the app (leave the database up)
docker compose -f docker-compose.selfhost.yml stop api worker
# 2. ask the version you are rolling back TO which schema revision it expects
IMAGE_TAG=0.79.0 docker compose -f docker-compose.selfhost.yml run --rm api alembic heads
# 3. step the schema back to exactly that revision — run this with the CURRENT (newer) image,
# because it is the one that ships the migrations being undone
docker compose -f docker-compose.selfhost.yml run --rm api alembic downgrade <revision from step 2>
# 4. pin the previous version in .env, then start again
docker compose -f docker-compose.selfhost.yml up -d
```
A downgrade drops the columns the newer version added, so anything stored only in those columns is
gone. That is the price of not having the dump.
### A migration failed on start
The container will keep restarting (`restart: unless-stopped`), and `worker` will sit waiting for an
`api` that never becomes healthy. Read the reason first:
```bash
docker compose -f docker-compose.selfhost.yml logs api | tail -50
```
Then either fix the cause and let it retry, or roll back: restore your pre-upgrade dump, pin the
previous `IMAGE_TAG`, and `up -d`. Do not delete the `siftlode_pgdata` volume to "start clean" unless
you have a dump you have actually tested restoring — that volume *is* your instance.
frontend/package.json
+1
View File
@@ -1,6 +1,7 @@
{
"name": "siftlode-frontend",
"private": true,
"license": "AGPL-3.0-only",
"type": "module",
"scripts": {
"dev": "vite",
frontend/src/components/AccountPanel.tsx
+252
View File
@@ -0,0 +1,252 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Check, Clock, Copy, Globe, KeyRound, Mail, Send, ShieldAlert } from "lucide-react";
import Modal from "./Modal";
import { LoadingState, StateMessage } from "./QueryState";
import { api } from "../lib/api";
import { qk } from "../lib/queryKeys";
import { notify } from "../lib/notifications";
import { copyText } from "../lib/clipboard";
import { invalidateUserAccess, isEmailOff } from "../lib/adminUsers";
import type { AdminUserAccess, AdminUserRow } from "../lib/api/admin";
// The admin's answer to "why can this person not get in?", and the place every recovery action for
// one account lives (E12 · S5, design option C).
//
// It exists because nothing in the admin UI showed an account's GATE STATE: an admin could see a
// "Pending approval" chip but not whether the address was confirmed, whether the password had been
// cleared by SEC-10, or which Google account was attached. Every support conversation started by
// guessing. The recovery actions sit under that readout rather than in the row, so the admin picks
// one having read what is actually wrong.
/** One line of the Access readout: an icon, a label, and the state in the tone that state deserves. */
function AccessRow({
icon: Icon,
label,
value,
tone = "muted",
}: {
icon: typeof Mail;
label: string;
value: string;
tone?: "muted" | "warn" | "ok";
}) {
const toneCls = tone === "warn" ? "text-warn" : tone === "ok" ? "text-success" : "text-muted";
return (
<div className="flex items-center gap-2 text-xs">
<Icon className={`w-3.5 h-3.5 shrink-0 ${toneCls}`} />
<span className="text-muted w-28 shrink-0">{label}</span>
<span className={`${toneCls} min-w-0 truncate`}>{value}</span>
</div>
);
}
export default function AccountPanel({
user,
onClose,
}: {
user: AdminUserRow;
onClose: () => void;
}) {
const { t } = useTranslation();
const qc = useQueryClient();
const [link, setLink] = useState<{ url: string; expires_at: string | null } | null>(null);
const [copied, setCopied] = useState(false);
const q = useQuery({
queryKey: qk.adminUserAccess(user.id),
queryFn: () => api.adminUserAccess(user.id),
});
// Every action here goes through a `quiet: true` endpoint, which suppresses the GLOBAL error
// dialog on the promise that the caller reports the failure inline. Without an `onError` that
// promise is broken and a refusal renders as nothing at all — on the broken-SMTP instance this
// panel exists for, the admin would read silence as "the mail went out".
const issue = useMutation({
mutationFn: () => api.createSetPasswordLink(user.id),
onSuccess: (r) => setLink({ url: r.url, expires_at: r.expires_at }),
onError: () => notify({ level: "error", message: t("users.account.linkFailed") }),
});
const resend = useMutation({
mutationFn: () => api.resendUserVerification(user.id),
onSuccess: () => notify({ level: "success", message: t("users.account.resendSent") }),
onError: (err) =>
notify({
level: "error",
message: isEmailOff(err)
? t("users.pendingAccess.resendEmailOff")
: t("users.pendingAccess.resendFailed"),
}),
});
const activate = useMutation({
mutationFn: () => api.activateUser(user.id),
onSuccess: () => invalidateUserAccess(qc, user.id),
onError: () => notify({ level: "error", message: t("users.pendingAccess.activateFailed") }),
});
const copy = async () => {
if (!link) return;
// Via the shared helper, whose synchronous `execCommand` path is what makes copying work on a
// plain-http LAN origin — where `navigator.clipboard` is undefined, and which is exactly the
// minimal self-host this no-email link was built for.
if (await copyText(link.url)) {
setCopied(true);
window.setTimeout(() => setCopied(false), 2000);
return;
}
// The link is on screen and selectable, so a failed copy is a convenience that failed — not the
// action failing. Say so rather than leaving the button looking inert.
notify({ level: "warning", message: t("users.account.copyFailed") });
};
const a: AdminUserAccess | undefined = q.data;
return (
<Modal
title={user.display_name ?? user.email.split("@")[0]}
onClose={onClose}
placement="drawer"
maxWidth="max-w-md"
>
<div className="text-2xs text-muted truncate -mt-2 mb-4">{user.email}</div>
{q.isError && !a ? (
<StateMessage tone="error" onRetry={() => q.refetch()}>
{t("common.loadError")}
</StateMessage>
) : !a ? (
<LoadingState />
) : (
<>
<div className="text-2xs uppercase tracking-wider text-muted mb-2">
{t("users.account.access")}
</div>
<div className="glass-card rounded-xl p-3 space-y-2 mb-5">
<AccessRow
icon={Mail}
label={t("users.account.email")}
value={
a.email_verified
? t("users.account.emailConfirmed")
: t("users.account.emailUnconfirmed")
}
tone={a.email_verified ? "ok" : "warn"}
/>
<AccessRow
icon={Clock}
label={t("users.account.approval")}
value={
a.is_active
? t("users.account.approvalDone")
: a.can_activate
? t("users.account.approvalWaitingOnYou")
: t("users.account.approvalNotRequested")
}
tone={a.is_active ? "ok" : "warn"}
/>
<AccessRow
icon={KeyRound}
label={t("users.account.password")}
value={
a.has_password
? t("users.account.passwordSet")
: a.password_revoked_at
? t("users.account.passwordRevoked")
: t("users.account.passwordNone")
}
// Not having a password is only a PROBLEM when it was taken away; a Google-only
// account never had one and is working exactly as intended.
tone={a.has_password ? "ok" : a.password_revoked_at ? "warn" : "muted"}
/>
<AccessRow
icon={Globe}
label={t("users.account.google")}
value={
!a.has_google
? t("users.account.googleNone")
: a.google_email
? t("users.account.googleAs", { email: a.google_email })
: t("users.account.googleConnected")
}
tone={a.has_google ? "ok" : "muted"}
/>
{/* The gate that explains a Google sign-in silently becoming a fresh access request. */}
{!a.is_allowed && (
<AccessRow
icon={ShieldAlert}
label={t("users.account.invite")}
value={t("users.account.inviteMissing")}
tone="warn"
/>
)}
</div>
<div className="text-2xs uppercase tracking-wider text-muted mb-2">
{t("users.account.recovery")}
</div>
<div className="space-y-2">
{a.can_activate && (
<button
onClick={() => activate.mutate()}
disabled={activate.isPending}
className="w-full glass-card glass-hover flex items-center gap-2 px-3 py-2.5 rounded-lg text-xs border border-accent/40 text-accent disabled:opacity-50 transition"
>
<Check className="w-4 h-4 shrink-0" />
<span className="flex-1 text-left">{t("users.account.activate")}</span>
</button>
)}
<button
onClick={() => resend.mutate()}
disabled={!a.email_configured || a.email_verified || resend.isPending}
className="w-full glass-card glass-hover flex items-center gap-2 px-3 py-2.5 rounded-lg text-xs disabled:opacity-45 transition"
>
<Send className="w-4 h-4 shrink-0" />
<span className="flex-1 text-left">{t("users.account.resend")}</span>
{/* Says WHY it is unavailable instead of just greying out — the no-mail case is the
whole reason the link below exists, so naming it here is the explanation. */}
{!a.email_configured && (
<span className="text-muted">{t("users.account.mailOff")}</span>
)}
{a.email_configured && a.email_verified && (
<span className="text-muted">{t("users.account.alreadyConfirmed")}</span>
)}
</button>
<button
onClick={() => issue.mutate()}
disabled={issue.isPending || user.is_demo}
className="w-full glass-card glass-hover flex items-center gap-2 px-3 py-2.5 rounded-lg text-xs border border-accent/40 text-accent disabled:opacity-50 transition"
>
<KeyRound className="w-4 h-4 shrink-0" />
<span className="flex-1 text-left">
{link ? t("users.account.setPasswordAgain") : t("users.account.setPassword")}
</span>
</button>
</div>
{link && (
<div className="mt-4 pt-4 border-t border-border space-y-2">
<div className="flex gap-2">
<div className="flex-1 min-w-0 bg-surface border border-border rounded-lg px-3 py-2 text-2xs font-mono truncate">
{link.url}
</div>
<button
onClick={copy}
className="shrink-0 glass-card glass-hover inline-flex items-center gap-1.5 px-3 py-2 rounded-lg text-xs border border-accent/40 text-accent transition"
>
{copied ? <Check className="w-3.5 h-3.5" /> : <Copy className="w-3.5 h-3.5" />}
{copied ? t("users.account.copied") : t("users.account.copy")}
</button>
</div>
<p className="text-2xs text-muted leading-relaxed">{t("users.account.linkNote")}</p>
</div>
)}
</>
)}
</Modal>
);
}
frontend/src/components/AdminUsers.tsx
+16 -16
View File
@@ -24,6 +24,7 @@ import {
type PendingAccessRow,
} from "../lib/api";
import { notify } from "../lib/notifications";
import { invalidateUserAccess, isEmailOff } from "../lib/adminUsers";
import { ADMIN_USERS_TAB_KEY } from "../lib/adminUsersTab";
import Tooltip from "./Tooltip";
import InlineNotice from "./InlineNotice";
@@ -31,6 +32,7 @@ import { Section } from "./ui/form";
import SegmentedToggle from "./SegmentedToggle";
import { LANGUAGES, type LangCode } from "../i18n";
import { LoadingState, StateMessage } from "./QueryState";
import AccountPanel from "./AccountPanel";
import { useConfirm } from "./ConfirmProvider";
import Tabs, { usePersistedTab } from "./Tabs";
import { PageToolbar } from "./PageShell";
@@ -107,6 +109,9 @@ function UsersRoles({ me }: { me: Me }) {
// Which rows have an action in flight, so we disable ONLY those rows' buttons (SC1) — a Set, since
// with per-row disabling the admin can now start actions on several rows concurrently.
const [pendingIds, setPendingIds] = useState<Set<number>>(new Set());
// The account panel (E12 S5): which user it is showing, or null. Held here rather than per row so
// only one is ever mounted.
const [panelFor, setPanelFor] = useState<AdminUserRow | null>(null);
const startPending = (id: number) => setPendingIds((s) => new Set(s).add(id));
const endPending = (id: number) =>
setPendingIds((s) => {
@@ -192,6 +197,7 @@ function UsersRoles({ me }: { me: Me }) {
return (
<Section card title={t("users.roles.title")}>
{panelFor && <AccountPanel user={panelFor} onClose={() => setPanelFor(null)} />}
<p className="text-xs text-muted leading-relaxed mb-3">{t("users.roles.intro")}</p>
{q.isError && !q.data ? (
<StateMessage tone="error" onRetry={() => q.refetch()}>
@@ -207,13 +213,17 @@ function UsersRoles({ me }: { me: Me }) {
const self = u.id === me.id;
return (
<div key={u.id} className="glass-card flex items-center gap-2.5 p-2.5 rounded-xl">
<div className="min-w-0 flex-1">
<button
onClick={() => setPanelFor(u)}
title={t("users.account.open")}
className="selectable cursor-pointer min-w-0 flex-1 text-left rounded-lg -m-1 p-1"
>
<div className="text-sm truncate">
{u.display_name ?? u.email.split("@")[0]}
{self && <span className="text-muted"> · {t("users.roles.you")}</span>}
</div>
<div className="text-2xs text-muted truncate">{u.email}</div>
</div>
</button>
{u.role === "admin" && <Badge tone="accent">{t("users.roles.admin")}</Badge>}
{u.is_demo && <Badge tone="muted">{t("users.roles.demo")}</Badge>}
{u.is_suspended && <Badge tone="warning">{t("users.roles.suspended")}</Badge>}
@@ -316,12 +326,9 @@ function PendingAccess() {
queryKey: qk.adminPendingAccess(),
queryFn: () => api.adminPendingAccess(),
});
// Activating changes /api/me's badge count too, so refresh both or the number lies until reload.
const refresh = () => {
qc.invalidateQueries({ queryKey: qk.adminPendingAccess() });
qc.invalidateQueries({ queryKey: qk.me() });
qc.invalidateQueries({ queryKey: qk.adminUsers() });
};
// Activating changes /api/me's badge count too, so refresh all of them or the number lies until
// reload. Shared with the account panel's copy of this mutation (see lib/adminUsers).
const refresh = () => invalidateUserAccess(qc);
const activate = useMutation({
mutationFn: (u: PendingAccessRow) => api.activateUser(u.id),
@@ -343,8 +350,7 @@ function PendingAccess() {
onError: (err) =>
notify({
level: "error",
message:
err instanceof HttpError && isEmailOff(err)
message: isEmailOff(err)
? t("users.pendingAccess.resendEmailOff")
: t("users.pendingAccess.resendFailed"),
}),
@@ -380,12 +386,6 @@ function PendingAccess() {
);
}
/** Did the resend fail because this instance has no working SMTP? */
function isEmailOff(err: HttpError): boolean {
const d = err.detailData as { code?: string; reason?: string } | null | undefined;
return d?.code === "resend" && d.reason === "email_off";
}
// One stuck sign-up. The row names WHICH gate is shut rather than just offering buttons — the same
// reasoning as the sign-in card's blocked panel, from the admin's side: an admin about to let someone
// in should see that the address has not been confirmed yet, because on a no-SMTP instance that is
frontend/src/components/Modal.tsx
+15 -2
View File
@@ -25,11 +25,17 @@ export default function Modal({
onClose,
children,
maxWidth = "max-w-lg",
placement = "center",
}: {
title: ReactNode;
onClose: () => void;
children: ReactNode;
maxWidth?: string;
/** `drawer` slides in from the right edge and runs full height; everything else about the shell —
* the layer stack, ESC, the focus trap, Back-to-close, the scroll lock — is identical. A drawer
* is a placement, not a second dialog implementation: duplicating that behaviour is exactly how
* two shells drift apart. */
placement?: "center" | "drawer";
}) {
const { t } = useTranslation();
// Browser/mouse Back closes the topmost modal instead of navigating away.
@@ -106,9 +112,12 @@ export default function Modal({
};
}, [restoreTo]);
const drawer = placement === "drawer";
return createPortal(
<div
className="fixed inset-0 z-overlay flex items-center justify-center p-4 sm:p-6 bg-black/70 overlay-backdrop"
className={`fixed inset-0 z-overlay flex bg-black/70 overlay-backdrop ${
drawer ? "justify-end" : "items-center justify-center p-4 sm:p-6"
}`}
onMouseDown={(e) => {
pressedOnBackdrop.current = e.target === e.currentTarget;
}}
@@ -122,7 +131,11 @@ export default function Modal({
<div
ref={panelRef}
tabIndex={-1}
className={`glass relative w-full ${maxWidth} max-h-full overflow-y-auto rounded-2xl outline-none`}
className={
drawer
? `glass relative w-full ${maxWidth} h-full overflow-y-auto rounded-none sm:rounded-l-2xl outline-none overlay-drawer`
: `glass relative w-full ${maxWidth} max-h-full overflow-y-auto rounded-2xl outline-none`
}
>
<div className="flex items-start justify-between gap-3 p-5 pb-3">
<h2 id={titleId} className="text-lg font-semibold leading-snug">
frontend/src/components/Stats.tsx
+10 -1
View File
@@ -6,6 +6,8 @@ import { History, Pause, Play, RefreshCw } from "lucide-react";
import { api, type AdminQuotaRow, type Me } from "../lib/api";
import { formatEta, quotaActionLabel } from "../lib/format";
import { notify } from "../lib/notifications";
import { notifyYouTubeActionError } from "../lib/youtubeErrors";
import { useWizard } from "./WizardProvider";
import { LS, useAccountPersistedState } from "../lib/storage";
import Tooltip from "./Tooltip";
import { LoadingState, StateMessage } from "./QueryState";
@@ -51,6 +53,8 @@ export default function Stats() {
function Overview({ me }: { me: Me }) {
const { t } = useTranslation();
const qc = useQueryClient();
// The 403 case needs somewhere to go, or the toast names a fix the user cannot reach from here.
const { openWizard } = useWizard();
const status = useQuery({ queryKey: qk.myStatus(), queryFn: api.myStatus });
const usage = useQuery({ queryKey: qk.myUsage(), queryFn: api.myUsage });
const s = status.data;
@@ -64,7 +68,12 @@ function Overview({ me }: { me: Me }) {
message: t("stats.sync.synced", { count: r.subscriptions ?? 0 }),
});
},
onError: () => notify({ level: "error", message: t("stats.sync.syncFailed") }),
// The same helper and the same wizard handle as the Channels page: it translates the 403 into
// "connect YouTube" WITH a button that opens the wizard — instead of surfacing the route's bare
// English detail under a Hungarian UI — and delegates everything else to notifyActionError so
// the server's own reason wins, which is what makes the SEC-14 cooldown read as a pace limit
// rather than "Sync failed".
onError: (e) => notifyYouTubeActionError(e, t("stats.sync.syncFailed"), openWizard),
});
const deepAll = useMutation({
mutationFn: () => api.deepAll(true),
frontend/src/components/VerifyBanner.tsx
+25 -1
View File
@@ -14,12 +14,36 @@ import type { VerifyEntry } from "../lib/useVerifyEntry";
// wrong-account case is why the address is named.
export default function VerifyBanner({ entry }: { entry: VerifyEntry }) {
const { t } = useTranslation();
// A reset link opened here can only be ACTED on signed out, so say that rather than nothing —
// silence is precisely what made the verify half a bug. The token stays in the address bar, so
// signing out from this same tab picks it up.
//
// Tested on `resetPending` ALONE, not `!outcome && resetPending`. The two can both be set (a
// reset link pasted over an unread confirmation, or a slow verify resolving after one), and this
// one goes first because it is the banner with something to DO. `dismiss` then peels it and the
// outcome shows underneath — where `!outcome &&` would have hidden this one until the other was
// dismissed, and dismissing that used to clear both.
if (entry.resetPending) {
return (
<Banner
tone="warning"
icon={AlertTriangle}
role="status"
onDismiss={entry.dismiss}
dismissTitle={t("header.banner.dismiss")}
>
{t("header.verify.resetSignedOut")}
</Banner>
);
}
if (!entry.outcome) return null;
const ok = entry.outcome !== "invalid";
const ok = entry.outcome === "ok" || entry.outcome === "ok_reset";
const text =
entry.outcome === "invalid"
? t("header.verify.invalid")
: entry.outcome === "unreachable"
? t("header.verify.unreachable")
: entry.email
? t("header.verify.confirmedNamed", { email: entry.email })
: t("header.verify.confirmed");
frontend/src/components/Welcome.tsx
+6
View File
@@ -555,6 +555,12 @@ function AuthCard({ verify }: { verify: VerifyEntry }) {
</button>
</Banner>
)}
{/* No resend CTA here, unlike the invalid banner below: the token was NOT spent — we simply
could not reach the server — so pushing the user to request a fresh link would act on a
statement we know to be false (DX-16). Retrying the same link is the correct next move. */}
{verifyOutcome === "unreachable" && (
<Banner tone="warn">{t("welcome.auth.verifyUnreachable")}</Banner>
)}
{verifyOutcome === "invalid" && (
<Banner tone="warn">
{t("welcome.auth.verifyInvalid")}{" "}
frontend/src/components/accountPanel.dom.test.tsx
+140
View File
@@ -0,0 +1,140 @@
// @vitest-environment jsdom
//
// The admin account panel's RECOVERY actions, and specifically that a refusal is visible.
//
// All three call `quiet: true` endpoints, which suppress the global error dialog on the promise that
// the caller reports the failure inline. Without an `onError` that promise is broken and a refusal
// renders as nothing at all — and on the broken-SMTP instance this panel exists for, an admin reads
// silence as "the mail went out". These pin that each failure says something, and that the resend
// distinguishes "this instance has no working mail" from a genuine send failure.
//
// i18n is a key-passthrough so assertions read stable keys; the API and the notify singleton are
// stubbed so the panel can be driven without a server.
import { describe, it, expect, afterEach, beforeEach, vi } from "vitest";
import { render, screen, cleanup, waitFor, fireEvent } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
return {
...actual,
useTranslation: () => ({ t: (key: string) => key, i18n: { language: "en" } }),
};
});
const notifyMock = vi.fn();
vi.mock("../lib/notifications", async (importOriginal) => {
const actual = await importOriginal<typeof import("../lib/notifications")>();
return { ...actual, notify: (...a: unknown[]) => notifyMock(...a) };
});
const access = {
email_verified: false,
email_configured: true,
is_active: false,
is_allowed: true,
can_activate: true,
has_password: false,
google_email: null,
invite_status: null,
};
const apiMock = {
adminUserAccess: vi.fn(async () => access),
createSetPasswordLink: vi.fn(),
resendUserVerification: vi.fn(),
activateUser: vi.fn(),
};
vi.mock("../lib/api", async (importOriginal) => {
const actual = await importOriginal<typeof import("../lib/api")>();
return { ...actual, api: apiMock };
});
const { HttpError } = await import("../lib/api");
const AccountPanel = (await import("./AccountPanel")).default;
const user = {
id: 7,
email: "stuck@example.com",
display_name: null,
role: "user",
is_demo: false,
is_active: false,
is_suspended: false,
};
function mount() {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
render(
<QueryClientProvider client={qc}>
{/* eslint-disable-next-line @typescript-eslint/no-explicit-any */}
<AccountPanel user={user as any} onClose={() => {}} />
</QueryClientProvider>,
);
}
/** Click the recovery button whose label is `key`, once the readout has loaded. */
async function clickAction(key: string) {
mount();
await screen.findByText("users.account.recovery");
fireEvent.click(screen.getByText(key));
}
beforeEach(() => {
notifyMock.mockClear();
Object.values(apiMock).forEach((m) => m.mockClear());
apiMock.adminUserAccess.mockResolvedValue(access);
});
afterEach(cleanup);
const errorMessages = () =>
notifyMock.mock.calls
.map(([a]) => a as { level?: string; message?: string })
.filter((a) => a.level === "error")
.map((a) => a.message);
describe("the account panel reports a refused recovery action", () => {
it("says the link could not be minted instead of leaving the button inert", async () => {
apiMock.createSetPasswordLink.mockRejectedValue(new HttpError(404, "gone", null));
await clickAction("users.account.setPassword");
await waitFor(() => expect(errorMessages()).toContain("users.account.linkFailed"));
// …and no link block was rendered, so the toast is the only thing that happened.
expect(screen.queryByText("users.account.copy")).toBeNull();
});
it("names the no-mail case, so a no-SMTP instance does not read as a bug", async () => {
apiMock.resendUserVerification.mockRejectedValue(
new HttpError(409, "email off", { code: "resend", reason: "email_off" }),
);
await clickAction("users.account.resend");
await waitFor(() => expect(errorMessages()).toContain("users.pendingAccess.resendEmailOff"));
});
it("reports a genuine send failure as a failure, not as a sent mail", async () => {
apiMock.resendUserVerification.mockRejectedValue(
new HttpError(422, "send failed", { code: "resend", reason: "send_failed" }),
);
await clickAction("users.account.resend");
await waitFor(() => expect(errorMessages()).toContain("users.pendingAccess.resendFailed"));
// The success toast must NOT also fire — that is the exact lie the backend guards against.
expect(notifyMock.mock.calls.map(([a]) => (a as { message?: string }).message)).not.toContain(
"users.account.resendSent",
);
});
it("reports a refused activation", async () => {
apiMock.activateUser.mockRejectedValue(
new HttpError(409, "not allowed", { code: "activate", reason: "not_allowed" }),
);
await clickAction("users.account.activate");
await waitFor(() => expect(errorMessages()).toContain("users.pendingAccess.activateFailed"));
});
});
frontend/src/components/authCardStates.dom.test.tsx
+6 -1
View File
@@ -11,7 +11,12 @@ import { describe, it, expect, afterEach, vi } from "vitest";
import { render, screen, fireEvent, cleanup, act } from "@testing-library/react";
import type { VerifyEntry } from "../lib/useVerifyEntry";
const noVerify: VerifyEntry = { outcome: null, email: null, dismiss: () => {} };
const noVerify: VerifyEntry = {
outcome: null,
email: null,
resetPending: false,
dismiss: () => {},
};
afterEach(() => {
cleanup();
frontend/src/components/welcomeMode.render.dom.test.tsx
+16 -1
View File
@@ -32,7 +32,12 @@ import type { VerifyEntry } from "../lib/useVerifyEntry";
// App confirms the emailed token and hands the outcome down; these tests are about the card's
// sub-views, so there is nothing to report.
const noVerify: VerifyEntry = { outcome: null, email: null, dismiss: () => {} };
const noVerify: VerifyEntry = {
outcome: null,
email: null,
resetPending: false,
dismiss: () => {},
};
const back = () =>
act(() => {
@@ -91,6 +96,7 @@ describe("the verification outcome handed down from App", () => {
const withOutcome = (outcome: VerifyEntry["outcome"]): VerifyEntry => ({
outcome,
email: "someone@example.com",
resetPending: false,
dismiss: () => {},
});
@@ -112,6 +118,15 @@ describe("the verification outcome handed down from App", () => {
expect(screen.getByText("welcome.auth.resendCta")).toBeTruthy();
});
// DX-16: a transport failure is not a verdict on the token, so it must NOT offer a fresh link —
// that would act on a statement we know to be false. Retrying the same link is the way out.
it("does not offer a new link when the server was merely unreachable", async () => {
render(<Welcome verify={withOutcome("unreachable")} />);
expect(await screen.findByText("welcome.auth.verifyUnreachable")).toBeTruthy();
expect(screen.queryByText("welcome.auth.resendCta")).toBeNull();
expect(screen.queryByText("welcome.auth.verifyInvalid")).toBeNull();
});
it("shows no banner when there was no token to confirm", async () => {
render(<Welcome verify={noVerify} />);
await screen.findByText("welcome.auth.signinTitle");
frontend/src/i18n/locales/en/auditActions.json
+1
View File
@@ -5,6 +5,7 @@
"user_delete": "User deleted",
"user_activate": "Account activated",
"user_verify_resend": "Confirmation link re-sent",
"user_setpw_link": "Set-password link issued",
"invite_approve": "Access request approved",
"invite_deny": "Access request denied",
"invite_add": "Email whitelisted",
frontend/src/i18n/locales/en/downloads.json
+12
View File
@@ -229,6 +229,14 @@
"tooLarge": "That file is too large (max {{size}}).",
"empty": "That file is empty."
},
"urlBlocked": {
"scheme": "Only http:// and https:// links can be downloaded.",
"no_host": "That link doesn't contain a website address.",
"unresolvable": "That address couldn't be looked up. Check the link and try again.",
"private": "That address is on the server's own network, so it can't be downloaded from.",
"generic": "That address can't be downloaded from.",
"malformed": "That link isn't a valid web address."
},
"fail": {
"title": "Download failed",
"detailsLink": "details",
@@ -273,6 +281,10 @@
"unknown": {
"short": "Download failed",
"body": "Something went wrong while downloading this item. The technical details below may help."
},
"quota_exceeded": {
"short": "Storage full",
"body": "This download was stopped because your storage quota is already used up. Delete something from your library, then try again."
}
}
},
frontend/src/i18n/locales/en/errors.json
+1
View File
@@ -3,6 +3,7 @@
"generic": "The server couldn't carry out that action. Please try again.",
"validation": "Some of the values you entered weren't valid. Please check them and try again.",
"server": "Server error",
"syncCooldown": "You've synced a few times just now — give it a few minutes. The YouTube budget is shared with everyone on this instance, and background syncing keeps running either way.",
"ok": "OK",
"offline": {
"title": "Connection lost",
frontend/src/i18n/locales/en/header.json
+2
View File
@@ -61,6 +61,8 @@
"confirmed": "Email address confirmed.",
"confirmedNamed": "Email address confirmed — {{email}}.",
"invalid": "That verification link is no longer valid — it has expired or was already used.",
"unreachable": "Couldn't reach the server to confirm that link, so it hasn't been used up. Try opening it again in a moment.",
"resetSignedOut": "Password-reset links open in a signed-out window. Sign out here, or open the link in a private window, and it will pick up where it left off.",
"reset": "The password this account registered with was cleared for security; set a new one before signing in."
}
}
frontend/src/i18n/locales/en/users.json
+34
View File
@@ -59,5 +59,39 @@
"title": "Delete account?",
"confirm": "Permanently delete {{email}} and all their data — subscriptions, tags, watch history, playlists and settings? This can't be undone.",
"done": "{{email}} deleted"
},
"account": {
"access": "Access",
"activate": "Let this account in",
"alreadyConfirmed": "already confirmed",
"approval": "Admin approval",
"approvalDone": "let in",
"approvalNotRequested": "no approved invite yet",
"approvalWaitingOnYou": "waiting on you",
"copied": "Copied",
"copy": "Copy",
"copyFailed": "Couldn't copy — select the link and copy it by hand.",
"email": "Email",
"emailConfirmed": "confirmed",
"emailUnconfirmed": "not confirmed",
"google": "Google",
"googleAs": "connected as {{email}}",
"googleConnected": "connected",
"googleNone": "not connected",
"invite": "Invite",
"inviteMissing": "this address is not on the allow-list",
"linkNote": "Valid 24 hours, works once. Opening it lets this person set a password and signs out anyone already using the account. It does not confirm their address.",
"mailOff": "no mail configured",
"open": "Open the account panel",
"password": "Password",
"passwordNone": "none — signs in with Google",
"passwordRevoked": "cleared for security",
"passwordSet": "set",
"recovery": "Recovery",
"resend": "Send a confirmation link",
"resendSent": "Confirmation link sent.",
"setPassword": "Create a set-password link",
"setPasswordAgain": "Create another link",
"linkFailed": "Couldn't create the link. Please try again."
}
}
frontend/src/i18n/locales/en/welcome.json
+1
View File
@@ -95,6 +95,7 @@
"verifyOkReset": "Email confirmed. The password this account registered with had been cleared for security — choose a new one to sign in.",
"verifyOkResetCta": "Set a new password",
"verifyInvalid": "That verification link is invalid or has expired.",
"verifyUnreachable": "Couldn't reach the server to confirm that link, so it hasn't been used up. Try opening it again in a moment.",
"resendCta": "Send a new link",
"resendDone": "If that email has an unconfirmed account, we've sent a new confirmation link. Check your inbox, and your spam folder if it hasn't arrived in a minute.",
"oauthCancelled": "Google sign-in was cancelled. You can try again or use email and password.",
frontend/src/i18n/locales/hu/auditActions.json
+1
View File
@@ -5,6 +5,7 @@
"user_delete": "Felhasználó törölve",
"user_activate": "Fiók aktiválva",
"user_verify_resend": "Megerősítő link újraküldve",
"user_setpw_link": "Jelszóbeállító link kiadva",
"invite_approve": "Hozzáférési kérelem jóváhagyva",
"invite_deny": "Hozzáférési kérelem elutasítva",
"invite_add": "Email fehérlistázva",
frontend/src/i18n/locales/hu/downloads.json
+12
View File
@@ -229,6 +229,14 @@
"tooLarge": "Ez a fájl túl nagy (legfeljebb {{size}}).",
"empty": "Ez a fájl üres."
},
"urlBlocked": {
"scheme": "Csak http:// és https:// hivatkozás tölthető le.",
"no_host": "Ez a hivatkozás nem tartalmaz webcímet.",
"unresolvable": "Ezt a címet nem sikerült feloldani. Ellenőrizd a hivatkozást, és próbáld újra.",
"private": "Ez a cím a szerver saját hálózatán van, ezért nem tölthető le róla.",
"generic": "Erről a címről nem lehet letölteni.",
"malformed": "Ez a hivatkozás nem érvényes webcím."
},
"fail": {
"title": "A letöltés nem sikerült",
"detailsLink": "részletek",
@@ -273,6 +281,10 @@
"unknown": {
"short": "A letöltés nem sikerült",
"body": "Valami hiba történt az elem letöltése közben. Az alábbi technikai részletek segíthetnek."
},
"quota_exceeded": {
"short": "Betelt a tárhely",
"body": "Ez a letöltés leállt, mert a tárhelykereted már elfogyott. Törölj valamit a könyvtáradból, aztán próbáld újra."
}
}
},
frontend/src/i18n/locales/hu/errors.json
+1
View File
@@ -3,6 +3,7 @@
"generic": "A szerver nem tudta végrehajtani a műveletet. Próbáld újra.",
"validation": "Néhány megadott érték érvénytelen volt. Ellenőrizd, majd próbáld újra.",
"server": "Szerverhiba",
"syncCooldown": "Az imént többször is szinkronizáltál — várj pár percet. A YouTube-keret közös az instance összes felhasználójával, a háttérszinkron pedig így is fut tovább.",
"ok": "OK",
"offline": {
"title": "Megszakadt a kapcsolat",
frontend/src/i18n/locales/hu/header.json
+2
View File
@@ -61,6 +61,8 @@
"confirmed": "Megerősítettük az e-mail-címedet.",
"confirmedNamed": "Megerősítettük az e-mail-címedet — {{email}}.",
"invalid": "Ez a megerősítő link már nem érvényes — lejárt, vagy már felhasználták.",
"unreachable": "Nem sikerült elérni a szervert a link megerősítéséhez, így az nem is használódott el. Próbáld megnyitni újra egy pillanat múlva.",
"resetSignedOut": "A jelszó-visszaállító linkek kijelentkezett ablakban nyílnak meg. Jelentkezz ki itt, vagy nyisd meg a linket privát ablakban, és onnan folytatódik.",
"reset": "A fiók regisztrációkori jelszavát biztonsági okból töröltük; belépés előtt állíts be újat."
}
}
frontend/src/i18n/locales/hu/users.json
+34
View File
@@ -59,5 +59,39 @@
"title": "Törlöd a fiókot?",
"confirm": "Véglegesen törlöd: {{email}} és minden adatát — feliratkozások, címkék, megtekintési előzmény, lejátszási listák és beállítások? Ez nem visszavonható.",
"done": "{{email}} törölve"
},
"account": {
"access": "Hozzáférés",
"activate": "Fiók beengedése",
"alreadyConfirmed": "már megerősítve",
"approval": "Admin jóváhagyás",
"approvalDone": "beengedve",
"approvalNotRequested": "még nincs jóváhagyott meghívó",
"approvalWaitingOnYou": "rád vár",
"copied": "Másolva",
"copy": "Másolás",
"copyFailed": "Nem sikerült másolni — jelöld ki a linket, és másold kézzel.",
"email": "E-mail",
"emailConfirmed": "megerősítve",
"emailUnconfirmed": "nincs megerősítve",
"google": "Google",
"googleAs": "összekötve mint {{email}}",
"googleConnected": "összekötve",
"googleNone": "nincs összekötve",
"invite": "Meghívó",
"inviteMissing": "ez a cím nincs az engedélyezési listán",
"linkNote": "24 óráig érvényes, egyszer használható. Aki megnyitja, beállíthat egy jelszót, és ezzel kilép mindenki, aki eddig használta a fiókot. A címet nem erősíti meg.",
"mailOff": "nincs beállítva levelezés",
"open": "Fiókpanel megnyitása",
"password": "Jelszó",
"passwordNone": "nincs — Google-lel lép be",
"passwordRevoked": "biztonsági okból törölve",
"passwordSet": "be van állítva",
"recovery": "Helyreállítás",
"resend": "Megerősítő link küldése",
"resendSent": "Megerősítő link elküldve.",
"setPassword": "Jelszóbeállító link készítése",
"setPasswordAgain": "Új link készítése",
"linkFailed": "Nem sikerült létrehozni a linket. Próbáld újra."
}
}
frontend/src/i18n/locales/hu/welcome.json
+1
View File
@@ -95,6 +95,7 @@
"verifyOkReset": "E-mail megerősítve. A fiók regisztrációkori jelszavát biztonsági okból korábban töröltük — válassz újat a belépéshez.",
"verifyOkResetCta": "Új jelszó beállítása",
"verifyInvalid": "Ez a megerősítő link érvénytelen vagy lejárt.",
"verifyUnreachable": "Nem sikerült elérni a szervert a link megerősítéséhez, így az nem is használódott el. Próbáld megnyitni újra egy pillanat múlva.",
"resendCta": "Új link küldése",
"resendDone": "Ha ehhez az e-mailhez tartozik megerősítetlen fiók, elküldtünk egy új megerősítő linket. Nézd meg a postafiókod, és a Spam mappát is, ha egy percen belül nem érkezik meg.",
"oauthCancelled": "A Google-belépés megszakadt. Próbáld újra, vagy lépj be e-maillel és jelszóval.",
frontend/src/index.css
+18
View File
@@ -657,6 +657,24 @@ html[data-perf="1"] .overlay-backdrop {
backdrop-filter: none;
}
/* Right-edge drawer (Modal placement="drawer"). The slide is what makes it read as a panel that
came from the side rather than a dialog that happens to be right-aligned — so it is part of the
affordance, not decoration. Honours prefers-reduced-motion, like every other motion here. */
.overlay-drawer {
animation: drawer-in 180ms cubic-bezier(0.22, 0.61, 0.36, 1) both;
}
@keyframes drawer-in {
from {
transform: translateX(1.5rem);
opacity: 0;
}
}
@media (prefers-reduced-motion: reduce) {
.overlay-drawer {
animation: none;
}
}
/* Skip layout/paint for off-screen feed cards so long lists stay smooth while
scrolling. contain-intrinsic-size reserves an approximate box so the scrollbar
stays stable; `auto` lets the browser remember each card's real size after it
frontend/src/lib/adminUsers.test.ts
+55
View File
@@ -0,0 +1,55 @@
// @vitest-environment jsdom
//
// jsdom because the shared module reaches lib/api, which pulls in i18n's localStorage probe.
import { describe, it, expect, vi } from "vitest";
import { QueryClient } from "@tanstack/react-query";
import { invalidateUserAccess, isEmailOff } from "./adminUsers";
import { HttpError } from "./api";
import { qk } from "./queryKeys";
// The set matters, not the call: letting someone in changes three things a user can see at once —
// the row, the pending queue, and the /api/me badge — and the account panel shipped invalidating
// only the first two, so the badge went on claiming the person was stuck until a reload. Pinning the
// keys here is what stops the two call sites drifting apart again, since neither component test
// asserts on invalidation.
describe("invalidateUserAccess", () => {
it("refreshes the row, the pending queue and the badge", () => {
const qc = new QueryClient();
const spy = vi.spyOn(qc, "invalidateQueries").mockImplementation(() => Promise.resolve());
invalidateUserAccess(qc);
const keys = spy.mock.calls.map(([a]) => JSON.stringify(a?.queryKey));
expect(keys).toEqual(
[qk.adminUsers(), qk.adminPendingAccess(), qk.me()].map((k) => JSON.stringify(k)),
);
});
it("also refreshes one account's gate readout when given an id", () => {
const qc = new QueryClient();
const spy = vi.spyOn(qc, "invalidateQueries").mockImplementation(() => Promise.resolve());
invalidateUserAccess(qc, 7);
expect(spy.mock.calls.map(([a]) => JSON.stringify(a?.queryKey))).toContain(
JSON.stringify(qk.adminUserAccess(7)),
);
});
});
// The admin-side resend deliberately reports failures (the public one is silent), so it has to be
// able to say WHICH — a no-SMTP instance is a configuration answer, not a bug.
describe("isEmailOff", () => {
it("recognises the no-mail refusal", () => {
expect(isEmailOff(new HttpError(409, "x", { code: "resend", reason: "email_off" }))).toBe(true);
});
it("is false for any other failure, including a non-HttpError", () => {
expect(isEmailOff(new HttpError(422, "x", { code: "resend", reason: "send_failed" }))).toBe(
false,
);
expect(isEmailOff(new HttpError(500, "x", null))).toBe(false);
expect(isEmailOff(new TypeError("Failed to fetch"))).toBe(false);
expect(isEmailOff(undefined)).toBe(false);
});
});
frontend/src/lib/adminUsers.ts
+36
View File
@@ -0,0 +1,36 @@
import type { QueryClient } from "@tanstack/react-query";
import { HttpError } from "./api";
import { qk } from "./queryKeys";
// Shared bits of the admin's user-management surface, extracted because the SAME two mutations
// (activate, resend-verification) are now offered from two places — the pending-access queue and
// the account panel — and the second copy quietly dropped half of what the first one does.
//
// Grouped invalidation lives per domain rather than in `qk` (see the contract note there);
// `invalidateDownloads` is the same pattern for the downloads domain.
/** Everything that goes stale when an account is LET IN (activate, resend its confirmation).
*
* Suspend and delete move the same counters and are deliberately NOT wired here yet — they have
* their own pre-existing staleness, tracked separately, and widening this in passing would put an
* unreviewed change in a review-fix diff.
*
* `me` is in the set because the admin badge counts pending activations, and `admin-pending-access`
* because the queue is exactly the list of accounts this action removes one from — refreshing only
* `admin-users` leaves the badge and the queue asserting a person is still stuck after they were
* let in. One function so the two call sites cannot drift apart again. */
export function invalidateUserAccess(qc: QueryClient, userId?: number) {
qc.invalidateQueries({ queryKey: qk.adminUsers() });
qc.invalidateQueries({ queryKey: qk.adminPendingAccess() });
qc.invalidateQueries({ queryKey: qk.me() });
if (userId !== undefined) qc.invalidateQueries({ queryKey: qk.adminUserAccess(userId) });
}
/** Did an admin resend fail because the instance has no working mail, rather than for a real
* error? The admin-side resend deliberately reports failures (the public one is silent), so the
* message must be able to say WHICH — a no-SMTP instance would otherwise read as a bug. */
export function isEmailOff(err: unknown): boolean {
if (!(err instanceof HttpError)) return false;
const d = err.detailData as { code?: string; reason?: string } | null | undefined;
return d?.code === "resend" && d.reason === "email_off";
}
frontend/src/lib/api/admin.ts
+24
View File
@@ -75,10 +75,26 @@ export interface AdminUserRow {
email_verified: boolean;
is_demo: boolean;
has_password: boolean;
/** When SEC-10 cleared this account's password, or null. Separates "never had one" (a Google-only
* account) from "we took it away" — different problems needing different help. */
password_revoked_at: string | null;
has_google: boolean;
/** The Google account's address, ONLY when it differs from the account address (E9.5 S3). */
google_email: string | null;
created_at: string | null;
}
/** The account panel's view of why someone can or cannot get in (E12 S5). */
export interface AdminUserAccess extends AdminUserRow {
/** The address passes the invite gate — so Google sign-in will be let through, not filed as a
* fresh access request. */
is_allowed: boolean;
can_activate: boolean;
/** Whether this instance has working mail at all. Decides whether the two mail-shaped recovery
* actions are offered or shown as unavailable — rather than failing after the click. */
email_configured: boolean;
}
export interface AuditRow {
id: number;
created_at: string | null;
@@ -197,6 +213,14 @@ export const adminApi = {
// because the caller is already an authenticated admin.
resendUserVerification: (id: number): Promise<{ ok: boolean; email: string }> =>
req(`/api/admin/users/${id}/resend-verification`, { method: "POST" }, { quiet: true }),
// What the account panel shows: the two computed gate answers, resolved for one user on open.
adminUserAccess: (id: number): Promise<AdminUserAccess> => req(`/api/admin/users/${id}/access`),
// Mint a one-time, no-email link that lets this account set a password (SEC-15 / SEC-16, and the
// everyday "no SMTP and someone forgot theirs"). Issuing does not revoke the current password.
createSetPasswordLink: (
id: number,
): Promise<{ ok: boolean; url: string; expires_at: string | null; email: string }> =>
req(`/api/admin/users/${id}/set-password-link`, { method: "POST" }, { quiet: true }),
adminInvites: (status?: string): Promise<Invite[]> =>
req(`/api/admin/invites${status ? `?status=${status}` : ""}`),
approveInvite: (id: number) => req(`/api/admin/invites/${id}/approve`, { method: "POST" }),
frontend/src/lib/api/core.ts
+8
View File
@@ -72,6 +72,14 @@ function localizeDetail(d: unknown): string {
if (o?.code === "upload_too_large")
return t("downloads.errors.upload.tooLarge", { size: formatBytes(o.limit) });
if (o?.code === "upload_empty") return t("downloads.errors.upload.empty");
// SEC-12: a download URL the server refuses to fetch. The reasons are deliberately distinct —
// "we could not look that host up" is a different thing to tell someone than "we will not fetch
// from that address" — so an unknown reason falls back to the generic refusal rather than a key.
if (o?.code === "url_blocked")
return t(`downloads.errors.urlBlocked.${o.reason}`, t("downloads.errors.urlBlocked.generic"));
// SEC-14: a manual sync trigger on cooldown. Not an error the user caused so much as a pace
// limit, and the copy says so.
if (o?.code === "sync_cooldown") return t("errors.syncCooldown");
return t("errors.generic");
}
frontend/src/lib/queryKeys.ts
+1
View File
@@ -85,6 +85,7 @@ export const qk = {
// --- admin ---
adminUsers: () => ["admin-users"] as const,
adminUserAccess: (id: number) => ["admin-user-access", id] as const,
adminInvites: () => ["admin-invites"] as const,
adminPendingAccess: () => ["admin-pending-access"] as const,
adminConfig: () => ["admin-config"] as const,
frontend/src/lib/releaseNotes.ts
+26
View File
@@ -14,6 +14,32 @@ export interface ReleaseEntry {
}
export const RELEASE_NOTES: ReleaseEntry[] = [
{
version: "0.81.0",
date: "2026-09-05",
summary:
"If you run your own Siftlode, this one is mostly for you: there is now a way back from a bad upgrade, backup scripts that actually restore, and a way to let someone in on an instance that cannot send mail. If you just use Siftlode, the visible part is that a download can no longer quietly blow past your storage limit, and that the box refuses addresses on the server's own network.",
features: [
"Admins get an account panel: click anyone in Users and it says, in one place, why that person can or cannot get in — whether their address is confirmed, whether an admin has approved them, whether they have a password, which Google account is attached, and whether their address is on the allow-list. Every recovery action for that account sits under the readout, so you pick one having read what is actually wrong instead of guessing.",
'Someone locked out of an instance with no email setup can now be let back in. "Create a set-password link" gives you a one-time link to hand them however you like — chat, in person, on paper. It lasts 24 hours, works once, and signs out anyone already using that account. Issuing it changes nothing by itself: a misclick cannot lock anyone out.',
"Self-hosters can pin a version. Put `IMAGE_TAG=0.81.0` in your `.env` and Siftlode stays there until you change the line, so an upgrade happens when you decide it does. The docs now also carry a rollback recipe — and say plainly that restoring the dump you took first is the way back that does not lose data.",
"The backup and restore scripts work. On Windows they no longer corrupt the dump on the way out of the container, and on the documented self-host layout — the one where you download a single compose file — they now find it. The docs spell out that a backup is three things, not one: the database dump, your `.env` (without `TOKEN_ENCRYPTION_KEY` the restored secrets are unreadable), and your downloaded media.",
"Plex works on the public compose files. The read-only media mount and its empty-volume fallback are in both of them, with a short setup section — before, the feature existed but the file you were told to run could not satisfy it.",
"Siftlode is licensed, under AGPL-3.0.",
],
fixes: [
"A download can no longer exceed your storage quota. The limit was only checked when you queued a job, so a single large download could pass that check and then fill the disk — and once you were at your cap, the download running at that moment was the one with no ceiling at all. Downloads are now bounded by what you have left, and a job that finds nothing left says so instead of running anyway.",
"The download box refuses an address on the server's own network, and tells you why in your own language. Pasting an internal address used to make the server fetch it and hand the result back, which turned the box into a way to probe the machine Siftlode runs on. A malformed link is now refused as one rather than failing with a server error.",
"One person holding down the sync button no longer spends the shared YouTube budget for everyone on the instance. Sync and playlist pull now say, in plain words, that you have synced a few times just now and that background syncing keeps running either way — a pace limit rather than a failure.",
"A password-reset link opened in a tab you are already signed in to now says what to do with it. It used to do nothing at all: the page loaded, and the link appeared broken. The link itself stays in the address bar, so signing out from that same tab picks it up.",
"Admin actions that fail now say so. Sending a confirmation link on an instance whose mail is misconfigured used to produce nothing on screen at all, which reads exactly like it worked.",
"The row you click to open a user's account panel now looks clickable.",
],
chores: [
"E12, five sprints. S1: the backup/restore scripts honour COMPOSE_FILE and the PowerShell pair moves the dump with `docker compose cp` rather than through the pipeline (binary does not survive it on 5.1, and `-Encoding Byte` does not exist on 7); docs gained an Upgrading section, and `siftlode release` now tags `v$VERSION` and refuses on a stale CHANGELOG.md. S2: AGPL-3.0, the Plex mount and `init: true` on the worker in both public compose files. S3 (SEC-12/13/14): a URL guard that resolves the host and refuses loopback/private/link-local at enqueue AND again in the worker, a generalised network-class error so it stops working as a port oracle, playlist_items/max_downloads/max_filesize derived from the caller's remaining quota, and a per-user cooldown on the manual sync triggers. S4: the three E9.5 deferrals (DX-16, DX-17, canonical resolution in the setup wizard). S5 (SEC-15/16): the account panel and the admin-issued set-password link, after a mockup round.",
"The closing review ran six rounds across eight dimensions and converged 47 → 34 → 15 → 7 → 6 → 0. Round one's 47 deduplicated to 20 and 18 survived 60 adversarial verifier runs. Rounds two, four and five each found that a previous round's own fix had broken something — the ops half every time, because no gate can run shell scripts or a documented command. The three worth recording: the per-user quota refusal was writing onto a MediaAsset shared between users, so one person hitting their cap failed a second person's queued job for the same video with the first one's reason; the restore scripts' automatic restart re-ran `alembic upgrade head`, undoing the restore on the very rollback path the docs recommend; and the command printed in its place resolved to the other compose file, whose volumes differ, recreating the database empty. Two fixes were reverted rather than patched a third time. Round five also caught that the loop had been running seven of the gate's eight lanes. Residuals documented rather than closed: SEC-17 (redirects walk past the URL guard — the same socket-level fix as DNS rebinding), SEC-18 (the playlist push routes), SEC-19 (the edit path does not consult the byte cap).",
],
},
{
version: "0.80.0",
date: "2026-09-05",
frontend/src/lib/useVerifyEntry.ts
+55 -5
View File
@@ -1,16 +1,26 @@
import { useEffect, useRef, useState } from "react";
import { api } from "./api";
import { HttpError } from "./api/core";
import { clearEntryUrl, entryHash } from "./entryUrl";
/** What confirming the emailed token produced. `ok_reset` means the address is confirmed but this
* account's password had been revoked (SEC-10), so signing in would only meet the uniform credentials
* error — the user has to set a new one. */
type VerifyOutcome = "ok" | "ok_reset" | "invalid";
* error — the user has to set a new one. `unreachable` is a TRANSPORT failure, kept apart from
* `invalid` on purpose (DX-16): the fragment is already cleared by the time we know, so telling
* someone their link is spent when the server was merely restarting sends them to request a fresh one
* on the strength of a false statement. The same rule is already written one file away, for the
* reset-link probe: "a network blip must never be cached as 'this link is dead'". */
type VerifyOutcome = "ok" | "ok_reset" | "invalid" | "unreachable";
export type VerifyEntry = {
outcome: VerifyOutcome | null;
/** The address that was confirmed, when the server named it. */
email: string | null;
/** A `#reset=` link opened in a tab that already holds a session. Only the signed-OUT auth card
* consumes a reset token, so nothing here can act on it — but silence is what made the verify half
* a bug, so this says out loud that the link needs a signed-out window. The fragment is deliberately
* left in the address bar (E9.5 review round 4), so signing out still recovers it. */
resetPending: boolean;
/** Dismiss the outcome — the banner is a one-time entry notice, not persistent state. */
dismiss: () => void;
};
@@ -38,12 +48,24 @@ export function useVerifyEntry(): VerifyEntry {
const [token, setToken] = useState<string | null>(() => entryHash.get("verify"));
const [outcome, setOutcome] = useState<VerifyOutcome | null>(null);
const [email, setEmail] = useState<string | null>(null);
const [resetPending, setResetPending] = useState<boolean>(() => !!entryHash.get("reset"));
const consumed = useRef(new Set<string>());
useEffect(() => {
const onHashChange = () => {
const next = new URLSearchParams(window.location.hash.replace(/^#/, "")).get("verify");
const hash = new URLSearchParams(window.location.hash.replace(/^#/, ""));
const next = hash.get("verify");
if (next) setToken(next);
// Both halves of the mailed pair arrive the same way — pasted into a running tab, which is a
// same-document navigation the import-time snapshot never sees. The verify half got this
// listener in E9.5 S2; the reset half was left out, so it stayed silent (DX-17).
//
// Set from the live hash rather than one-way, because the flag must not outlive the token it
// describes: navigating away from a `#reset=` (a pasted verify link, the back button) leaves
// the banner promising that signing out will resume a reset whose token is gone — and, being
// checked first, hiding whatever arrived in its place. `clearEntryUrl`'s own replaceState
// fires no hashchange, so this cannot undo the deliberate keep-the-reset-fragment behaviour.
setResetPending(!!hash.get("reset"));
};
// replaceState (how clearEntryUrl strips the fragment) does NOT fire hashchange, so this cannot
// feed itself.
@@ -67,8 +89,36 @@ export function useVerifyEntry(): VerifyEntry {
setOutcome(r.password_reset_required ? "ok_reset" : "ok");
setEmail(r.email ?? null);
})
.catch(() => setOutcome("invalid"));
// Only the server's own verdict on the token means "invalid". `/auth/verify` refuses a spent
// or expired token with a 400 carrying {code:"verify"}; a 502 during a deploy, or a dropped
// connection (no HttpError at all), says nothing about the token and must not be reported as
// if it did. The token itself is unspent in that case — but the fragment has already been
// cleared, so the honest message is "we could not reach the server", not "request a new link".
.catch((e: unknown) => {
const spent =
e instanceof HttpError &&
e.status === 400 &&
(e.detailData as { code?: string } | undefined)?.code === "verify";
setOutcome(spent ? "invalid" : "unreachable");
});
}, [token]);
return { outcome, email, dismiss: () => setOutcome(null) };
return {
outcome,
email,
resetPending,
// Dismiss peels ONE layer. The two states are independent — a reset link can arrive over an
// unread confirmation, and a slow /auth/verify can resolve after one did — so clearing both at
// once silently discards whichever the user had not read yet. The banner shows the reset
// guidance first (it is the one with something to do), and dismissing it reveals the outcome
// underneath rather than taking it with it.
dismiss: () => {
if (resetPending) {
setResetPending(false);
return;
}
setOutcome(null);
setEmail(null);
},
};
}
frontend/src/lib/verifyEntry.dom.test.tsx
+129 -4
View File
@@ -30,7 +30,13 @@ async function mountAt(url: string, verifyEmail: ReturnType<typeof vi.fn>) {
function Probe() {
const entry = useVerifyEntry();
return <div data-testid="outcome">{`${entry.outcome ?? "none"}:${entry.email ?? "-"}`}</div>;
return (
<>
<div data-testid="outcome">{`${entry.outcome ?? "none"}:${entry.email ?? "-"}`}</div>
<div data-testid="reset">{entry.resetPending ? "reset-pending" : "-"}</div>
<button data-testid="dismiss" onClick={entry.dismiss} />
</>
);
}
await act(async () => {
render(<Probe />);
@@ -67,14 +73,51 @@ describe("confirming an emailed verification link", () => {
expect(outcome()).toBe("ok_reset:revoked@example.com");
});
it("reports invalid when the link is expired or already used", async () => {
const verifyEmail = vi.fn().mockRejectedValue(new Error("400"));
// Errors are built INSIDE the rejection, not hoisted above the test: mountAt calls
// vi.resetModules(), so an HttpError imported earlier would be a different class identity than the
// one the hook imports afterwards, and `instanceof` would quietly miss.
const rejectWith = (make: (E: typeof import("./api/core").HttpError) => unknown) =>
vi.fn().mockImplementation(async () => {
const { HttpError } = await import("./api/core");
throw make(HttpError);
});
it("reports invalid when the SERVER says the link is expired or already used", async () => {
const verifyEmail = rejectWith((E) => new E(400, "verify", { code: "verify" }));
const outcome = await mountAt("/#verify=tok-dead", verifyEmail);
expect(outcome()).toBe("invalid:-");
});
// DX-16: the fragment is already cleared by the time this resolves, so reporting a transport
// failure as "that link is no longer valid" pushes the user into requesting a fresh link on the
// strength of a statement that is false — the token is untouched and retrying it works.
it("does not call the link dead on a gateway failure mid-deploy", async () => {
const verifyEmail = rejectWith((E) => new E(502, "Bad Gateway"));
const outcome = await mountAt("/#verify=tok-live", verifyEmail);
expect(outcome()).toBe("unreachable:-");
});
it("does not call the link dead when the connection drops with no HTTP status at all", async () => {
const verifyEmail = vi.fn().mockRejectedValue(new TypeError("Failed to fetch"));
const outcome = await mountAt("/#verify=tok-offline", verifyEmail);
expect(outcome()).toBe("unreachable:-");
});
// A 400 that is not the verify refusal is still not a verdict on the token.
it("does not call the link dead on a 400 that is not the verify refusal", async () => {
const verifyEmail = rejectWith((E) => new E(400, "nope", { code: "other" }));
const outcome = await mountAt("/#verify=tok-other", verifyEmail);
expect(outcome()).toBe("unreachable:-");
});
it("clears the one-time query flags it owns even with nothing to confirm", async () => {
const verifyEmail = vi.fn();
@@ -97,6 +140,83 @@ describe("confirming an emailed verification link", () => {
expect(window.location.hash).toBe("#reset=unspent");
expect(window.location.search).toBe(""); // the flag it does own still goes
expect(screen.getByTestId("reset").textContent).toBe("reset-pending");
});
// DX-17: the verify half got a hashchange listener in E9.5 S2, the reset half did not — so pasting
// a reset link into a tab already running the app was silently ignored. Our mails were plain text
// until 0.78.8, which makes pasting the normal way to use one.
it("notices a reset link pasted into a tab that is already running", async () => {
await mountAt("/", vi.fn());
expect(screen.getByTestId("reset").textContent).toBe("-");
await act(async () => {
window.history.replaceState(window.history.state, "", "/#reset=pasted");
window.dispatchEvent(new HashChangeEvent("hashchange"));
});
expect(screen.getByTestId("reset").textContent).toBe("reset-pending");
});
// The flag must not outlive the token it describes. Navigating away from a `#reset=` fragment —
// a pasted verify link, the back button — leaves the address bar with no reset token, so a
// still-true flag promises a recovery that signing out can no longer complete AND (being checked
// first by VerifyBanner) hides whatever arrived in its place.
it("drops the reset notice once the fragment carrying it is gone", async () => {
const verifyEmail = vi.fn().mockResolvedValue({ ok: true, email: "someone@example.com" });
const outcome = await mountAt("/#reset=going-away", verifyEmail);
expect(screen.getByTestId("reset").textContent).toBe("reset-pending");
await act(async () => {
window.history.replaceState(window.history.state, "", "/#verify=tok-next");
window.dispatchEvent(new HashChangeEvent("hashchange"));
});
expect(screen.getByTestId("reset").textContent).toBe("-");
expect(outcome()).toBe("ok:someone@example.com");
});
// …and a fragment change that carries neither token drops it too (the back button out of a
// reset link), which a one-way "set it when you see one" listener would not.
it("drops the reset notice when the fragment changes to no token at all", async () => {
await mountAt("/#reset=going-away", vi.fn());
expect(screen.getByTestId("reset").textContent).toBe("reset-pending");
await act(async () => {
window.history.replaceState(window.history.state, "", "/#somewhere-else");
window.dispatchEvent(new HashChangeEvent("hashchange"));
});
expect(screen.getByTestId("reset").textContent).toBe("-");
});
// A reset link can arrive over an unread confirmation (and a slow /auth/verify can resolve after
// one did), so both states can be set at once. Neither may eat the other: `dismiss` used to clear
// both, which is how the reset guidance could be destroyed before it was ever rendered.
it("keeps a pasted reset link and an unread confirmation, and dismisses them one at a time", async () => {
const verifyEmail = vi.fn().mockResolvedValue({ ok: true, email: "someone@example.com" });
const outcome = await mountAt("/#verify=tok-first", verifyEmail);
expect(outcome()).toBe("ok:someone@example.com");
await act(async () => {
window.history.replaceState(window.history.state, "", "/#reset=pasted");
window.dispatchEvent(new HashChangeEvent("hashchange"));
});
// Both live: the reset is pending AND the confirmation is still unread.
expect(screen.getByTestId("reset").textContent).toBe("reset-pending");
expect(outcome()).toBe("ok:someone@example.com");
// One dismiss peels the reset guidance and leaves the confirmation standing…
await act(async () => screen.getByTestId("dismiss").click());
expect(screen.getByTestId("reset").textContent).toBe("-");
expect(outcome()).toBe("ok:someone@example.com");
// …and the next one clears that.
await act(async () => screen.getByTestId("dismiss").click());
expect(outcome()).toBe("none:-");
});
// This hook runs on the FIRST commit of App, which is several commits before the setup wizard, the
@@ -145,7 +265,12 @@ describe("a verification link that arrives as a fragment-only navigation", () =>
const verifyEmail = vi
.fn()
.mockResolvedValueOnce({ ok: true, email: "first@example.com" })
.mockRejectedValueOnce(new Error("400"));
// The server's own refusal, not a bare Error: only a 400 carrying {code:"verify"} means the
// token is spent, and this case is about the SECOND outcome replacing the first.
.mockImplementationOnce(async () => {
const { HttpError } = await import("./api/core");
throw new HttpError(400, "verify", { code: "verify" });
});
const outcome = await mountAt("/#verify=first-token", verifyEmail);
expect(outcome()).toBe("ok:first@example.com");
scripts/backup.ps1
+53 -2
View File
@@ -1,10 +1,61 @@
# Dumps the Siftlode Postgres database to backupssiftlode-<timestamp>.dump (custom format).
# Dumps the Siftlode Postgres database to backups\siftlode-<timestamp>.dump (custom format).
# Run from the project root: .\scripts\backup.ps1
#
# A dump is only HALF a backup. The other half is your .env — it holds TOKEN_ENCRYPTION_KEY, without
# which every encrypted value in the restored database (Google credentials, SMTP password, Plex
# token, stored refresh tokens) is unreadable. See docs/self-hosting.md § Backups.
$ErrorActionPreference = "Stop"
# The dump is moved with `docker compose cp`, never through the PowerShell pipeline. Piping a binary
# stream corrupts it on Windows PowerShell 5.1 (text decoding), and on PowerShell 7 the old
# `-Encoding Byte` workaround does not even exist any more — it throws "'Byte' is not a supported
# encoding name", so this script simply failed there.
$remote = "/tmp/siftlode-backup.dump"
# Best-effort cleanup of the in-container dump. The try/catch is the load-bearing part: on Windows
# PowerShell 5.1 ANY redirected native stderr (`2>$null` and `2>&1` alike) becomes an ErrorRecord,
# and the $ErrorActionPreference above makes it TERMINATING — so one stray Compose warning would
# kill the script here, masking the real diagnostic below or reporting a finished backup as a
# failure. Do not remove the empty catch. A cleanup step must never decide the verdict.
function Remove-RemoteDump($path) {
try { docker compose exec -T db rm -f $path 2>&1 | Out-Null } catch { }
}
# `docker compose` finds docker-compose.yml on its own and honours $env:COMPOSE_FILE. The documented
# self-host layout (docs/self-hosting.md § Get the files) has NEITHER — just the selfhost file.
if (-not $env:COMPOSE_FILE -and -not (Test-Path "docker-compose.yml") -and (Test-Path "docker-compose.selfhost.yml")) {
$env:COMPOSE_FILE = "docker-compose.selfhost.yml"
}
$user = if ($env:POSTGRES_USER) { $env:POSTGRES_USER } else { "siftlode" }
$dbname = if ($env:POSTGRES_DB) { $env:POSTGRES_DB } else { "siftlode" }
New-Item -ItemType Directory -Force -Path "backups" | Out-Null
$ts = Get-Date -Format "yyyyMMdd-HHmmss"
$out = "backups\siftlode-$ts.dump"
docker compose exec -T db pg_dump -U $user -Fc $dbname | Set-Content -NoNewline -Encoding Byte $out
docker compose exec -T db pg_dump -U $user -Fc $dbname -f $remote
# Capture BEFORE the cleanup call: any native command overwrites $LASTEXITCODE, so interpolating it
# after Remove-RemoteDump would report the cleanup's code (0) inside a branch that only runs when
# pg_dump failed — a message contradicting itself. Same reason $cpCode is captured below.
$dumpCode = $LASTEXITCODE
if ($dumpCode -ne 0) {
Remove-RemoteDump $remote
throw "BACKUP FAILED — pg_dump returned $dumpCode; no file was written."
}
docker compose cp "db:$remote" $out
$cpCode = $LASTEXITCODE
Remove-RemoteDump $remote
# Remove the partial file: `cp` streams, so a mid-stream failure leaves a truncated dump under
# the final name — non-zero length, so the size check below would not catch it either.
if ($cpCode -ne 0) {
Remove-Item $out -ErrorAction SilentlyContinue
throw "BACKUP FAILED — could not copy the dump out of the container (exit $cpCode)."
}
if (-not (Test-Path $out) -or (Get-Item $out).Length -eq 0) {
if (Test-Path $out) { Remove-Item $out }
throw "BACKUP FAILED — the dump came out empty; no backup was written."
}
Write-Output "Wrote $out"
Write-Output "Remember: back up your .env too (TOKEN_ENCRYPTION_KEY) — this dump alone cannot be fully restored."
scripts/backup.sh
+29 -1
View File
@@ -1,11 +1,39 @@
#!/bin/sh
# Dumps the Siftlode Postgres database to backups/siftlode-<timestamp>.dump (custom format).
# Run from the project root: ./scripts/backup.sh
#
# A dump is only HALF a backup. The other half is your `.env` — it holds TOKEN_ENCRYPTION_KEY,
# without which every encrypted value in the restored database (Google credentials, SMTP password,
# Plex token, stored refresh tokens) is unreadable. See docs/self-hosting.md § Backups.
set -e
# `docker compose` finds docker-compose.yml on its own and honours $COMPOSE_FILE. The documented
# self-host layout (docs/self-hosting.md § Get the files) has NEITHER — just the selfhost file — so
# without this the command below failed there with "no configuration file provided".
if [ -z "${COMPOSE_FILE:-}" ] && [ ! -f docker-compose.yml ] && [ -f docker-compose.selfhost.yml ]; then
COMPOSE_FILE=docker-compose.selfhost.yml
export COMPOSE_FILE
fi
USER_NAME="${POSTGRES_USER:-siftlode}"
DB_NAME="${POSTGRES_DB:-siftlode}"
mkdir -p backups
TS=$(date +%Y%m%d-%H%M%S)
OUT="backups/siftlode-$TS.dump"
docker compose exec -T db pg_dump -U "$USER_NAME" -Fc "$DB_NAME" > "$OUT"
# Write to a temp name and only promote it on success: a failed pg_dump would otherwise leave a
# truncated (often 0-byte) file sitting in backups/ that looks exactly like a real backup.
TMP="$OUT.part"
if ! docker compose exec -T db pg_dump -U "$USER_NAME" -Fc "$DB_NAME" > "$TMP"; then
rm -f "$TMP"
echo "BACKUP FAILED — pg_dump returned an error; no file was written." >&2
exit 1
fi
if [ ! -s "$TMP" ]; then
rm -f "$TMP"
echo "BACKUP FAILED — pg_dump produced an empty file; no backup was written." >&2
exit 1
fi
mv "$TMP" "$OUT"
echo "Wrote $OUT"
echo "Remember: back up your .env too (TOKEN_ENCRYPTION_KEY) — this dump alone cannot be fully restored."
scripts/gen-changelog.mjs
+86
View File
@@ -0,0 +1,86 @@
// Generates CHANGELOG.md from frontend/src/lib/releaseNotes.ts — the same entries the app shows
// under Release Notes, so a self-hoster deciding whether to upgrade reads the same text an
// in-app user does, without having to install the app first (DX-14).
//
// Why derive rather than hand-write: two lists of "what changed in 0.80.0" WILL drift, and the
// in-app one is the one that gets written during a release. This makes the file a build product of
// it. RELEASE_NOTES is a plain data literal by design, so it is imported by stripping the single TS
// type annotation off the declaration — no transpiler, no dependency.
//
// node scripts/gen-changelog.mjs # write CHANGELOG.md
// node scripts/gen-changelog.mjs --check # exit 1 if CHANGELOG.md is stale (used by `siftlode release`)
import { readFile, writeFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import path from "node:path";
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
const SOURCE = path.join(root, "frontend", "src", "lib", "releaseNotes.ts");
const TARGET = path.join(root, "CHANGELOG.md");
async function loadEntries() {
const src = await readFile(SOURCE, "utf8");
const decl = "export const RELEASE_NOTES";
const at = src.indexOf(decl);
if (at === -1) {
throw new Error(`${SOURCE} no longer declares ${decl} — the generator's assumption broke, fix it rather than hand-writing the changelog.`);
}
const body = src.slice(at).replace(/^export const RELEASE_NOTES\s*:[^=]*=/, "export const RELEASE_NOTES =");
const mod = await import(`data:text/javascript,${encodeURIComponent(body)}`);
if (!Array.isArray(mod.RELEASE_NOTES) || mod.RELEASE_NOTES.length === 0) {
throw new Error("RELEASE_NOTES imported as something other than a non-empty array.");
}
return mod.RELEASE_NOTES;
}
function section(title, items) {
if (!items || items.length === 0) return "";
return `### ${title}\n\n${items.map((line) => `- ${line}`).join("\n")}\n\n`;
}
function render(entries) {
const head = [
"# Changelog",
"",
"Every released version of Siftlode, newest first.",
"",
"**Generated from `frontend/src/lib/releaseNotes.ts` by `scripts/gen-changelog.mjs` — do not edit by hand;**",
"edit the release notes and re-run the generator.",
"",
"Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).",
"",
"",
].join("\n");
const body = entries
.map((e) => {
const sha = e.sha ? ` (\`${e.sha}\`)` : "";
let out = `## ${e.version} — ${e.date}${sha}\n\n`;
if (e.summary) out += `${e.summary}\n\n`;
out += section("Features", e.features);
out += section("Fixes", e.fixes);
out += section("Internal", e.chores);
return out;
})
.join("");
return `${head}${body}`.replace(/\n{3,}$/, "\n");
}
const entries = await loadEntries();
const rendered = render(entries);
if (process.argv.includes("--check")) {
const current = await readFile(TARGET, "utf8").catch(() => null);
if (current !== rendered) {
console.error(
`CHANGELOG.md is out of date with releaseNotes.ts (newest entry: ${entries[0].version}).\n` +
"Run: node scripts/gen-changelog.mjs — then commit the result.",
);
process.exit(1);
}
console.log(`CHANGELOG.md is current (${entries.length} releases, newest ${entries[0].version}).`);
} else {
await writeFile(TARGET, rendered, "utf8");
console.log(`Wrote CHANGELOG.md — ${entries.length} releases, newest ${entries[0].version}.`);
}
scripts/restore.ps1
+68 -2
View File
@@ -1,9 +1,75 @@
# Restores a Siftlode Postgres dump created by backup.ps1 into the running db container.
# Usage: .\scripts\restore.ps1 backupssiftlode-YYYYMMDD-HHMMSS.dump
# Usage: .\scripts\restore.ps1 backups\siftlode-YYYYMMDD-HHMMSS.dump
#
# Restoring the database does NOT restore the secrets that decrypt it. If this dump is from an
# instance with a different TOKEN_ENCRYPTION_KEY than the current .env, the encrypted sysconfig
# values and refresh tokens will not be readable. See docs/self-hosting.md § Backups.
param([Parameter(Mandatory = $true)][string]$DumpFile)
$ErrorActionPreference = "Stop"
$WarningPreference = "Continue"
# See backup.ps1: the dump travels with `docker compose cp`, never through the PowerShell pipeline,
# which corrupts binary on 5.1 and cannot express `-Encoding Byte` at all on 7.
$remote = "/tmp/siftlode-restore.dump"
# See backup.ps1: on Windows PowerShell 5.1 a redirected native stderr is a TERMINATING error under
# $ErrorActionPreference = "Stop", so without the catch a cleanup step could kill the script after
# the restore already succeeded — and the obvious next move, re-running it, wipes the database again.
function Remove-RemoteDump($path) {
try { docker compose exec -T db rm -f $path 2>&1 | Out-Null } catch { }
}
if (-not $env:COMPOSE_FILE -and -not (Test-Path "docker-compose.yml") -and (Test-Path "docker-compose.selfhost.yml")) {
$env:COMPOSE_FILE = "docker-compose.selfhost.yml"
}
$user = if ($env:POSTGRES_USER) { $env:POSTGRES_USER } else { "siftlode" }
$dbname = if ($env:POSTGRES_DB) { $env:POSTGRES_DB } else { "siftlode" }
if (-not (Test-Path $DumpFile)) { throw "Dump file not found: $DumpFile" }
Get-Content -Encoding Byte $DumpFile | docker compose exec -T db pg_restore -U $user -d $dbname --clean --if-exists
docker compose cp $DumpFile "db:$remote"
$cpCode = $LASTEXITCODE
if ($cpCode -ne 0) { throw "RESTORE FAILED — could not copy the dump into the container (exit $cpCode)." }
# Stop the app before dropping its tables. `--clean` DROPs each object in its own transaction, and a
# running api (with its scheduler) holds locks that make those DROPs queue behind it — abort the wait
# and the database is left half-dropped. It also keeps the app from writing into a schema that is
# only partly back.
#
# It is NOT restarted automatically, on purpose. Every start runs `alembic upgrade head`, so on the
# documented rollback path (restore a pre-upgrade dump, THEN pin the older IMAGE_TAG) an automatic
# restart would re-apply the very migrations the restore just undid. On a FAILED restore it would
# put the app back on a half-restored schema, which is the second thing the stop exists to prevent.
# The operator starts it when they are ready — with `up -d`, not `start`: `start` relaunches the
# EXISTING container, so it ignores a newly pinned IMAGE_TAG and comes back on the image being
# rolled back FROM.
docker compose stop api worker
$stopCode = $LASTEXITCODE
if ($stopCode -ne 0) { throw "Could not stop api/worker (exit $stopCode) — refusing to restore while the app may still be writing." }
docker compose exec -T db pg_restore -U $user -d $dbname --clean --if-exists $remote
$restoreCode = $LASTEXITCODE
Remove-RemoteDump $remote
# pg_restore exits non-zero for ignorable warnings too (--clean --if-exists on a fresh database
# reports every missing object it tried to drop), so a non-zero code is reported as "check this",
# not swallowed and not dressed up as success.
if ($restoreCode -ne 0) {
# Write-Warning, not Write-Output: `>` redirects only the SUCCESS stream, so on stdout this
# explanation would be filed into the log while the bare exception stayed on screen. (The
# warning stream is its own stream, not the error stream — $WarningPreference is pinned at the
# top so an inherited "SilentlyContinue" cannot drop it.)
Write-Warning "api and worker are left STOPPED: the database may be half-restored. Fix the dump and re-run before starting them."
throw "pg_restore reported errors while restoring $DumpFile (exit $restoreCode) — read the output above before treating this instance as restored."
}
Write-Output "Restored $DumpFile"
# Deliberately not a runnable line when we could not resolve the file: both compose files can be
# present (a source checkout), and the two declare DIFFERENT volume names — so a pasted bare
# 'docker compose up -d' would recreate the database on an empty volume and strand the restore.
$cf = if ($env:COMPOSE_FILE) { $env:COMPOSE_FILE } else { "<the compose file you started with>" }
Write-Output "api and worker are stopped. Start them again with:"
Write-Output " docker compose -f $cf up -d"
Write-Output "It must be the SAME file you started with: 'up -d' recreates from whatever file it reads,"
Write-Output "and a different one points at different volumes. Rolling a release back? Pin the older"
Write-Output "IMAGE_TAG in .env FIRST: every start re-runs the migrations, and 'start' would relaunch the"
Write-Output "existing container on the image you are rolling back FROM."
scripts/restore.sh
+52 -1
View File
@@ -1,11 +1,62 @@
#!/bin/sh
# Restores a Siftlode Postgres dump created by backup.sh into the running db container.
# Usage: ./scripts/restore.sh backups/siftlode-YYYYMMDD-HHMMSS.dump
#
# Restoring the database does NOT restore the secrets that decrypt it. If this dump is from an
# instance with a different TOKEN_ENCRYPTION_KEY than the current .env, the encrypted sysconfig
# values and refresh tokens will not be readable. See docs/self-hosting.md § Backups.
set -e
# See backup.sh: the documented self-host layout ships only docker-compose.selfhost.yml, so plain
# `docker compose` has no configuration file to find there.
if [ -z "${COMPOSE_FILE:-}" ] && [ ! -f docker-compose.yml ] && [ -f docker-compose.selfhost.yml ]; then
COMPOSE_FILE=docker-compose.selfhost.yml
export COMPOSE_FILE
fi
FILE="$1"
if [ -z "$FILE" ]; then echo "Usage: restore.sh <dumpfile>"; exit 1; fi
if [ ! -f "$FILE" ]; then echo "Dump file not found: $FILE"; exit 1; fi
USER_NAME="${POSTGRES_USER:-siftlode}"
DB_NAME="${POSTGRES_DB:-siftlode}"
docker compose exec -T db pg_restore -U "$USER_NAME" -d "$DB_NAME" --clean --if-exists < "$FILE"
# Stop the app before dropping its tables. `--clean` DROPs each object in its own transaction, and a
# running api (with its scheduler) holds locks that make those DROPs queue behind it — abort the wait
# and the database is left half-dropped. It also keeps the app from writing into a schema that is
# only partly back.
#
# It is NOT restarted automatically, on purpose. Every start runs `alembic upgrade head`, so on the
# documented rollback path (restore a pre-upgrade dump, THEN pin the older IMAGE_TAG) an automatic
# restart would re-apply the very migrations the restore just undid. On a FAILED restore it would
# put the app back on a half-restored schema, which is the second thing the stop exists to prevent.
# The operator starts it when they are ready — with `up -d`, not `start`: `start` relaunches the
# EXISTING container, so it ignores a newly pinned IMAGE_TAG and comes back on the image being
# rolled back FROM.
if ! docker compose stop api worker; then
echo "Could not stop api/worker — refusing to restore while the app may still be writing." >&2
exit 1
fi
# pg_restore exits non-zero for ignorable warnings too (--clean --if-exists on a fresh database
# reports every missing object it tried to drop), so a non-zero code is reported as "check this",
# not swallowed and not dressed up as success.
if docker compose exec -T db pg_restore -U "$USER_NAME" -d "$DB_NAME" --clean --if-exists < "$FILE"; then
echo "Restored $FILE"
echo "api and worker are stopped. Start them again with:"
if [ -n "${COMPOSE_FILE:-}" ]; then
echo " docker compose -f $COMPOSE_FILE up -d"
else
# Deliberately NOT a runnable line. Both compose files can be present (a source checkout), and
# the two declare DIFFERENT volume names — so a pasted bare 'docker compose up -d' would recreate
# the database on an empty volume and strand what was just restored.
echo " docker compose -f <the compose file you started with> up -d"
fi
echo "It must be the SAME file you started with: 'up -d' recreates from whatever file it reads,"
echo "and a different one points at different volumes. Rolling a release back? Pin the older"
echo "IMAGE_TAG in .env FIRST: every start re-runs the migrations, and 'start' would relaunch the"
echo "existing container on the image you are rolling back FROM."
else
echo "pg_restore reported errors while restoring $FILE — read the output above before treating this instance as restored." >&2
echo "api and worker are left STOPPED: the database may be half-restored. Fix the dump and re-run before starting them." >&2
exit 1
fi