https://github.com/mjpieters/aiolimiter is a better fit and may be used for copy/pasting
Mar 25 2022
Mar 23 2022
Jan 25 2022
Jan 24 2022
Jan 22 2022
Jan 21 2022
https://github.com/vutran1710/PyrateLimiter looks really good but is not for IO throttling
https://www.django-rest-framework.org/api-guide/throttling/ is already used by swh-web/swh/web/api/throttling.py but is targeting REST API throttling, not a good fit.
Jan 10 2022
Nevermind. I just realized that https://forge.softwareheritage.org/D6796 was in the "Accepted" state and has been for almost two weeks. :facepalm:
Jan 3 2022
Happy new year ! Or, in French, "Bonne année deux mille vingt d'oeufs" (pun on words courtesy Easter-Eggs).
Dec 27 2021
Dec 22 2021
I updated D6796 to add a high level description of the files, classes and logic. I also split the WineryObjStorage class into four classes in an attempt to clarify the separation between the read and the write logic. I will now refrain from doing anything else: it can be difficult on the reviewers to follow a moving target :-D
@olasd @vlorentz @vsellier @ardumont I'd be happy to start polishing the implementation after your first wave of review (thanks a lot for the time you spent on this, it is much appreciated 👍 ). I'm however concerned that it might make your work difficult because of the magnitude of the patch. An alternative would be to merge this first implementation, as long as you feel confident it won't break anything when deployed. It should not because it won't be instantiated. That would allow me to propose patches that are smaller and easier to review for:
Dec 20 2021
@douardda will you have time for reviewing this week? I stand ready to address your comments. I'll stop working on that for now so that it is not a moving target.
Dec 14 2021
Dec 13 2021
Dec 12 2021
Duplicate of T3533
Dec 11 2021
I think D6796 is ready for review and should be merged. It is not final and won't run in production but it is a working implementation of the object storage. From there I intend to setup a benchmarking environment on grid5000. Optimizations will be implemented to improve the benchmark results where it matters the most. The benchmarks that were run during the summer already provided insights which motivated the implementation of the shard packer in C.
Dec 8 2021
D6796 is a draft implementation that is not ready for review but can be reviewed anyways in case someone is curious.
Nov 15 2021
Alternative hardware bill of material: