Page MenuHomeSoftware Heritage

dachary (Loïc Dachary)
User

Projects

User does not belong to any projects.

User Details

User Since
Jan 8 2021, 11:21 PM (90 w, 6 d)

Recent Activity

Jan 26 2022

dachary closed D7036: switch to sha256 for hashing & better read performances.
Jan 26 2022, 10:10 AM
dachary committed rDOBJS0b6bee78ab5c: switch to sha256 for hashing & better read performances (authored by dachary).
switch to sha256 for hashing & better read performances
Jan 26 2022, 10:10 AM

Jan 25 2022

dachary added a comment to T3527: Self-host Software Heritage on grid5000.

I'mt not exactly sure why I thought that would be necessary for benchmarking. In any case... it's not ;-)

Jan 25 2022, 9:53 PM · Object storage
dachary closed T3527: Self-host Software Heritage on grid5000, a subtask of T3432: Add winery backend, as Wontfix.
Jan 25 2022, 9:53 PM · Object storage
dachary closed T3527: Self-host Software Heritage on grid5000 as Wontfix.
Jan 25 2022, 9:53 PM · Object storage
dachary closed T3525: grid5000 tools and documentation, a subtask of T3432: Add winery backend, as Resolved.
Jan 25 2022, 9:52 PM · Object storage
dachary closed T3525: grid5000 tools and documentation as Resolved.
Jan 25 2022, 9:52 PM · Object storage
dachary added a comment to T3525: grid5000 tools and documentation.

The documentation is at:

Jan 25 2022, 9:52 PM · Object storage
dachary closed T3634: Create swh-perfecthash module, a subtask of T3104: Persistent readonly perfect hash table, as Resolved.
Jan 25 2022, 9:51 PM · Object storage (RedHat collaboration)
dachary closed T3634: Create swh-perfecthash module as Resolved.
Jan 25 2022, 9:51 PM · Object storage
dachary closed T3528: Add winery backend: grid5000 benchmark, a subtask of T3432: Add winery backend, as Resolved.
Jan 25 2022, 9:50 PM · Object storage
dachary closed T3528: Add winery backend: grid5000 benchmark as Resolved.
Jan 25 2022, 9:50 PM · Object storage
dachary added a comment to T3528: Add winery backend: grid5000 benchmark.

It's documented in the winery test environment and was actually able to use the instructions successfully (after a few fixes...). It does work an this can be closed as resolved.

Jan 25 2022, 9:50 PM · Object storage
dachary added a comment to T3432: Add winery backend.

Added a wiki page to be a more accessible version of the benchmark process than the README in the sources.

Jan 25 2022, 9:48 PM · Object storage
dachary updated the diff for D7036: switch to sha256 for hashing & better read performances.

Adjust the disk names on grid5000 to reflect changes at https://www.grid5000.fr/w/Grenoble:Hardware#dahu

Jan 25 2022, 6:10 PM
dachary added a comment to D7033: the desired key len is 32 for sha256.
In D7033#182832, @olasd wrote:

Thanks! (and sorry for the hash algo ping-pong)

Jan 25 2022, 4:48 PM
dachary requested review of D7036: switch to sha256 for hashing & better read performances.
Jan 25 2022, 4:20 PM
dachary closed D7033: the desired key len is 32 for sha256.
Jan 25 2022, 3:48 PM
dachary committed rDOPH3b5a0b09c9d3: the desired key len is 32 for sha256 (authored by dachary).
the desired key len is 32 for sha256
Jan 25 2022, 3:48 PM
dachary requested review of D7033: the desired key len is 32 for sha256.
Jan 25 2022, 3:37 PM
dachary closed D7031: the key has a fixed len: do not hardcode it.
Jan 25 2022, 3:24 PM
dachary committed rDOPHd79745c582c8: the key has a fixed len: do not hardcode it (authored by dachary).
the key has a fixed len: do not hardcode it
Jan 25 2022, 3:24 PM
dachary requested review of D7031: the key has a fixed len: do not hardcode it.
Jan 25 2022, 3:20 PM
dachary closed D7030: implement statistics collection during benchmarks.
Jan 25 2022, 1:04 PM
dachary committed rDOBJS7b5538279be0: implement statistics collection during benchmarks (authored by dachary).
implement statistics collection during benchmarks
Jan 25 2022, 1:04 PM
dachary added inline comments to D7030: implement statistics collection during benchmarks.
Jan 25 2022, 12:58 PM
dachary updated the diff for D7030: implement statistics collection during benchmarks.

address review comments

Jan 25 2022, 12:58 PM
dachary updated the diff for D7030: implement statistics collection during benchmarks.

remove spurious print

Jan 25 2022, 11:25 AM
dachary requested review of D7030: implement statistics collection during benchmarks.
Jan 25 2022, 11:10 AM
dachary closed D7026: grid5000: fix errors and make it faster.
Jan 25 2022, 10:50 AM
dachary committed rDOBJS554c12a851ff: grid5000: fix errors and make it faster (authored by dachary).
grid5000: fix errors and make it faster
Jan 25 2022, 10:50 AM

Jan 24 2022

dachary added a comment to D7026: grid5000: fix errors and make it faster.

Could you add a short readme or something in winery-test-environment/mitogen-strategy/, so future reader know the rationale and where this is coming from?

Jan 24 2022, 6:27 PM
dachary updated the diff for D7026: grid5000: fix errors and make it faster.
  • Added a README.txt to explain the need for the mitogen plugins
  • Added mitogen as a test dependency
Jan 24 2022, 6:27 PM
dachary updated the diff for D7026: grid5000: fix errors and make it faster.

ensure the SSH key is 600 otherwise it leads to non-intuitive failures

Jan 24 2022, 4:01 PM
dachary updated the summary of D7026: grid5000: fix errors and make it faster.
Jan 24 2022, 3:44 PM
dachary requested review of D7026: grid5000: fix errors and make it faster.
Jan 24 2022, 3:42 PM
dachary changed the status of T3531: IO throttling: benchmark, a subtask of T3532: IO throttling, from Open to Work in Progress.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary changed the status of T3531: IO throttling: benchmark from Open to Work in Progress.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary closed T3530: IO throttling: implementation, a subtask of T3532: IO throttling, as Resolved.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary closed T3530: IO throttling: implementation as Resolved.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary closed D7022: winery: implement read/write throttling.
Jan 24 2022, 2:13 PM
dachary committed rDOBJS5904f7c155de: winery: implement read/write throttling (authored by dachary).
winery: implement read/write throttling
Jan 24 2022, 2:13 PM
dachary added inline comments to D7022: winery: implement read/write throttling.
Jan 24 2022, 2:12 PM
dachary updated the diff for D7022: winery: implement read/write throttling.
  • Remove spurious table added to sharedbase
  • Reduce the number of PG in the Ceph cluster: 100 is enough for test purposes
Jan 24 2022, 12:21 PM
dachary updated the diff for D7022: winery: implement read/write throttling.

Looks good, as usually.

Jan 24 2022, 12:12 PM
dachary added inline comments to D7022: winery: implement read/write throttling.
Jan 24 2022, 12:11 PM

Jan 22 2022

dachary added a comment to D7022: winery: implement read/write throttling.

This patch contains:

Jan 22 2022, 4:20 PM
dachary requested review of D7022: winery: implement read/write throttling.
Jan 22 2022, 4:17 PM
dachary changed the status of T3532: IO throttling, a subtask of T3432: Add winery backend, from Open to Work in Progress.
Jan 22 2022, 4:14 PM · Object storage
dachary changed the status of T3532: IO throttling from Open to Work in Progress.
Jan 22 2022, 4:14 PM · Object storage (RedHat collaboration)

Jan 21 2022

dachary added a comment to T3530: IO throttling: implementation.

https://github.com/mjpieters/aiolimiter is a better fit and may be used for copy/pasting

Jan 21 2022, 4:28 PM · Object storage (RedHat collaboration)
dachary added a comment to T3530: IO throttling: implementation.

https://github.com/vutran1710/PyrateLimiter looks really good but is not for IO throttling

Jan 21 2022, 4:18 PM · Object storage (RedHat collaboration)
dachary added a comment to T3530: IO throttling: implementation.

https://www.django-rest-framework.org/api-guide/throttling/ is already used by swh-web/swh/web/api/throttling.py but is targeting REST API throttling, not a good fit.

Jan 21 2022, 4:06 PM · Object storage (RedHat collaboration)
dachary changed the status of T3530: IO throttling: implementation, a subtask of T3532: IO throttling, from Open to Work in Progress.
Jan 21 2022, 3:53 PM · Object storage (RedHat collaboration)
dachary changed the status of T3530: IO throttling: implementation from Open to Work in Progress.
Jan 21 2022, 3:53 PM · Object storage (RedHat collaboration)

Jan 10 2022

dachary closed D6796: winery: basic implementation of the backend.
Jan 10 2022, 9:42 AM
dachary committed rDOBJS9a88cafbc72c: winery: first implementation (authored by dachary).
winery: first implementation
Jan 10 2022, 9:42 AM
dachary updated the diff for D6796: winery: basic implementation of the backend.

rebase on top of master

Jan 10 2022, 9:38 AM
dachary added a comment to T3533: Winery backend: basic implementation.

Nevermind. I just realized that https://forge.softwareheritage.org/D6796 was in the "Accepted" state and has been for almost two weeks. :facepalm:

Jan 10 2022, 9:32 AM · Object storage (RedHat collaboration)
dachary added a comment to T3533: Winery backend: basic implementation.

Gentle ping?

Jan 10 2022, 9:30 AM · Object storage (RedHat collaboration)

Jan 3 2022

dachary added a comment to T3533: Winery backend: basic implementation.

Happy new year ! Or, in French, "Bonne année deux mille vingt d'oeufs" (pun on words courtesy Easter-Eggs).

Jan 3 2022, 11:24 AM · Object storage (RedHat collaboration)

Dec 28 2021

dachary added a comment to D6796: winery: basic implementation of the backend.

Hum, it looks like I forgot to submit two replies a week ago or so, doing that now.

Dec 28 2021, 10:48 AM

Dec 27 2021

dachary added a comment to T3533: Winery backend: basic implementation.

Gentle ping?

Dec 27 2021, 1:46 PM · Object storage (RedHat collaboration)

Dec 22 2021

dachary added a comment to T3533: Winery backend: basic implementation.

I updated D6796 to add a high level description of the files, classes and logic. I also split the WineryObjStorage class into four classes in an attempt to clarify the separation between the read and the write logic. I will now refrain from doing anything else: it can be difficult on the reviewers to follow a moving target :-D

Dec 22 2021, 2:11 PM · Object storage (RedHat collaboration)
dachary updated the diff for D6796: winery: basic implementation of the backend.
  • Update misleading comment in the check function
Dec 22 2021, 2:08 PM
dachary updated the diff for D6796: winery: basic implementation of the backend.
  • add implementation notes to the documentation
Dec 22 2021, 2:00 PM
dachary updated the diff for D6796: winery: basic implementation of the backend.
  • Reworked the Winery class hierarchy to separate reader and writer
Dec 22 2021, 12:11 PM
dachary updated the diff for D6796: winery: basic implementation of the backend.

s/LOGGER/logger/

Dec 22 2021, 1:16 AM
dachary updated the diff for D6796: winery: basic implementation of the backend.

is actually necessary for jenkins to be happy. Not sure why it passes on my machine, that needs to be investigated.

Dec 22 2021, 1:10 AM
dachary updated subscribers of T3533: Winery backend: basic implementation.

@olasd @vlorentz @vsellier @ardumont I'd be happy to start polishing the implementation after your first wave of review (thanks a lot for the time you spent on this, it is much appreciated 👍 ). I'm however concerned that it might make your work difficult because of the magnitude of the patch. An alternative would be to merge this first implementation, as long as you feel confident it won't break anything when deployed. It should not because it won't be instantiated. That would allow me to propose patches that are smaller and easier to review for:

Dec 22 2021, 1:04 AM · Object storage (RedHat collaboration)
dachary updated the diff for D6796: winery: basic implementation of the backend.
  • address olasd & vlorentz comments
Dec 22 2021, 12:50 AM
dachary added inline comments to D6796: winery: basic implementation of the backend.
Dec 22 2021, 12:45 AM

Dec 21 2021

dachary added a comment to D6796: winery: basic implementation of the backend.

On the code, I've left a few inline comments.

Dec 21 2021, 11:20 PM

Dec 20 2021

dachary updated subscribers of T3533: Winery backend: basic implementation.

@douardda will you have time for reviewing this week? I stand ready to address your comments. I'll stop working on that for now so that it is not a moving target.

Dec 20 2021, 9:21 AM · Object storage (RedHat collaboration)

Dec 16 2021

dachary updated the diff for D6796: winery: basic implementation of the backend.
  • implement benchmarking
Dec 16 2021, 6:44 PM

Dec 15 2021

dachary committed rDOPHfb2e2c464209: Updated debian directory for version 0.1.1 (authored by dachary).
Updated debian directory for version 0.1.1
Dec 15 2021, 3:36 PM
dachary committed rDOPH20e0379449a4: Import upstream version 0.1.1 (authored by dachary).
Import upstream version 0.1.1
Dec 15 2021, 3:36 PM
dachary committed rDOPH5c8f49adc225: pristine-tar data for swh-perfecthash_0.1.1.orig.tar.gz (authored by dachary).
pristine-tar data for swh-perfecthash_0.1.1.orig.tar.gz
Dec 15 2021, 3:36 PM

Dec 14 2021

dachary updated the diff for D6796: winery: basic implementation of the backend.
  • avoid race condition when two processes try to create a table
Dec 14 2021, 6:12 PM
dachary closed D6837: allocate 5012 as the winery service port.
Dec 14 2021, 5:38 PM
dachary committed rDDOC5ecdc4cf7397: allocate 5012 as the winery service port (authored by dachary).
allocate 5012 as the winery service port
Dec 14 2021, 5:38 PM
dachary added a comment to D6834: docker: add the swh-winery-db and swh-winery services.

Ah wait I got it, docker/services/swh-winery/entrypoint.sh launches winery itself, not an actual objstorage backend; you're only reusing the scafholding.

Dec 14 2021, 5:35 PM · Object storage
dachary added a revision to T3533: Winery backend: basic implementation: D6837: allocate 5012 as the winery service port.
Dec 14 2021, 5:34 PM · Object storage (RedHat collaboration)
dachary closed D6836: codespell: cmak is not a spelling mistake.
Dec 14 2021, 4:50 PM
dachary committed rDENV7bce5ba9bb65: codespell: cmak is not a spelling mistake (authored by dachary).
codespell: cmak is not a spelling mistake
Dec 14 2021, 4:50 PM
dachary updated the diff for D6836: codespell: cmak is not a spelling mistake.

rebase

Dec 14 2021, 4:50 PM
dachary closed D6835: doc: transform markdown links into rst links.
Dec 14 2021, 4:21 PM
dachary committed rDENV8a0c1ebd264c: doc: transform markdown links into rst links (authored by dachary).
doc: transform markdown links into rst links
Dec 14 2021, 4:21 PM
dachary added a comment to D6835: doc: transform markdown links into rst links.

And thanks ;)

Dec 14 2021, 4:20 PM
dachary added a comment to D6834: docker: add the swh-winery-db and swh-winery services.

Instead of defining a new service, could you provide an alternative docker-compose config file? This way, it can be used as to switch all services to use it, just by adding a CLI parameter. eg. we do this to replace the postgres storage backend with cassandra: https://docs.softwareheritage.org/devel/getting-started/using-docker.html#cassandra

Dec 14 2021, 4:17 PM · Object storage
dachary added a comment to D6835: doc: transform markdown links into rst links.

lgtm

iirc, you can replace all your __ by _ at the end of the link.

Dec 14 2021, 4:09 PM
dachary requested review of D6836: codespell: cmak is not a spelling mistake.
Dec 14 2021, 4:04 PM
dachary updated the diff for D6835: doc: transform markdown links into rst links.

remove extra space added by mistake

Dec 14 2021, 4:01 PM
dachary requested review of D6835: doc: transform markdown links into rst links.
Dec 14 2021, 3:59 PM
dachary updated the test plan for D6834: docker: add the swh-winery-db and swh-winery services.
Dec 14 2021, 3:54 PM · Object storage
dachary updated the summary of D6834: docker: add the swh-winery-db and swh-winery services.
Dec 14 2021, 3:53 PM · Object storage
dachary added a comment to D6834: docker: add the swh-winery-db and swh-winery services.

It depends on https://forge.softwareheritage.org/D6796 and will fail until it is merged. It can be tested from sources with an override like this:

Dec 14 2021, 3:52 PM · Object storage
dachary added a project to D6834: docker: add the swh-winery-db and swh-winery services: Object storage.
Dec 14 2021, 3:49 PM · Object storage
dachary added a task to D6834: docker: add the swh-winery-db and swh-winery services: T3533: Winery backend: basic implementation.
Dec 14 2021, 3:49 PM · Object storage
dachary added a revision to T3533: Winery backend: basic implementation: D6834: docker: add the swh-winery-db and swh-winery services.
Dec 14 2021, 3:49 PM · Object storage (RedHat collaboration)