@douardda gentle ping?

The previous jenkins run shows the line was not covered by a test

I pushed https://forge.softwareheritage.org/D6700 for inclusion in the documentation and from the output of

For the record, the documentation is disabled for now (see 1 and 2). I'll try to get it to build.

rebase

Alternative hardware bill of material:

clang-format C code

Anyway, there's a couple of issues I can think of:...

Could you document the expected shard layout in RestructuredText format? This code is setting the (current version of the) layout "forever", so we should make sure to properly document it.

move to _hash_cffi swh.perfecthash._hash_cffi

@ardumont I revisited all comments and noticed I missed a few, sorry about that. They should be good now :-) Is there anything else you need me to do for this proposed patch? Or should I just be patient and wait for whatever comes next in the review merge process? Thanks for your guidance!

add two missing type annotations

adjust lookup speed results and expectations according to today's benchmark results

comment on hardcoded value

Thanks a lot for the reviews!

address ardrumont comments

fix documentation bugs

document the benchmark process

But then Fed4Fire killed the experiment prematurely, ignoring the Grid5000 extension. I'll have another go at it.

When trying to extend the duration of an experiment (slice in the Fed4Fire parlance), an error occurs.

The CLI is actually more complicated because it requires input that are difficult to figure out:

@olasd thanks for adding the dependencies 🎉 D6545

The https://jfed.ilabt.imec.be/downloads/ CLI may be easier to use than the graphical client when repeatin experiments.

googletest is also needed

run the valgrind tests

fix typo in the commit comment

It's at https://forge.softwareheritage.org/source/swh-jenkins-dockerfiles/browse/master/base-buster/Dockerfile$39

Contribution to the Grid5000 documentation for Fed4Fire.

Using Export As ansible, I unzipped the result.

Created another experiment (with 25min lifetime only) and it's going better:

Linked Grid5000 & Fed4Fire accounts.

I wanted to terminate the experiment but it looks like it must expire (although Grid5000 has the option to terminate a job).

The Grid5000 machines are found in the "

https://www.grid5000.fr/w/Fed4FIRE is the better documentation to use Grid5000 via Fed4Fire

@olasd I'd like to add dependencies to the CI job running swh-perfecthash (valgrind) so that it can verify the C implementation is clean. Would you be so kind as to point me in the right direction? I looked in https://forge.softwareheritage.org/source/swh-jenkins-jobs but could not find the keyword cmph and concluded it must be in another repository.

There is a monitor that shows all testbeds, among which is grid5000:

Followed the tutorial to run a first experiment which failed https://doc.fed4fire.eu/firstexperiment.html

Followed the tutorial https://doc.fed4fire.eu/tools.html

@ardumont I believe all your comments were addressed. It is lucky that you did not review the actual logic: while running benchmarks I ran into a bug that lead to a significant refactor (using fread/fwrite instead of mmap and addresses). Not to the point where you'll have to re-read everything from scratch though 😅

using mmap turned out to be more involved than expected, use
read/write instead

Thanks a lot for this review @ardumont & @douardda ! I'll ping you for review when:

the return value of mmap is -1 on error, not NULL

There is an error on mmap which was not detected, therefore no information on why it failed. This was fixed.

• time tox -e py3 -- --basetemp=/mnt/pytest -s --shard-size $((100 * 1024)) --object-max-size $((4 * 1024)) -k test_build_speed number of objects = 45973118 baseline 163.73826217651367, write_duration 300.58917450904846, build_duration 26.01908826828003, total_duration 326.6082627773285

return on error if the write method exceeds the file capacity

Running benchmarks directly on grid5000

• oarsub -I -l "{cluster='dahu'}/host=1,walltime=1" -t deploy
• kadeploy3 -f $OAR_NODE_FILE -e debian11-x64-base -k
• ssh root@$(tail -1 $OAR_NODE_FILE)
• mkfs.ext4 /dev/sdb1
• mount /dev/sdb1 /mnt
• apt-get install -y python3-venv libcmph-dev gcc git
• git clone https://git.easter-eggs.org/biceps/swh-perfecthash/
• python3 -m venv bench
• source bench/bin/activate
• pip install -r requirements.txt -r requirements-test.txt
• cd swh-perfecthash
• tox -e py3
• time tox -e py3 -- --basetemp=/mnt/pytest -s --shard-size $((100 * 1024)) --object-max-size $((100 * 1024)) -k test_build_speed
• rm -fr /mnt/pytest

/opt/jFed/jFed-Experimenter works but I'll have to wait on the approval of the account before proceeding further.

Created a project in https://portal.fed4fire.eu/ with the intention of using grid5000. It is pending approval from an administrator (see T3670).

This is a duplicate of T3634

The draft implementation is available publicly at https://git.easter-eggs.org/biceps/swh-perfecthash/-/tree/wip-hash while D6424 is under review.

The implementation of the benchmarks is prepared at:

For the record I created a "draft" repository for contributions to https://forge.softwareheritage.org/source/swh-perfecthash/ at https://git.easter-eggs.org/biceps/swh-perfecthash. It is only meant to be a publicly available sandbox.

create and lookup a Read Shard with a perfect hash

@ardumont @olasd this is ready for review... I think :-D

create and lookup a Read Shard with a perfect hash

This is a working C implementation with tests and decent code coverage (all but error conditions). The run of the tests is valgrind clean. Next step is to run the python tests. Then wrap up and document so it can be reviewed.