- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Dec 1 2021
@douardda gentle ping?
Nov 29 2021
The previous jenkins run shows the line was not covered by a test
I pushed https://forge.softwareheritage.org/D6700 for inclusion in the documentation and from the output of
Nov 22 2021
Nov 18 2021
rebase
Nov 15 2021
Alternative hardware bill of material:
Nov 10 2021
clang-format C code
Nov 9 2021
Anyway, there's a couple of issues I can think of:...
Could you document the expected shard layout in RestructuredText format? This code is setting the (current version of the) layout "forever", so we should make sure to properly document it.
Nov 4 2021
move to _hash_cffi swh.perfecthash._hash_cffi
Nov 3 2021
Nov 2 2021
@ardumont I revisited all comments and noticed I missed a few, sorry about that. They should be good now :-) Is there anything else you need me to do for this proposed patch? Or should I just be patient and wait for whatever comes next in the review merge process? Thanks for your guidance!
add two missing type annotations
Oct 26 2021
adjust lookup speed results and expectations according to today's benchmark results
comment on hardcoded value
Thanks a lot for the reviews!
address ardrumont comments
fix documentation bugs
document the benchmark process
But then Fed4Fire killed the experiment prematurely, ignoring the Grid5000 extension. I'll have another go at it.
When trying to extend the duration of an experiment (slice in the Fed4Fire parlance), an error occurs.
The CLI is actually more complicated because it requires input that are difficult to figure out:
Oct 25 2021
The https://jfed.ilabt.imec.be/downloads/ CLI may be easier to use than the graphical client when repeatin experiments.
googletest is also needed
run the valgrind tests
fix typo in the commit comment
Oct 21 2021
Contribution to the Grid5000 documentation for Fed4Fire.
Using Export As ansible, I unzipped the result.
Created another experiment (with 25min lifetime only) and it's going better:
I wanted to terminate the experiment but it looks like it must expire (although Grid5000 has the option to terminate a job).
The Grid5000 machines are found in the "
https://www.grid5000.fr/w/Fed4FIRE is the better documentation to use Grid5000 via Fed4Fire
@olasd I'd like to add dependencies to the CI job running swh-perfecthash (valgrind) so that it can verify the C implementation is clean. Would you be so kind as to point me in the right direction? I looked in https://forge.softwareheritage.org/source/swh-jenkins-jobs but could not find the keyword cmph and concluded it must be in another repository.
There is a monitor that shows all testbeds, among which is grid5000:
Followed the tutorial to run a first experiment which failed https://doc.fed4fire.eu/firstexperiment.html
Followed the tutorial https://doc.fed4fire.eu/tools.html
$ time tox -e py3 -- --basetemp=/mnt/pytest -s --shard-size $((100 * 1024)) --object-max-size $((4 * 1024)) -k test_build_speed number of objects = 45973694, total size = 105903024192 baseline 165.74853587150574, write_duration 495.07564210891724, build_duration 24.210500478744507, total_duration 519.2861425876617
@ardumont I believe all your comments were addressed. It is lucky that you did not review the actual logic: while running benchmarks I ran into a bug that lead to a significant refactor (using fread/fwrite instead of mmap and addresses). Not to the point where you'll have to re-read everything from scratch though 😅
using mmap turned out to be more involved than expected, use
read/write instead
Oct 20 2021
Oct 19 2021
the return value of mmap is -1 on error, not NULL
There is an error on mmap which was not detected, therefore no information on why it failed. This was fixed.
- time tox -e py3 -- --basetemp=/mnt/pytest -s --shard-size $((100 * 1024)) --object-max-size $((4 * 1024)) -k test_build_speed number of objects = 45973118 baseline 163.73826217651367, write_duration 300.58917450904846, build_duration 26.01908826828003, total_duration 326.6082627773285
return on error if the write method exceeds the file capacity
Running benchmarks directly on grid5000
- oarsub -I -l "{cluster='dahu'}/host=1,walltime=1" -t deploy
- kadeploy3 -f $OAR_NODE_FILE -e debian11-x64-base -k
- ssh root@$(tail -1 $OAR_NODE_FILE)
- mkfs.ext4 /dev/sdb1
- mount /dev/sdb1 /mnt
- apt-get install -y python3-venv libcmph-dev gcc git
- git clone https://git.easter-eggs.org/biceps/swh-perfecthash/
- python3 -m venv bench
- source bench/bin/activate
- pip install -r requirements.txt -r requirements-test.txt
- cd swh-perfecthash
- tox -e py3
- time tox -e py3 -- --basetemp=/mnt/pytest -s --shard-size $((100 * 1024)) --object-max-size $((100 * 1024)) -k test_build_speed
- rm -fr /mnt/pytest
/opt/jFed/jFed-Experimenter works but I'll have to wait on the approval of the account before proceeding further.
Created a project in https://portal.fed4fire.eu/ with the intention of using grid5000. It is pending approval from an administrator (see T3670).
Oct 18 2021
This is a duplicate of T3634
The draft implementation is available publicly at https://git.easter-eggs.org/biceps/swh-perfecthash/-/tree/wip-hash while D6424 is under review.
The implementation of the benchmarks is prepared at:
For the record I created a "draft" repository for contributions to https://forge.softwareheritage.org/source/swh-perfecthash/ at https://git.easter-eggs.org/biceps/swh-perfecthash. It is only meant to be a publicly available sandbox.
Oct 12 2021
create and lookup a Read Shard with a perfect hash
create and lookup a Read Shard with a perfect hash
This is a working C implementation with tests and decent code coverage (all but error conditions). The run of the tests is valgrind clean. Next step is to run the python tests. Then wrap up and document so it can be reviewed.
working implementation, with tests
Oct 11 2021
compiles but crashes
In D6424#167442, @dachary wrote:skeleton
In D6424#166886, @ardumont wrote:@dachary It'd be nice if you could describe what this is about in the commit message and
the diff description (if you actually provide a commit description, then when you create
the diff, the commit message is used as a description bootstrap). I know it's more work
for you but it happens that:
- it helps the reviewers to have some context directly here (without having to follow
between a multitude of tasks. FYI, I've followed through the task but it's not enough,
i need to also dig in that arborescence of tasks).
- is also how we are doing that in every other modules ;)
- the curious could learn a thing or 2 even if they don't do a proper review.
Please and thanks in advance.
Cheers,
Oct 8 2021
I'm curious to know where the libcmph-dev dependency was added?