Page MenuHomeSoftware Heritage
Feed All Stories

Oct 19 2021

aeviso updated the diff for D6339: Add support for remote backend on existing storage tests.

rebase

Oct 19 2021, 4:32 PM
Harbormaster failed to build B24520: rDWAPPS6c2792e9090f: assets/admin/origin-save: Enable to add note when rejecting a request for rDWAPPS6c2792e9090f: assets/admin/origin-save: Enable to add note when rejecting a request!
Oct 19 2021, 4:30 PM
aeviso updated the diff for D6165: Add new RabbitMQ-based client/server API.
  • Fix config file parsing for server initilization
  • Send several items per message in the remote provenance storage
  • Export batch size and prefetch count as parameters for remote storage
Oct 19 2021, 4:30 PM
aeviso updated the diff for D6473: Improve PostgreSQL storage scheme for the `with-path-denormalized` flavor.

rebase

Oct 19 2021, 4:29 PM
aeviso requested review of D6507: Add metrics on retries when flushing cache on the provenance backend.
Oct 19 2021, 4:27 PM
aeviso requested review of D6506: Export page size as a parameter for postgresql storage.
Oct 19 2021, 4:26 PM
anlambert closed D6502: assets/origin-save: Display rejection reason in request info popover.
Oct 19 2021, 4:12 PM
anlambert committed rDWAPPS96b3d287ed50: assets/origin-save: Display rejection reason in request info popover (authored by anlambert).
assets/origin-save: Display rejection reason in request info popover
Oct 19 2021, 4:12 PM
anlambert committed rDWAPPS6c2792e9090f: assets/admin/origin-save: Enable to add note when rejecting a request (authored by anlambert).
assets/admin/origin-save: Enable to add note when rejecting a request
Oct 19 2021, 4:12 PM
anlambert closed D6501: assets/admin/origin-save: Enable to add note when rejecting a request.
Oct 19 2021, 4:12 PM
anlambert committed rDWAPPS9cf83ddceafe: Makefile.local: Remove correct sqlite tests database (authored by anlambert).
Makefile.local: Remove correct sqlite tests database
Oct 19 2021, 4:12 PM
swh-public-ci added a comment to D6501: assets/admin/origin-save: Enable to add note when rejecting a request.

Build is green

Oct 19 2021, 4:10 PM
swh-public-ci added a comment to D6502: assets/origin-save: Display rejection reason in request info popover.

Build is green

Oct 19 2021, 4:10 PM
anlambert updated the diff for D6502: assets/origin-save: Display rejection reason in request info popover.

Rebase

Oct 19 2021, 3:54 PM
anlambert updated the diff for D6501: assets/admin/origin-save: Enable to add note when rejecting a request.

Rebase and fix a couple of typos in cypress test comments

Oct 19 2021, 3:53 PM
anlambert closed D6503: api/private/deposit_list: Allow to filter deposits by client username.
Oct 19 2021, 3:52 PM
anlambert committed rDDEP801a0274c44d: api/private/deposit_list: Allow to filter deposits by client username (authored by anlambert).
api/private/deposit_list: Allow to filter deposits by client username
Oct 19 2021, 3:52 PM
ardumont retitled D6395: lister: Add new maven lister from maven-lister: Add maven lister with review D6133. to maven-lister: Add maven lister with review.
Oct 19 2021, 3:50 PM
ardumont added a comment to D6395: lister: Add new maven lister.

Hello, I wanted to have a look at your diff but then I saw old commits from me in there.
So, you need to rebase your work on latest master and then properly diff with the right
range of commits. I gather something like this would do:

Oct 19 2021, 3:49 PM
anlambert added a comment to D6502: assets/origin-save: Display rejection reason in request info popover.

lgtm

Oct 19 2021, 3:48 PM
ardumont accepted D6502: assets/origin-save: Display rejection reason in request info popover.

lgtm

Oct 19 2021, 3:44 PM
ardumont accepted D6501: assets/admin/origin-save: Enable to add note when rejecting a request.

Great!

Oct 19 2021, 3:42 PM
vlorentz closed D6494: cassandra: Fix incomplete check of content existence in object_find_by_sha1_git.
Oct 19 2021, 3:21 PM
vlorentz committed rDSTOa98671047710: cassandra: Fix incomplete check of content existence in object_find_by_sha1_git (authored by vlorentz).
cassandra: Fix incomplete check of content existence in object_find_by_sha1_git
Oct 19 2021, 3:21 PM
zack committed rMSLD05f893596a65: talk IDIA: last bits (authored by zack).
talk IDIA: last bits
Oct 19 2021, 3:12 PM
ardumont accepted D6504: converters: Fix detection of tree entries with non-standard commit/tree mode..

What i understood of this improvment is that only the detection logic of {tree, commit} got improved.
This keeps whatever is ingested with their "non-standard" mode.

Oct 19 2021, 2:57 PM
ardumont added 1 blocking reviewer(s) for D6504: converters: Fix detection of tree entries with non-standard commit/tree mode.: olasd.
Oct 19 2021, 2:56 PM
ardumont added a comment to T1537: semi-automated addition of new "forges".
  • (1) add an entry to the list of "forges" being listed.
  • (2) immediately do the full listing, with high scheduling priority
Oct 19 2021, 2:47 PM · Scheduling utilities, Web app
anlambert added a comment to D6503: api/private/deposit_list: Allow to filter deposits by client username.

That was quick ;)
Thanks.

Oct 19 2021, 2:41 PM
ardumont accepted D6503: api/private/deposit_list: Allow to filter deposits by client username.

That was quick ;)
Thanks.

Oct 19 2021, 2:29 PM
anlambert closed D6505: browse/content: Fix error when a staff user browses a not found content.
Oct 19 2021, 2:27 PM
anlambert committed rDWAPPSb72fd7f91746: browse/content: Fix error when a staff user browses a not found content (authored by anlambert).
browse/content: Fix error when a staff user browses a not found content
Oct 19 2021, 2:27 PM
vlorentz accepted D6505: browse/content: Fix error when a staff user browses a not found content.
Oct 19 2021, 2:26 PM
anlambert requested review of D6505: browse/content: Fix error when a staff user browses a not found content.
Oct 19 2021, 2:20 PM
aeviso committed R230:0b732cfa737f: Improve statistics logic (authored by aeviso).
Improve statistics logic
Oct 19 2021, 2:08 PM
aeviso committed R230:2ca8e31bebaf: Move client/server config to a file (authored by aeviso).
Move client/server config to a file
Oct 19 2021, 2:08 PM
aeviso committed R230:ed4824526e80: Update revision client/server scripts (authored by aeviso).
Update revision client/server scripts
Oct 19 2021, 2:08 PM
aeviso committed R230:50ed46d1d81c: Make find first/all scripts backend agnostic (authored by aeviso).
Make find first/all scripts backend agnostic
Oct 19 2021, 2:08 PM
aeviso committed R230:128e4a53b5aa: Update auxiliar scripts and make the compliant with black and isort (authored by aeviso).
Update auxiliar scripts and make the compliant with black and isort
Oct 19 2021, 2:08 PM
vlorentz requested review of D6504: converters: Fix detection of tree entries with non-standard commit/tree mode..
Oct 19 2021, 2:02 PM
ardumont updated the task description for T1538: Add "forge" now.
Oct 19 2021, 2:01 PM · Add Forge Now , Roadmap 2022, meta-task, Roadmap 2021
anlambert requested review of D6503: api/private/deposit_list: Allow to filter deposits by client username.
Oct 19 2021, 1:51 PM
anlambert added a revision to T3174: Filter deposit-admin view by deposit client on admin (moderation) page: D6503: api/private/deposit_list: Allow to filter deposits by client username.
Oct 19 2021, 1:48 PM · Monitoring, SWORD deposit, Web app
anlambert updated the summary of D6502: assets/origin-save: Display rejection reason in request info popover.
Oct 19 2021, 12:33 PM
anlambert updated the summary of D6501: assets/admin/origin-save: Enable to add note when rejecting a request.
Oct 19 2021, 12:32 PM
anlambert requested review of D6502: assets/origin-save: Display rejection reason in request info popover.
Oct 19 2021, 12:30 PM
anlambert requested review of D6501: assets/admin/origin-save: Enable to add note when rejecting a request.
Oct 19 2021, 12:28 PM
moranegg triaged T3673: Define and document workflow for "save requests" for tarballs as Normal priority.
Oct 19 2021, 12:15 PM · Ambassadors
moranegg changed the status of T3672: Create ambassadors community and onboard participants from Open to Work in Progress.
Oct 19 2021, 11:57 AM · Ambassadors
ardumont added inline comments to D6492: Add support for pathslicing in seaweedfs backend.
Oct 19 2021, 11:55 AM
douardda created P1203 (An Untitled Masterwork).
Oct 19 2021, 11:43 AM
ardumont added a comment to D6492: Add support for pathslicing in seaweedfs backend.

this actually requires quite some refactoring in the seaweedfs backend,
mostly to support the (somewhat useless) list_content() method.

Oct 19 2021, 11:40 AM
ardumont accepted D6494: cassandra: Fix incomplete check of content existence in object_find_by_sha1_git.

lgtm

Oct 19 2021, 11:38 AM
olasd added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

Sent a summary of this discussion to the swh-devel list for input:

Oct 19 2021, 11:36 AM · Git loader
anlambert added a revision to T3256: Propose reason for rejecting a save code now: D6502: assets/origin-save: Display rejection reason in request info popover.
Oct 19 2021, 11:21 AM · Save Code Now, Easy hack, Web app
ardumont retitled D6424: Perfect hashmap C implementation from C stub compiled via cffi and tested in python to Perfect hashmap C implementation.
Oct 19 2021, 11:20 AM
anlambert added a revision to T3256: Propose reason for rejecting a save code now: D6501: assets/admin/origin-save: Enable to add note when rejecting a request.
Oct 19 2021, 11:20 AM · Save Code Now, Easy hack, Web app
anlambert added a revision to T3257: Template the save code now rejection reason to ease operator workflow: D6501: assets/admin/origin-save: Enable to add note when rejecting a request.
Oct 19 2021, 11:20 AM · Save Code Now, Easy hack, Web app
ardumont added a comment to D6424: Perfect hashmap C implementation.

Thanks for the great work!

Oct 19 2021, 11:17 AM
ardumont added inline comments to D6424: Perfect hashmap C implementation.
Oct 19 2021, 11:11 AM
vsellier renamed T3671: staging - swh-search (metadata indexer) is unable to update a document due to an unparseable date from staging - swh-search unable to update a document due to an unparseable date to staging - swh-search (metadata indexer) is unable to update a document due to an unparseable date.
Oct 19 2021, 11:03 AM · Intrinsic metadata, Archive search
ardumont added inline comments to D6424: Perfect hashmap C implementation.
Oct 19 2021, 11:02 AM
vsellier updated the task description for T3671: staging - swh-search (metadata indexer) is unable to update a document due to an unparseable date.
Oct 19 2021, 10:54 AM · Intrinsic metadata, Archive search
vsellier triaged T3671: staging - swh-search (metadata indexer) is unable to update a document due to an unparseable date as Normal priority.
Oct 19 2021, 10:48 AM · Intrinsic metadata, Archive search
zack committed rMSLDcc87f86babff: academic adoption module: minor style and typo fixes (authored by zack).
academic adoption module: minor style and typo fixes
Oct 19 2021, 10:44 AM
zack committed rMSLD94c77a7be1a7: check in slides for talk at Telecom/IDIA PFDay (authored by zack).
check in slides for talk at Telecom/IDIA PFDay
Oct 19 2021, 10:44 AM
vsellier added a comment to T3630: staging - journal0 needs more space.

journal0 is stopped for the moment.
It's decommissioned in puppet:

root@pergamon:/usr/local/sbin# ./swh-puppet-master-decommission journal0.internal.staging.swh.network
+ puppet node deactivate journal0.internal.staging.swh.network
Submitted 'deactivate node' for journal0.internal.staging.swh.network with UUID 02929cbb-6b34-4fab-85c1-46c9397fa949
+ puppet node clean journal0.internal.staging.swh.network
Notice: Revoked certificate with serial 199
Notice: Removing file Puppet::SSL::Certificate journal0.internal.staging.swh.network at '/var/lib/puppet/ssl/ca/signed/journal0.internal.staging.swh.network.pem'
journal0.internal.staging.swh.network
+ puppet cert clean journal0.internal.staging.swh.network
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 199
+ systemctl restart apache2
Oct 19 2021, 10:44 AM · System administration
vsellier closed D6499: staging: decomission journal0 vm.
Oct 19 2021, 10:39 AM
vsellier committed rSPRE99db922ccb3d: staging: decomission journal0 vm (authored by vsellier).
staging: decomission journal0 vm
Oct 19 2021, 10:39 AM
ardumont added a comment to D6424: Perfect hashmap C implementation.

I've updated the description with one of your explanatory comment (feel free to adapt if i'm wrong ;).

Oct 19 2021, 10:37 AM
vsellier committed rDDOC8784d489ade6: staging: Migration of journal0 to storage1 (authored by vsellier).
staging: Migration of journal0 to storage1
Oct 19 2021, 10:37 AM
vsellier closed D6498: staging: Migration of journal0 to storage1.
Oct 19 2021, 10:37 AM
vsellier closed D6500: staging/kafka: deploy consumer monitoring in storage1.
Oct 19 2021, 10:31 AM
vsellier committed rSPSITEc9232bdcd9df: staging/kafka: deploy consumer monitoring in storage1 (authored by vsellier).
staging/kafka: deploy consumer monitoring in storage1
Oct 19 2021, 10:31 AM
ardumont accepted D6500: staging/kafka: deploy consumer monitoring in storage1.
Oct 19 2021, 10:30 AM
vsellier requested review of D6500: staging/kafka: deploy consumer monitoring in storage1.
Oct 19 2021, 10:24 AM
vsellier added a revision to T3630: staging - journal0 needs more space: D6500: staging/kafka: deploy consumer monitoring in storage1.
Oct 19 2021, 10:24 AM · System administration
ardumont updated the summary of D6424: Perfect hashmap C implementation.
Oct 19 2021, 10:22 AM
ardumont updated the summary of D6424: Perfect hashmap C implementation.
Oct 19 2021, 10:18 AM
ardumont added a project to T3668: Improve the seaweedfs backend: Object storage.
Oct 19 2021, 10:15 AM · Object storage
swh-public-ci added a comment to D6424: Perfect hashmap C implementation.

Build is green

Oct 19 2021, 9:25 AM
dachary updated the diff for D6424: Perfect hashmap C implementation.

the return value of mmap is -1 on error, not NULL

Oct 19 2021, 9:24 AM
dachary added a comment to T3521: Persistent readonly perfect hash table: benchmarks.

There is an error on mmap which was not detected, therefore no information on why it failed. This was fixed.

Oct 19 2021, 8:19 AM · Object storage (RedHat collaboration)
dachary added a comment to T3521: Persistent readonly perfect hash table: benchmarks.
  • time tox -e py3 -- --basetemp=/mnt/pytest -s --shard-size $((100 * 1024)) --object-max-size $((4 * 1024)) -k test_build_speed number of objects = 45973118 baseline 163.73826217651367, write_duration 300.58917450904846, build_duration 26.01908826828003, total_duration 326.6082627773285
Oct 19 2021, 5:36 AM · Object storage (RedHat collaboration)
swh-public-ci added a comment to D6424: Perfect hashmap C implementation.

Build is green

Oct 19 2021, 5:10 AM
dachary updated the diff for D6424: Perfect hashmap C implementation.

return on error if the write method exceeds the file capacity

Oct 19 2021, 5:09 AM
dachary added a comment to T3521: Persistent readonly perfect hash table: benchmarks.

Running benchmarks directly on grid5000

  • oarsub -I -l "{cluster='dahu'}/host=1,walltime=1" -t deploy
  • kadeploy3 -f $OAR_NODE_FILE -e debian11-x64-base -k
  • ssh root@$(tail -1 $OAR_NODE_FILE)
  • mkfs.ext4 /dev/sdb1
  • mount /dev/sdb1 /mnt
  • apt-get install -y python3-venv libcmph-dev gcc git
  • git clone https://git.easter-eggs.org/biceps/swh-perfecthash/
  • python3 -m venv bench
  • source bench/bin/activate
  • pip install -r requirements.txt -r requirements-test.txt
  • cd swh-perfecthash
  • tox -e py3
  • time tox -e py3 -- --basetemp=/mnt/pytest -s --shard-size $((100 * 1024)) --object-max-size $((100 * 1024)) -k test_build_speed
  • rm -fr /mnt/pytest
Oct 19 2021, 4:36 AM · Object storage (RedHat collaboration)
dachary added a comment to T3670: fed4fire setup for winery benchmarks.

/opt/jFed/jFed-Experimenter works but I'll have to wait on the approval of the account before proceeding further.

Oct 19 2021, 3:50 AM · Object storage
dachary updated the task description for T3670: fed4fire setup for winery benchmarks.
Oct 19 2021, 3:48 AM · Object storage
dachary added a parent task for T3670: fed4fire setup for winery benchmarks: T3432: Add winery backend.
Oct 19 2021, 3:44 AM · Object storage
dachary added a subtask for T3432: Add winery backend: T3670: fed4fire setup for winery benchmarks.
Oct 19 2021, 3:44 AM · Object storage
dachary changed the status of T3670: fed4fire setup for winery benchmarks from Open to Work in Progress.
Oct 19 2021, 3:44 AM · Object storage
dachary added a comment to T3521: Persistent readonly perfect hash table: benchmarks.

Created a project in https://portal.fed4fire.eu/ with the intention of using grid5000. It is pending approval from an administrator (see T3670).

Oct 19 2021, 3:11 AM · Object storage (RedHat collaboration)

Oct 18 2021

vsellier added a comment to T3630: staging - journal0 needs more space.

After a fight with zookeeper the new broker is nowfully functional on storage1.
It seems the zookeeper content was not synchronized between the 2 nodes. It was solved by stopping everything and copying the content of '/var/lib/zookeeper' to the second server. For example the credentials were not present on the second zookeeper.
After the battle, I realized I should have dig deeper to find the reason but I try to react fast to reduce the risk of corruption.

Oct 18 2021, 11:52 PM · System administration
dachary changed the status of T3521: Persistent readonly perfect hash table: benchmarks, a subtask of T3104: Persistent readonly perfect hash table, from Open to Work in Progress.
Oct 18 2021, 9:02 PM · Object storage (RedHat collaboration)
dachary changed the status of T3521: Persistent readonly perfect hash table: benchmarks from Open to Work in Progress.
Oct 18 2021, 9:02 PM · Object storage (RedHat collaboration)
dachary changed the status of T3520: Persistent readonly perfect hash table: implementation, a subtask of T3104: Persistent readonly perfect hash table, from Open to Work in Progress.
Oct 18 2021, 9:02 PM · Object storage (RedHat collaboration)
dachary changed the status of T3520: Persistent readonly perfect hash table: implementation from Open to Work in Progress.
Oct 18 2021, 9:02 PM · Object storage (RedHat collaboration)
dachary changed the status of T3104: Persistent readonly perfect hash table, a subtask of T3432: Add winery backend, from Open to Work in Progress.
Oct 18 2021, 9:02 PM · Object storage
dachary changed the status of T3104: Persistent readonly perfect hash table, a subtask of T3054: Scale out object storage design, from Open to Work in Progress.
Oct 18 2021, 9:02 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task