Page MenuHomeSoftware Heritage

cassandra - benchmark the vault
Closed, ResolvedPublic

Description

As the other services, the vault needs to be tested to check if everything is ok

Event Timeline

vsellier changed the task status from Open to Work in Progress.Oct 21 2021, 5:43 PM
vsellier triaged this task as Normal priority.
vsellier created this task.

The deployment scripts to deploy the vault and the associated components have to be adapted to be deployed on the grid5000 cluster. work in progress...

vsellier closed this task as Resolved.EditedOct 22 2021, 11:48 AM
vsellier moved this task from Backlog to done on the System administration board.

EDITED: removed, commented on the wrong taks

vsellier reopened this task as Work in Progress.Oct 22 2021, 11:49 AM
vsellier moved this task from done to in-progress on the System administration board.

arf wrong issue, restoring statuses

When I tried to generate cook some vault in productions, several error where logged.
Some on missing commit, for example:

missing commit 338fb3eff7e93e121e7fe347b391a100ed0003c5
missing commit 73bc01d26ae7d61a74108b620e32458ef30c75c6
missing commit 78272793666b7354f6be373a73aa62426116d884
missing commit 83f434d5f256886fcfcfc354c3a63b91a71fce63

All these hash are identified in the work on T75 so they are not blockers

There is also a lot of error relative to the content retrieval, for example for this pretty big repository https://github.com/Azure/sonic-mgmt:

Nov 01 19:02:31 worker12 python3[3135858]: [2021-11-01 19:02:31,430: ERROR/ForkPoolWorker-47] Unexpected errors from git-fsck after co
                                           dangling commit 2d9f7117a24a9628935a6e4229e34427318125c7
                                           dangling commit 2f67b4fcf698a2c5423a588bb3e57f687c691870
                                           dangling commit 32544dee88d8428aa7a1d621dd368bf3a4ac8d48
...
                                           dangling tree 00d2aab8eccfe79c2f17f4eab0fba13adae826e8
                                           dangling tree 032cfa076aec13560361bb31dfd36b3c2e44415c
                                           dangling tree 0e9e9724f63d5750b232d36e31ac026d1c2aa5fc
...
                                           error: 027652dcf3fe817463bc717765a9501624c04ba8: object corrupt or missing: ./objects/02/7652dcf3fe817463bc717765a9501624c04ba8
                                           error: 1a0fdcd2a2b9ff97077cfc9f4168e090fd6e1e30: object corrupt or missing: ./objects/1a/0fdcd2a2b9ff97077cfc9f4168e090fd6e1e30
                                           error: 41afa35ce9f600622b9e1e6f329c84ffa620d82b: object corrupt or missing: ./objects/41/afa35ce9f600622b9e1e6f329c84ffa620d82b
                                           error: 459b8928a9530317aa786b4c52303af14a770fdb: object corrupt or missing: ./objects/45/9b8928a9530317aa786b4c52303af14a770fdb
                                           error: 54817fa1753f50d0c83579089693dc5751a2d266: object corrupt or missing: ./objects/54/817fa1753f50d0c83579089693dc5751a2d266
                                           error: 5b6554d4cc1bbd12da6c88736c41a370caefbf80: object corrupt or missing: ./objects/5b/6554d4cc1bbd12da6c88736c41a370caefbf80
                                           error: 5bf3acc4bc87224d95f6563c60cb143116d90d27: object corrupt or missing: ./objects/5b/f3acc4bc87224d95f6563c60cb143116d90d27
                                           error: 5d1b7a0df9fce5d86b07333470b147bf1ac07d28: object corrupt or missing: ./objects/5d/1b7a0df9fce5d86b07333470b147bf1ac07d28
                                           error: 6b224f4c4adce562ce68483ae05a080ae5374858: object corrupt or missing: ./objects/6b/224f4c4adce562ce68483ae05a080ae5374858
                                           error: 6c90e3ba21859ff45ea8819b0ae90ef6eb93747d: object corrupt or missing: ./objects/6c/90e3ba21859ff45ea8819b0ae90ef6eb93747d
                                           error: 702b8b880a287f6c248c032fb2dc5c604dfc25ec: object corrupt or missing: ./objects/70/2b8b880a287f6c248c032fb2dc5c604dfc25ec
                                           error: 76721387b06c098f2fcb8ef164d07c121cb18037: object corrupt or missing: ./objects/76/721387b06c098f2fcb8ef164d07c121cb18037
                                           error: 79b3a102c9ff4f871a2fd4274550c785a7d0e205: object corrupt or missing: ./objects/79/b3a102c9ff4f871a2fd4274550c785a7d0e205
                                           error: 7d03efc485b2191feb1cf466a0d6abb4428529a3: object corrupt or missing: ./objects/7d/03efc485b2191feb1cf466a0d6abb4428529a3
                                           error: 8285a78fbb86b6dfa2de04aedae22c93fc8f15af: object corrupt or missing: ./objects/82/85a78fbb86b6dfa2de04aedae22c93fc8f15af
                                           error: 83f434d5f256886fcfcfc354c3a63b91a71fce63: object corrupt or missing: ./objects/83/f434d5f256886fcfcfc354c3a63b91a71fce63
                                           error: 8b988a1644054c5cdc3960ff688c2e555bae249d: object corrupt or missing: ./objects/8b/988a1644054c5cdc3960ff688c2e555bae249d
                                           error: 8eefbcac4a98fddc09381221023105249349eda7: object corrupt or missing: ./objects/8e/efbcac4a98fddc09381221023105249349eda7
                                           error: 8f2e6721d7a02ce9686b64edbcc8df59ba4b2acb: object corrupt or missing: ./objects/8f/2e6721d7a02ce9686b64edbcc8df59ba4b2acb
                                           error: 9f57e1fc819cef598c79002955398ac2ecf385a4: object corrupt or missing: ./objects/9f/57e1fc819cef598c79002955398ac2ecf385a4
                                           error: a02f18fd37719a297d66a5ac2315472db517bb90: object corrupt or missing: ./objects/a0/2f18fd37719a297d66a5ac2315472db517bb90
                                           error: a2898159d7de647efa13a176672b8ac5ac0af818: object corrupt or missing: ./objects/a2/898159d7de647efa13a176672b8ac5ac0af818
                                           error: a3c2a0f20aadc8b63f677918b96061787dfa3627: object corrupt or missing: ./objects/a3/c2a0f20aadc8b63f677918b96061787dfa3627
                                           error: ae589f1135b6b7c1da9aa639670a783ef63a2884: object corrupt or missing: ./objects/ae/589f1135b6b7c1da9aa639670a783ef63a2884
                                           error: b32286077362313b7fb493fd6465627f5ecc00a6: object corrupt or missing: ./objects/b3/2286077362313b7fb493fd6465627f5ecc00a6
                                           error: b541434add6eeefcb691a9c2e81a4bad9eab846c: object corrupt or missing: ./objects/b5/41434add6eeefcb691a9c2e81a4bad9eab846c
                                           error: bd864dabc7cc84c7a10ca063e8cc313726df1833: object corrupt or missing: ./objects/bd/864dabc7cc84c7a10ca063e8cc313726df1833
                                           error: c67d040960374014770fa52b0932da9e7e97b9f0: object corrupt or missing: ./objects/c6/7d040960374014770fa52b0932da9e7e97b9f0
                                           error: e2948819ae5871a0369d5811c19ab81d9ea5965b: object corrupt or missing: ./objects/e2/948819ae5871a0369d5811c19ab81d9ea5965b
                                           error: ec19e408bfc93b6f57008c55b2b558b046d308b6: object corrupt or missing: ./objects/ec/19e408bfc93b6f57008c55b2b558b046d308b6
                                           error: eee799556d3638dac331f665f14673efd9d6535e: object corrupt or missing: ./objects/ee/e799556d3638dac331f665f14673efd9d6535e
                                           error: f76612d0b268d933b03cf9ba451fc092aabce6c8: object corrupt or missing: ./objects/f7/6612d0b268d933b03cf9ba451fc092aabce6c8
                                           error: sha1 mismatch for ./objects/02/7652dcf3fe817463bc717765a9501624c04ba8 (expected 027652dcf3fe817463bc717765a9501624c04ba8)
                                           error: sha1 mismatch for ./objects/1a/0fdcd2a2b9ff97077cfc9f4168e090fd6e1e30 (expected 1a0fdcd2a2b9ff97077cfc9f4168e090fd6e1e30)
                                           error: sha1 mismatch for ./objects/41/afa35ce9f600622b9e1e6f329c84ffa620d82b (expected 41afa35ce9f600622b9e1e6f329c84ffa620d82b)
                                           error: sha1 mismatch for ./objects/45/9b8928a9530317aa786b4c52303af14a770fdb (expected 459b8928a9530317aa786b4c52303af14a770fdb)
                                           error: sha1 mismatch for ./objects/54/817fa1753f50d0c83579089693dc5751a2d266 (expected 54817fa1753f50d0c83579089693dc5751a2d266)
                                           error: sha1 mismatch for ./objects/5b/6554d4cc1bbd12da6c88736c41a370caefbf80 (expected 5b6554d4cc1bbd12da6c88736c41a370caefbf80)
                                           error: sha1 mismatch for ./objects/5b/f3acc4bc87224d95f6563c60cb143116d90d27 (expected 5bf3acc4bc87224d95f6563c60cb143116d90d27)
                                           error: sha1 mismatch for ./objects/5d/1b7a0df9fce5d86b07333470b147bf1ac07d28 (expected 5d1b7a0df9fce5d86b07333470b147bf1ac07d28)
                                           error: sha1 mismatch for ./objects/6b/224f4c4adce562ce68483ae05a080ae5374858 (expected 6b224f4c4adce562ce68483ae05a080ae5374858)
                                           error: sha1 mismatch for ./objects/6c/90e3ba21859ff45ea8819b0ae90ef6eb93747d (expected 6c90e3ba21859ff45ea8819b0ae90ef6eb93747d)
                                           error: sha1 mismatch for ./objects/70/2b8b880a287f6c248c032fb2dc5c604dfc25ec (expected 702b8b880a287f6c248c032fb2dc5c604dfc25ec)
                                           error: sha1 mismatch for ./objects/76/721387b06c098f2fcb8ef164d07c121cb18037 (expected 76721387b06c098f2fcb8ef164d07c121cb18037)
                                           error: sha1 mismatch for ./objects/79/b3a102c9ff4f871a2fd4274550c785a7d0e205 (expected 79b3a102c9ff4f871a2fd4274550c785a7d0e205)
                                           error: sha1 mismatch for ./objects/7d/03efc485b2191feb1cf466a0d6abb4428529a3 (expected 7d03efc485b2191feb1cf466a0d6abb4428529a3)
                                           error: sha1 mismatch for ./objects/82/85a78fbb86b6dfa2de04aedae22c93fc8f15af (expected 8285a78fbb86b6dfa2de04aedae22c93fc8f15af)
                                           error: sha1 mismatch for ./objects/83/f434d5f256886fcfcfc354c3a63b91a71fce63 (expected 83f434d5f256886fcfcfc354c3a63b91a71fce63)
                                           error: sha1 mismatch for ./objects/8b/988a1644054c5cdc3960ff688c2e555bae249d (expected 8b988a1644054c5cdc3960ff688c2e555bae249d)
                                           error: sha1 mismatch for ./objects/8e/efbcac4a98fddc09381221023105249349eda7 (expected 8eefbcac4a98fddc09381221023105249349eda7)
                                           error: sha1 mismatch for ./objects/8f/2e6721d7a02ce9686b64edbcc8df59ba4b2acb (expected 8f2e6721d7a02ce9686b64edbcc8df59ba4b2acb)
                                           error: sha1 mismatch for ./objects/9f/57e1fc819cef598c79002955398ac2ecf385a4 (expected 9f57e1fc819cef598c79002955398ac2ecf385a4)
                                           error: sha1 mismatch for ./objects/a0/2f18fd37719a297d66a5ac2315472db517bb90 (expected a02f18fd37719a297d66a5ac2315472db517bb90)
                                           error: sha1 mismatch for ./objects/a2/898159d7de647efa13a176672b8ac5ac0af818 (expected a2898159d7de647efa13a176672b8ac5ac0af818)
                                           error: sha1 mismatch for ./objects/a3/c2a0f20aadc8b63f677918b96061787dfa3627 (expected a3c2a0f20aadc8b63f677918b96061787dfa3627)
                                           error: sha1 mismatch for ./objects/ae/589f1135b6b7c1da9aa639670a783ef63a2884 (expected ae589f1135b6b7c1da9aa639670a783ef63a2884)
                                           error: sha1 mismatch for ./objects/b3/2286077362313b7fb493fd6465627f5ecc00a6 (expected b32286077362313b7fb493fd6465627f5ecc00a6)
                                           error: sha1 mismatch for ./objects/b5/41434add6eeefcb691a9c2e81a4bad9eab846c (expected b541434add6eeefcb691a9c2e81a4bad9eab846c)
                                           error: sha1 mismatch for ./objects/bd/864dabc7cc84c7a10ca063e8cc313726df1833 (expected bd864dabc7cc84c7a10ca063e8cc313726df1833)
                                           error: sha1 mismatch for ./objects/c6/7d040960374014770fa52b0932da9e7e97b9f0 (expected c67d040960374014770fa52b0932da9e7e97b9f0)
                                           error: sha1 mismatch for ./objects/e2/948819ae5871a0369d5811c19ab81d9ea5965b (expected e2948819ae5871a0369d5811c19ab81d9ea5965b)
                                           error: sha1 mismatch for ./objects/ec/19e408bfc93b6f57008c55b2b558b046d308b6 (expected ec19e408bfc93b6f57008c55b2b558b046d308b6)
                                           error: sha1 mismatch for ./objects/ee/e799556d3638dac331f665f14673efd9d6535e (expected eee799556d3638dac331f665f14673efd9d6535e)
                                           error: sha1 mismatch for ./objects/f7/6612d0b268d933b03cf9ba451fc092aabce6c8 (expected f76612d0b268d933b03cf9ba451fc092aabce6c8)
                                           missing commit 83f434d5f256886fcfcfc354c3a63b91a71fce63

The corrupted / missing contents are also identified in T75 so these errors can be ignored

After some tweaks[1], the git bare cooking is finally working correctly on g5k.

These are the results of the cooking of some random repositories:

repoprod flatcassadra flatprod git_barecassandra git_bare
https://github.com/Azure/sonic-mgmt (3575 commits)44943514
https://github.com/Upsham/upsham.github.io (25 commits)22.0128
https://github.com/gaasedelen/lighthouse (539 commits)18.9219.94
https://github.com/bshaffer/oauth2-server-php (935 commits)23.722.78
https://github.com/isandlaTech/cohorte-utilities (677 commits)1361407.311071
https://github.com/nvaccess/nvda (20215 commits)1662002156413416
https://github.com/openfisca/openfisca-paris (268 commits)19.2817216269
https://github.com/rt-2pm2/cyphy_testbed (581 commits)3951565760
https://github.com/PranavShirole/pranavshirole.github.io (692 commits)4877681356
https://github.com/pi-hole/docker-pi-hole (1155 commits)108455440
https://github.com/reitermarkus/brew (24000 commits)2536202098213782
https://github.com/alphagov/notifications-template-preview (947 commits)1024377558

It seems the tar archive generation is globally slower with cassandra, for the git bare cooking, it's difficult to generalize, but it looks faster for big repositories

[1]

  • in the loader configuration, remove the filter
  • in the storage, change the allowOverwrite to True
  • run a loading of the repository
  • use an objstorage querying azure
vsellier moved this task from in-progress to done on the System administration board.

@vsellier Did you have swh-graph enabled in the vault configuration for these benchmarks?