As the other services, the vault needs to be tested to check if everything is ok
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T2213 Storage | ||
| Migrated | gitlab-migration | T2214 Scale-out graph and database storage in production | ||
| Migrated | gitlab-migration | T1892 Cassandra as a storage backend | ||
| Migrated | gitlab-migration | T3357 Perform some tests of the cassandra storage on Grid5000 | ||
| Migrated | gitlab-migration | T3683 cassandra - benchmark the vault |
Event Timeline
Comment Actions
The deployment scripts to deploy the vault and the associated components have to be adapted to be deployed on the grid5000 cluster. work in progress...
Comment Actions
When I tried to generate cook some vault in productions, several error where logged.
Some on missing commit, for example:
missing commit 338fb3eff7e93e121e7fe347b391a100ed0003c5 missing commit 73bc01d26ae7d61a74108b620e32458ef30c75c6 missing commit 78272793666b7354f6be373a73aa62426116d884 missing commit 83f434d5f256886fcfcfc354c3a63b91a71fce63
All these hash are identified in the work on T75 so they are not blockers
There is also a lot of error relative to the content retrieval, for example for this pretty big repository https://github.com/Azure/sonic-mgmt:
Nov 01 19:02:31 worker12 python3[3135858]: [2021-11-01 19:02:31,430: ERROR/ForkPoolWorker-47] Unexpected errors from git-fsck after co
dangling commit 2d9f7117a24a9628935a6e4229e34427318125c7
dangling commit 2f67b4fcf698a2c5423a588bb3e57f687c691870
dangling commit 32544dee88d8428aa7a1d621dd368bf3a4ac8d48
...
dangling tree 00d2aab8eccfe79c2f17f4eab0fba13adae826e8
dangling tree 032cfa076aec13560361bb31dfd36b3c2e44415c
dangling tree 0e9e9724f63d5750b232d36e31ac026d1c2aa5fc
...
error: 027652dcf3fe817463bc717765a9501624c04ba8: object corrupt or missing: ./objects/02/7652dcf3fe817463bc717765a9501624c04ba8
error: 1a0fdcd2a2b9ff97077cfc9f4168e090fd6e1e30: object corrupt or missing: ./objects/1a/0fdcd2a2b9ff97077cfc9f4168e090fd6e1e30
error: 41afa35ce9f600622b9e1e6f329c84ffa620d82b: object corrupt or missing: ./objects/41/afa35ce9f600622b9e1e6f329c84ffa620d82b
error: 459b8928a9530317aa786b4c52303af14a770fdb: object corrupt or missing: ./objects/45/9b8928a9530317aa786b4c52303af14a770fdb
error: 54817fa1753f50d0c83579089693dc5751a2d266: object corrupt or missing: ./objects/54/817fa1753f50d0c83579089693dc5751a2d266
error: 5b6554d4cc1bbd12da6c88736c41a370caefbf80: object corrupt or missing: ./objects/5b/6554d4cc1bbd12da6c88736c41a370caefbf80
error: 5bf3acc4bc87224d95f6563c60cb143116d90d27: object corrupt or missing: ./objects/5b/f3acc4bc87224d95f6563c60cb143116d90d27
error: 5d1b7a0df9fce5d86b07333470b147bf1ac07d28: object corrupt or missing: ./objects/5d/1b7a0df9fce5d86b07333470b147bf1ac07d28
error: 6b224f4c4adce562ce68483ae05a080ae5374858: object corrupt or missing: ./objects/6b/224f4c4adce562ce68483ae05a080ae5374858
error: 6c90e3ba21859ff45ea8819b0ae90ef6eb93747d: object corrupt or missing: ./objects/6c/90e3ba21859ff45ea8819b0ae90ef6eb93747d
error: 702b8b880a287f6c248c032fb2dc5c604dfc25ec: object corrupt or missing: ./objects/70/2b8b880a287f6c248c032fb2dc5c604dfc25ec
error: 76721387b06c098f2fcb8ef164d07c121cb18037: object corrupt or missing: ./objects/76/721387b06c098f2fcb8ef164d07c121cb18037
error: 79b3a102c9ff4f871a2fd4274550c785a7d0e205: object corrupt or missing: ./objects/79/b3a102c9ff4f871a2fd4274550c785a7d0e205
error: 7d03efc485b2191feb1cf466a0d6abb4428529a3: object corrupt or missing: ./objects/7d/03efc485b2191feb1cf466a0d6abb4428529a3
error: 8285a78fbb86b6dfa2de04aedae22c93fc8f15af: object corrupt or missing: ./objects/82/85a78fbb86b6dfa2de04aedae22c93fc8f15af
error: 83f434d5f256886fcfcfc354c3a63b91a71fce63: object corrupt or missing: ./objects/83/f434d5f256886fcfcfc354c3a63b91a71fce63
error: 8b988a1644054c5cdc3960ff688c2e555bae249d: object corrupt or missing: ./objects/8b/988a1644054c5cdc3960ff688c2e555bae249d
error: 8eefbcac4a98fddc09381221023105249349eda7: object corrupt or missing: ./objects/8e/efbcac4a98fddc09381221023105249349eda7
error: 8f2e6721d7a02ce9686b64edbcc8df59ba4b2acb: object corrupt or missing: ./objects/8f/2e6721d7a02ce9686b64edbcc8df59ba4b2acb
error: 9f57e1fc819cef598c79002955398ac2ecf385a4: object corrupt or missing: ./objects/9f/57e1fc819cef598c79002955398ac2ecf385a4
error: a02f18fd37719a297d66a5ac2315472db517bb90: object corrupt or missing: ./objects/a0/2f18fd37719a297d66a5ac2315472db517bb90
error: a2898159d7de647efa13a176672b8ac5ac0af818: object corrupt or missing: ./objects/a2/898159d7de647efa13a176672b8ac5ac0af818
error: a3c2a0f20aadc8b63f677918b96061787dfa3627: object corrupt or missing: ./objects/a3/c2a0f20aadc8b63f677918b96061787dfa3627
error: ae589f1135b6b7c1da9aa639670a783ef63a2884: object corrupt or missing: ./objects/ae/589f1135b6b7c1da9aa639670a783ef63a2884
error: b32286077362313b7fb493fd6465627f5ecc00a6: object corrupt or missing: ./objects/b3/2286077362313b7fb493fd6465627f5ecc00a6
error: b541434add6eeefcb691a9c2e81a4bad9eab846c: object corrupt or missing: ./objects/b5/41434add6eeefcb691a9c2e81a4bad9eab846c
error: bd864dabc7cc84c7a10ca063e8cc313726df1833: object corrupt or missing: ./objects/bd/864dabc7cc84c7a10ca063e8cc313726df1833
error: c67d040960374014770fa52b0932da9e7e97b9f0: object corrupt or missing: ./objects/c6/7d040960374014770fa52b0932da9e7e97b9f0
error: e2948819ae5871a0369d5811c19ab81d9ea5965b: object corrupt or missing: ./objects/e2/948819ae5871a0369d5811c19ab81d9ea5965b
error: ec19e408bfc93b6f57008c55b2b558b046d308b6: object corrupt or missing: ./objects/ec/19e408bfc93b6f57008c55b2b558b046d308b6
error: eee799556d3638dac331f665f14673efd9d6535e: object corrupt or missing: ./objects/ee/e799556d3638dac331f665f14673efd9d6535e
error: f76612d0b268d933b03cf9ba451fc092aabce6c8: object corrupt or missing: ./objects/f7/6612d0b268d933b03cf9ba451fc092aabce6c8
error: sha1 mismatch for ./objects/02/7652dcf3fe817463bc717765a9501624c04ba8 (expected 027652dcf3fe817463bc717765a9501624c04ba8)
error: sha1 mismatch for ./objects/1a/0fdcd2a2b9ff97077cfc9f4168e090fd6e1e30 (expected 1a0fdcd2a2b9ff97077cfc9f4168e090fd6e1e30)
error: sha1 mismatch for ./objects/41/afa35ce9f600622b9e1e6f329c84ffa620d82b (expected 41afa35ce9f600622b9e1e6f329c84ffa620d82b)
error: sha1 mismatch for ./objects/45/9b8928a9530317aa786b4c52303af14a770fdb (expected 459b8928a9530317aa786b4c52303af14a770fdb)
error: sha1 mismatch for ./objects/54/817fa1753f50d0c83579089693dc5751a2d266 (expected 54817fa1753f50d0c83579089693dc5751a2d266)
error: sha1 mismatch for ./objects/5b/6554d4cc1bbd12da6c88736c41a370caefbf80 (expected 5b6554d4cc1bbd12da6c88736c41a370caefbf80)
error: sha1 mismatch for ./objects/5b/f3acc4bc87224d95f6563c60cb143116d90d27 (expected 5bf3acc4bc87224d95f6563c60cb143116d90d27)
error: sha1 mismatch for ./objects/5d/1b7a0df9fce5d86b07333470b147bf1ac07d28 (expected 5d1b7a0df9fce5d86b07333470b147bf1ac07d28)
error: sha1 mismatch for ./objects/6b/224f4c4adce562ce68483ae05a080ae5374858 (expected 6b224f4c4adce562ce68483ae05a080ae5374858)
error: sha1 mismatch for ./objects/6c/90e3ba21859ff45ea8819b0ae90ef6eb93747d (expected 6c90e3ba21859ff45ea8819b0ae90ef6eb93747d)
error: sha1 mismatch for ./objects/70/2b8b880a287f6c248c032fb2dc5c604dfc25ec (expected 702b8b880a287f6c248c032fb2dc5c604dfc25ec)
error: sha1 mismatch for ./objects/76/721387b06c098f2fcb8ef164d07c121cb18037 (expected 76721387b06c098f2fcb8ef164d07c121cb18037)
error: sha1 mismatch for ./objects/79/b3a102c9ff4f871a2fd4274550c785a7d0e205 (expected 79b3a102c9ff4f871a2fd4274550c785a7d0e205)
error: sha1 mismatch for ./objects/7d/03efc485b2191feb1cf466a0d6abb4428529a3 (expected 7d03efc485b2191feb1cf466a0d6abb4428529a3)
error: sha1 mismatch for ./objects/82/85a78fbb86b6dfa2de04aedae22c93fc8f15af (expected 8285a78fbb86b6dfa2de04aedae22c93fc8f15af)
error: sha1 mismatch for ./objects/83/f434d5f256886fcfcfc354c3a63b91a71fce63 (expected 83f434d5f256886fcfcfc354c3a63b91a71fce63)
error: sha1 mismatch for ./objects/8b/988a1644054c5cdc3960ff688c2e555bae249d (expected 8b988a1644054c5cdc3960ff688c2e555bae249d)
error: sha1 mismatch for ./objects/8e/efbcac4a98fddc09381221023105249349eda7 (expected 8eefbcac4a98fddc09381221023105249349eda7)
error: sha1 mismatch for ./objects/8f/2e6721d7a02ce9686b64edbcc8df59ba4b2acb (expected 8f2e6721d7a02ce9686b64edbcc8df59ba4b2acb)
error: sha1 mismatch for ./objects/9f/57e1fc819cef598c79002955398ac2ecf385a4 (expected 9f57e1fc819cef598c79002955398ac2ecf385a4)
error: sha1 mismatch for ./objects/a0/2f18fd37719a297d66a5ac2315472db517bb90 (expected a02f18fd37719a297d66a5ac2315472db517bb90)
error: sha1 mismatch for ./objects/a2/898159d7de647efa13a176672b8ac5ac0af818 (expected a2898159d7de647efa13a176672b8ac5ac0af818)
error: sha1 mismatch for ./objects/a3/c2a0f20aadc8b63f677918b96061787dfa3627 (expected a3c2a0f20aadc8b63f677918b96061787dfa3627)
error: sha1 mismatch for ./objects/ae/589f1135b6b7c1da9aa639670a783ef63a2884 (expected ae589f1135b6b7c1da9aa639670a783ef63a2884)
error: sha1 mismatch for ./objects/b3/2286077362313b7fb493fd6465627f5ecc00a6 (expected b32286077362313b7fb493fd6465627f5ecc00a6)
error: sha1 mismatch for ./objects/b5/41434add6eeefcb691a9c2e81a4bad9eab846c (expected b541434add6eeefcb691a9c2e81a4bad9eab846c)
error: sha1 mismatch for ./objects/bd/864dabc7cc84c7a10ca063e8cc313726df1833 (expected bd864dabc7cc84c7a10ca063e8cc313726df1833)
error: sha1 mismatch for ./objects/c6/7d040960374014770fa52b0932da9e7e97b9f0 (expected c67d040960374014770fa52b0932da9e7e97b9f0)
error: sha1 mismatch for ./objects/e2/948819ae5871a0369d5811c19ab81d9ea5965b (expected e2948819ae5871a0369d5811c19ab81d9ea5965b)
error: sha1 mismatch for ./objects/ec/19e408bfc93b6f57008c55b2b558b046d308b6 (expected ec19e408bfc93b6f57008c55b2b558b046d308b6)
error: sha1 mismatch for ./objects/ee/e799556d3638dac331f665f14673efd9d6535e (expected eee799556d3638dac331f665f14673efd9d6535e)
error: sha1 mismatch for ./objects/f7/6612d0b268d933b03cf9ba451fc092aabce6c8 (expected f76612d0b268d933b03cf9ba451fc092aabce6c8)
missing commit 83f434d5f256886fcfcfc354c3a63b91a71fce63Comment Actions
The corrupted / missing contents are also identified in T75 so these errors can be ignored
Comment Actions
After some tweaks[1], the git bare cooking is finally working correctly on g5k.
These are the results of the cooking of some random repositories:
| repo | prod flat | cassadra flat | prod git_bare | cassandra git_bare |
|---|---|---|---|---|
| https://github.com/Azure/sonic-mgmt (3575 commits) | 4494 | 3514 | ||
| https://github.com/Upsham/upsham.github.io (25 commits) | 22.01 | 28 | ||
| https://github.com/gaasedelen/lighthouse (539 commits) | 18.92 | 19.94 | ||
| https://github.com/bshaffer/oauth2-server-php (935 commits) | 23.7 | 22.78 | ||
| https://github.com/isandlaTech/cohorte-utilities (677 commits) | 136 | 1407.31 | 1071 | |
| https://github.com/nvaccess/nvda (20215 commits) | 166 | 200 | 21564 | 13416 |
| https://github.com/openfisca/openfisca-paris (268 commits) | 19.28 | 17 | 216 | 269 |
| https://github.com/rt-2pm2/cyphy_testbed (581 commits) | 39 | 51 | 565 | 760 |
| https://github.com/PranavShirole/pranavshirole.github.io (692 commits) | 48 | 77 | 681 | 356 |
| https://github.com/pi-hole/docker-pi-hole (1155 commits) | 10 | 8 | 455 | 440 |
| https://github.com/reitermarkus/brew (24000 commits) | 253 | 620 | 20982 | 13782 |
| https://github.com/alphagov/notifications-template-preview (947 commits) | 10 | 24 | 377 | 558 |
It seems the tar archive generation is globally slower with cassandra, for the git bare cooking, it's difficult to generalize, but it looks faster for big repositories
[1]
- in the loader configuration, remove the filter
- in the storage, change the allowOverwrite to True
- run a loading of the repository
- use an objstorage querying azure
Comment Actions
@vsellier Did you have swh-graph enabled in the vault configuration for these benchmarks?