stuff related to the mirroring infrastructure, protocol, and tooling used to maintain the Software Heritage mirror network
May 11 2021
May 4 2021
Apr 27 2021
Where should this information be available?
- on the main website
- a private location
Apr 23 2021
Apr 22 2021
These revisions are probably coming from https://gitlab.com/gitlab-org/gitlab-test (or a clone)
Ah fun, one of the revisions with this pb, on staging (ba3343bc4fa403a8dfbfcab7fc1a8c29ee34bd69) seems to have been crafted by https://gitlab.com/gitlab-org/gitlab-foss/-/blob/staging-26-fix_add_deploy_key_spec/spec/models/merge_request_diff_commit_spec.rb
Apr 21 2021
See T3170 (error generated by the same invalid kafka messages).
Apr 19 2021
Apr 8 2021
Just got this one below. Note that this occurred just when the replayer actually started to insert object in the storage (before that, since the start of the replayer process, only kafka scaffolding took place for quite some time, around 30mn!)
Apr 6 2021
Apr 2 2021
Currently, the mirror test session is running with:
easy fix: modify the replayer to ignore this 'metadata' column while inserting revisions
09:45 <+vlorentz> douardda: yes and the only way around it (short of dropping data) is T3089 09:46 -swhbot:#swh-devel- T3089 (submitter: vlorentz, owner: vlorentz, status: Open): Remove the 'metadata' column of the 'revision' table <https://forge.softwareheritage.org/T3089> 09:46 <+vlorentz> or switching to cassandra 09:46 <+vlorentz> the good news is, they couldn't be inserted in the storage either, so you can safely drop them for now
Mar 15 2021
Mar 11 2021
Mar 4 2021
Dec 23 2020
Nov 17 2020
The new cluster in rocquencourt is using the built-in Kafka ACLs now (9993a81ffc7a1c8bd519b33ae63ac1145105f624).
Oct 16 2020
Same as before but with 1M (fresh) sha1s:
Since the results on uffizi above did suffer from a few caveats, I've made a few more tests:
- a first result has been obtained with a dataset that had only objects stored on the XFS part of the objstorage
- a second dataset has been created (with the order by sha256 part to spread the sha1s)
- but results are a mix hot/cold cache tests
Oct 15 2020
Sep 22 2020
(the backfill had, in fact, completed within a month)
At this point, I don't think we'll make it much better with postgres as source.
Apr 28 2020
The kafka producer in swh.journal now reads message receipts and fails if they're negative, or if they didn't arrive within two minutes.