Same as before but with 1M (fresh) sha1s:

Since the results on uffizi above did suffer from a few caveats, I've made a few more tests:

• a first result has been obtained with a dataset that had only objects stored on the XFS part of the objstorage
• a second dataset has been created (with the order by sha256 part to spread the sha1s)
• but results are a mix hot/cold cache tests

Some results:

(the backfill had, in fact, completed within a month)

At this point, I don't think we'll make it much better with postgres as source.

The kafka producer in swh.journal now reads message receipts and fails if they're negative, or if they didn't arrive within two minutes.

c = swh.journal.client.JournalClient(**{
    'group_id': 'olasd-test-sasl-1',
    'brokers': ['kafka%02d.euwest.azure.softwareheritage.org:9093' % i for i in range(1,7)],
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'SCRAM-SHA-512',
    'sasl.username': '<username>',
    'sasl.password': '<password>',
    'debug': 'consumer',
})

(yes, passing dotted config parameters in kwargs is... not the cleanest)

Following documentation with the following links:

• https://kafka.apache.org/documentation/#security
• https://rmoff.net/2018/08/02/kafka-listeners-explained/
• https://docs.confluent.io/current/kafka/encryption.html#ssl-overview
• sort of https://medium.com/code-tech/kafka-in-aws-with-ssl-offloading-using-load-balancer-c337da1435c3
• kinda https://www.slideshare.net/ConfluentInc/how-to-lock-down-apache-kafka-and-keep-your-streams-safe

We've now hit T2003 hard as the client caught up with the head of the local kafka cluster. That's why the curve is flattening out currently, as I stopped the replayers until the queue is implemented.

We'll need to address T2003 before this can be closed (if we go the journal client route), so marking accordingly.

I've launched 16 content backfillers in parallel for each hex digit prefix which should help with this.

I don't think we're going to do this but rather use the journal client approach. (Even more so considering that writing to S3 takes 500ms for each object, which sounds like a silly artificial limit to put on a synchronous process).

The content topic has fully replicated to the new cluster over the weekend.

A new Kafka cluster has been spun up on azure virtual machines, with 6 machines each with 8TB of storage available.

1 month is good enough. Let's stick to this.

With 16 processes in parallel still, adding more CPUs gives an ETA of ~1 month, which stays pretty bad.

Running the directory backfiller (single instance) against belvedere yields an ETA of 250 days, which is around a 3x speedup from somerset.

have we now any insight on the behavior of the backfiller against belvedere?

I'm enclined to prefer option 2, since performance is an issue we cannot underestimate...