Page MenuHomeSoftware Heritage

Enable the journal-writer for the swh-idx-storage in staging
Closed, ResolvedPublic

Description

equivalent of T2780 but for the staging

Event Timeline

vsellier triaged this task as Normal priority.Nov 26 2020, 5:40 PM
vsellier created this task.
vsellier claimed this task.

The swh-indexer stack is deployed on staging and the initial loading is done.
The volumes are quite low :

 % for topic in $(./kafka-topics.sh --bootstrap-server $SERVER --list | grep indexed)                                                         
do              
  echo -n "$topic : "
  ./kafka-run-class.sh kafka.tools.GetOffsetShell  --broker-list $SERVER --topic $topic | awk -F: '{s+=$3}END{print s}'
done

swh.journal.indexed.content_metadata : 4
swh.journal.indexed.origin_intrinsic_metadata : 20
swh.journal.indexed.revision_intrinsic_metadata : 20

IMO, This task can be closed, but the retention / partitioning still have to be defined