backfill.sh for origin-visit-status
ActivePublic
Actions

Authored by ardumont on Feb 1 2021, 11:51 AM.

Tags

None

Subscribers

None

	#!/bin/bash

	objtype=origin_visit_status
	first=0

	mkdir -p logs/$objtype
	inc=10000000
	# 151 747 682
	for i in `seq 0 15`; do
	prefix=$(( first + i * $inc))
	if [ $prefix -eq $first ]; then
	start=$first
	else
	start=$prefix
	fi
	end=$((prefix + $inc));

	echo "Starting ${objtype} backfill for range $start -> $end"
	swh --log-config logging.yml storage --config-file backfill.yml backfill --start-object $start --end-object $end $objtype 2>&1 \| tee -a >(gzip -c > logs/$objtype/$prefix.log.gz) &
	done

	wait

Event Timeline

Related to T2993
Adapted from P927

backfill.yml:

---
journal_writer:
  cls: kafka
  brokers:
  - kafka1.internal.softwareheritage.org
  - kafka2.internal.softwareheritage.org
  - kafka3.internal.softwareheritage.org
  - kafka4.internal.softwareheritage.org
  prefix: swh.journal.objects
  client_id: swh.journal.backfiller
  anonymize: true
  producer_config:
    message.max.bytes: 104857600
    compression.type: zstd
storage:
  cls: local
  db: host=belvedere.internal.softwareheritage.org port=5433 user=guest password=guest dbname=softwareheritage

logging.yml:

version: 1

handlers:
  console:
    class: logging.StreamHandler
    formatter: detail
    level: INFO
    stream: ext://sys.stdout

formatters:
  detail:
    format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
    datefmt: '%Y-%m-%dT%H:%M:%S'

loggers:
  swh: {}

root:
  level: INFO
  handlers:
  - console

olasd updated the paste's language from autodetect to bash.Feb 1 2021, 12:00 PM

olasd mentioned this in T3000: Investigate scheduler journal client discrepancies.Feb 1 2021, 2:14 PM

backfill.sh for origin-visit-statusActivePublicActions

Event Timeline

backfill.sh for origin-visit-status
ActivePublic
Actions