backfill script with the right backfill.yml and logging.yml configuration files
ActivePublic
Actions

Authored by vsellier on Jan 20 2021, 6:45 PM.

Tags

None

Subscribers

None

	#!/bin/bash

	objtype=origin_visit_status
	first=0

	mkdir -p logs/$objtype
	inc=50000
	# 458591
	for i in `seq 0 10`; do
	prefix=$(( first + i * $inc))
	if [ $prefix -eq $first ]; then
	start=$first
	else
	start=$prefix
	fi
	end=$((prefix + $inc));

	echo "Starting ${objtype} backfill for range $start -> $end"
	swh --log-config logging.yml storage --config-file backfill.yml backfill --start-object $start --end-object $end $objtype 2>&1 \| tee -a >(gzip -c > logs/$objtype/$prefix.log.gz) &
	done

	wait

Event Timeline

vsellier created this paste.Jan 20 2021, 6:45 PM

vsellier mentioned this in T2978: Deploy visit-stats journal client on staging.

ardumont edited the content of this paste. (Show Details)Jan 26 2021, 2:31 PM

ardumont mentioned this in P934 backfill.sh for origin-visit-status.Feb 1 2021, 11:54 AM

ardumont mentioned this in T2993: Deploy visit-stats journal client on production.

ardumont changed the title of this paste from backfill script to backfill script with the right backfill.yml and logging.yml configuration files.Feb 1 2021, 11:59 AM

vsellier mentioned this in T3009: Manage backfiller configuration in puppet.Feb 1 2021, 12:10 PM

backfill script with the right backfill.yml and logging.yml configuration filesActivePublicActions

Event Timeline

backfill script with the right backfill.yml and logging.yml configuration files
ActivePublic
Actions