Query: All Stories

	Include stories about projects I am a member of.

The backfill is also done for the production.
It tooks less than 4h30

...
2021-08-25T19:25:25 INFO     swh.storage.backfill Processing extid range 700000 to 700001

Rename bundle types and use SWHIDs everywhere instead of raw sha1_git

Remove page refresh hack from updateVaultItemList.

Add new 'deprecated' tag for API routes, and 'un-hide' deprecate vault routes…

vault API: Rename bundle types and use SWHIDs to identify objects

status:

scheduler v0.17.1 deployed on production [1] (db migrated) and staging.
then swh-scheduler-journal-client service restarted.

D6139 should address the bottleneck in the flame graph

Add SQL enum for relation get filter options

search: Add visit_types_count to interface

Update test according to @ardumont suggesstion

Updated backport on buster-swh from debian/0.17.1-1_swh1 (unstable-swh)

Merge tag 'debian/0.17.1-1_swh1' into debian/buster-swh

pristine-tar data for swh-scheduler_0.17.1.orig.tar.gz

Updated debian changelog for version 0.17.1

Update upstream source from tag 'debian/upstream/0.17.1'

New upstream version 0.17.1

journal_client: Ensure queue position does not overflow

Fix another typo in the sql this time

oops, wasn't supposed to be in this diff

re-hide 'raw' endpoints

I bumped the priority since scheduler runners (next-gen) are depending on the results of journal client (scheduler metrics as well).

Clarify code change
Update docstring to mention the possible side when calculating offset.
Add a migration script to adapt existing values in db

lgtm, one suggestion inline.

@vlorentz should we close this one?

For the rpm support, [1] may help.

Rename count_visit_types to visit_types_count.

tarball: Add support for tbz and tbz2 tarball formats

It was really faster than expected in staging. The backfilling is already done:

on production:

vsellier@kafka1 ~ % /opt/kafka/bin/kafka-topics.sh  --bootstrap-server $SERVER --describe --topic swh.journal.objects.extid | grep "^Topic"
Topic: swh.journal.objects.extid	PartitionCount: 256	ReplicationFactor: 2	Configs: cleanup.policy=compact,max.message.bytes=104857600
vsellier@kafka1 ~ % /opt/kafka/bin/kafka-configs.sh --bootstrap-server ${SERVER} --alter  --add-config 'cleanup.policy=[compact,delete],retention.ms=86400000' --entity-type=topics --entity-name swh.journal.objects.extid
Completed updating config for topic swh.journal.objects.extid.

In the kafka logs:

...
[2021-08-25 14:56:19,495] INFO [Log partition=swh.journal.objects.extid-162, dir=/srv/kafka/logdir] Found deletable segments with base offsets [0] due to retention time 86400000ms breach (kafka.log.Log)
[2021-08-25 14:56:19,495] INFO [Log partition=swh.journal.objects.extid-162, dir=/srv/kafka/logdir] Scheduling segments for deletion LogSegment(baseOffset=0, size=2720767, lastModifiedTime=1629815520833, largestTime=1629815520702) (kafka.log.Log)
[2021-08-25 14:56:19,495] INFO [Log partition=swh.journal.objects.extid-162, dir=/srv/kafka/logdir] Incremented log start offset to 20623 due to segment deletion (kafka.log.Log)
....

vsellier@kafka1 ~ % /opt/kafka/bin/kafka-configs.sh --bootstrap-server ${SERVER} --alter  --delete-config 'cleanup.policy' --entity-type=topics --entity-name swh.journal.objects.extid                                                
Completed updating config for topic swh.journal.objects.extid.
vsellier@kafka1 ~ % /opt/kafka/bin/kafka-configs.sh --bootstrap-server ${SERVER} --alter  --delete-config 'retention.ms' --entity-type=topics --entity-name swh.journal.objects.extid 
Completed updating config for topic swh.journal.objects.extid.
vsellier@kafka1 ~ % /opt/kafka/bin/kafka-configs.sh --bootstrap-server ${SERVER} --alter  --add-config 'cleanup.policy=compact' --entity-type=topics --entity-name swh.journal.objects.extid
Completed updating config for topic swh.journal.objects.extid.
vsellier@kafka1 ~ % /opt/kafka/bin/kafka-topics.sh  --bootstrap-server $SERVER --describe --topic swh.journal.objects.extid | grep "^Topic"                                          
Topic: swh.journal.objects.extid	PartitionCount: 256	ReplicationFactor: 2	Configs: cleanup.policy=compact,max.message.bytes=104857600

MongoDB backend implementation for provenancestorage.

the retention policy was restore to compact on staging:

vsellier@journal0 ~ % /opt/kafka/bin/kafka-configs.sh --bootstrap-server journal0.internal.staging.swh.network:9092 --alter  --delete-config 'cleanup.policy' --entity-type=topics --entity-name swh.journal.objects.extid

% /opt/kafka/bin/kafka-configs.sh --bootstrap-server journal0.internal.staging.swh.network:9092 --alter  --add-config 'cleanup.policy=compact' --entity-type=topics --entity-name swh.journal.objects.extid
Completed updating config for topic swh.journal.objects.extid.

% /opt/kafka/bin/kafka-topics.sh  --bootstrap-server $SERVER --describe --topic swh.journal.objects.extid | grep "^Topic"
Topic: swh.journal.objects.extid	PartitionCount: 64	ReplicationFactor: 1	Configs: cleanup.policy=compact,max.message.bytes=104857600,min.cleanable.dirty.ratio=0.01

set 3 origins types per row instead of 4 to improve readability

MongoDB backend implementation for provenancestorage.

Added the necessary directory for mongo pytest to work

MongoDB backend implementation for provenancestorage.

Made default provenance storage to psql

Pytest is using mongomock

I've duplicated the credentials for the relevant forges, and updated the following instance names:

All Stories
Use Results
Edit Query
Hide Query

Aug 26 2021

Aug 25 2021

All StoriesUse ResultsEdit QueryHide Query

Aug 26 2021

Aug 25 2021

All Stories
Use Results
Edit Query
Hide Query