Differential D8369

indexer.cli: Allow batch_size configuration on journal client
ClosedPublic
Actions

Authored by ardumont on Aug 31 2022, 5:20 PM.

Details

Reviewers

None

Group Reviewers

Reviewers

Maniphest Tasks

T4477: staging origin intrinsic metadata indexer are stuck

Commits

rDCIDX752e5d3f733e: indexer.cli: Allow batch_size configuration on journal client

Summary

Related to T4477

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 31242
Build 48870: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 48869: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	1,159 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__content_mimetype cli_runner = <click.testing.CliRunner object at 0x7fcb1467da90> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_4/indexer.yml' kafka_prefix = 'farusiwnpa', kafka_server = '127.0.0.1:40417'
	1,258 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__fossology_license cli_runner = <click.testing.CliRunner object at 0x7fcb1444d978> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_5/indexer.yml' kafka_prefix = 'fyhmtxdscb', kafka_server = '127.0.0.1:40417'
	3,604 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__origin_extrinsic_metadata[*] cli_runner = <click.testing.CliRunner object at 0x7fcb146fe048> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_3/indexer.yml' kafka_prefix = 'hwlhumkwyb', kafka_server = '127.0.0.1:40417'
	2,648 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__origin_extrinsic_metadata[extrinsic_metadata] cli_runner = <click.testing.CliRunner object at 0x7fcb146fccc0> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_2/indexer.yml' kafka_prefix = 'rwcbvryqzn', kafka_server = '127.0.0.1:40417'
	1,595 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__origin_intrinsic_metadata[*] cli_runner = <click.testing.CliRunner object at 0x7fcb16c9b320> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_1/indexer.yml' kafka_prefix = 'poywbzwmhy', kafka_server = '127.0.0.1:40417'
		View Full Test Results (7 Failed · 342 Passed · 11 Skipped)

Event Timeline

ardumont created this revision.Aug 31 2022, 5:20 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptAug 31 2022, 5:20 PM

Build has FAILED

Patch application report for D8369 (id=30201)

Rebasing onto 83abf119f7...

Current branch diff-target is up to date.

Changes applied before test

commit d938585bf70ecdb8899e2eec4cd54bd6bca8c251
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 31 17:16:58 2022 +0200

    indexer.cli: Allow batch_size configuration on journal client
    
    Related to T4477

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/469/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/469/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 31 2022, 5:26 PM

Harbormaster failed remote builds in B31242: Diff 30201!

vlorentz added a subscriber: vlorentz.Aug 31 2022, 5:30 PM

vlorentz added inline comments.

swh/indexer/cli.py
301

Default batch size like swh.journal's client

Build has FAILED

Patch application report for D8369 (id=30204)

Rebasing onto 83abf119f7...

Current branch diff-target is up to date.

Changes applied before test

commit 5db4202534aa776ca2748a20db8996752cd56e63
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 31 17:16:58 2022 +0200

    indexer.cli: Allow batch_size configuration on journal client
    
    Related to T4477

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/470/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/470/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 31 2022, 5:36 PM

Harbormaster failed remote builds in B31245: Diff 30204!

Adapt according to review

Build is green

Patch application report for D8369 (id=30205)

Rebasing onto 83abf119f7...

Current branch diff-target is up to date.

Changes applied before test

commit 752e5d3f733e5fa9a43d086713eeb40358cc76d6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 31 17:16:58 2022 +0200

    indexer.cli: Allow batch_size configuration on journal client
    
    Related to T4477

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/471/ for more details.

Harbormaster completed remote builds in B31246: Diff 30205.Aug 31 2022, 5:51 PM

ardumont requested review of this revision.Aug 31 2022, 5:51 PM

This revision was not accepted when it landed; it landed in state Needs Review.Aug 31 2022, 6:11 PM

Closed by commit rDCIDX752e5d3f733e: indexer.cli: Allow batch_size configuration on journal client (authored by ardumont). · Explain Why

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDCIDX752e5d3f733e: indexer.cli: Allow batch_size configuration on journal client.