Differential D8369

indexer.cli: Allow batch_size configuration on journal client
ClosedPublic
Actions

Authored by ardumont on Aug 31 2022, 5:20 PM.

Details

Reviewers

None

Group Reviewers

Reviewers

Maniphest Tasks

T4477: staging origin intrinsic metadata indexer are stuck

Commits

rDCIDX752e5d3f733e: indexer.cli: Allow batch_size configuration on journal client

Summary

Related to T4477

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 31245
Build 48876: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 48875: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	1,486 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__content_mimetype cli_runner = <click.testing.CliRunner object at 0x7fbd6847c438> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_4/indexer.yml' kafka_prefix = 'urqmhkysvg', kafka_server = '127.0.0.1:33111'
	1,648 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__fossology_license cli_runner = <click.testing.CliRunner object at 0x7fbd6835e470> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_5/indexer.yml' kafka_prefix = 'tqljpnpsrp', kafka_server = '127.0.0.1:33111'
	2,279 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__origin_extrinsic_metadata[*] cli_runner = <click.testing.CliRunner object at 0x7fbd685150b8> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_3/indexer.yml' kafka_prefix = 'zhojmhoflg', kafka_server = '127.0.0.1:33111'
	3,188 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__origin_extrinsic_metadata[extrinsic_metadata] cli_runner = <click.testing.CliRunner object at 0x7fbd68523b70> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_2/indexer.yml' kafka_prefix = 'xzcfrzqlop', kafka_server = '127.0.0.1:33111'
	998 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_journal_client_index__origin_intrinsic_metadata[*] cli_runner = <click.testing.CliRunner object at 0x7fbd6851bd30> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_journal_client_index_1/indexer.yml' kafka_prefix = 'mxcayypftu', kafka_server = '127.0.0.1:33111'
		View Full Test Results (7 Failed · 342 Passed · 11 Skipped)

Event Timeline

ardumont created this revision.Aug 31 2022, 5:20 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptAug 31 2022, 5:20 PM

Build has FAILED

Patch application report for D8369 (id=30201)

Rebasing onto 83abf119f7...

Current branch diff-target is up to date.

Changes applied before test

commit d938585bf70ecdb8899e2eec4cd54bd6bca8c251
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 31 17:16:58 2022 +0200

    indexer.cli: Allow batch_size configuration on journal client
    
    Related to T4477

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/469/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/469/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 31 2022, 5:26 PM

Harbormaster failed remote builds in B31242: Diff 30201!

vlorentz added a subscriber: vlorentz.Aug 31 2022, 5:30 PM

vlorentz added inline comments.

swh/indexer/cli.py
300

Default batch size like swh.journal's client

Build has FAILED

Patch application report for D8369 (id=30204)

Rebasing onto 83abf119f7...

Current branch diff-target is up to date.

Changes applied before test

commit 5db4202534aa776ca2748a20db8996752cd56e63
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 31 17:16:58 2022 +0200

    indexer.cli: Allow batch_size configuration on journal client
    
    Related to T4477

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/470/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/470/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 31 2022, 5:36 PM

Harbormaster failed remote builds in B31245: Diff 30204!

Adapt according to review

Build is green

Patch application report for D8369 (id=30205)

Rebasing onto 83abf119f7...

Current branch diff-target is up to date.

Changes applied before test

commit 752e5d3f733e5fa9a43d086713eeb40358cc76d6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 31 17:16:58 2022 +0200

    indexer.cli: Allow batch_size configuration on journal client
    
    Related to T4477

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/471/ for more details.

Harbormaster completed remote builds in B31246: Diff 30205.Aug 31 2022, 5:51 PM

ardumont requested review of this revision.Aug 31 2022, 5:51 PM

This revision was not accepted when it landed; it landed in state Needs Review.Aug 31 2022, 6:11 PM

Closed by commit rDCIDX752e5d3f733e: indexer.cli: Allow batch_size configuration on journal client (authored by ardumont). · Explain Why

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDCIDX752e5d3f733e: indexer.cli: Allow batch_size configuration on journal client.