Differential D6650

git_bare: Fetch directories concurrently, using threads.
ClosedPublic
Actions

Authored by vlorentz on Nov 17 2021, 6:22 PM.

Details

Reviewers

olasd

Group Reviewers

Reviewers

Maniphest Tasks

T3731: Try parallelizing directory_get_entries with an async swh-storage client

Commits

rDVAU5fcbeb0be2e3: git_bare: Fetch directories concurrently, using threads.

Summary

This divides the total wall time by ~2.

Diff Detail

Repository

rDVAU Software Heritage Vault

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Nov 17 2021, 6:22 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptNov 17 2021, 6:22 PM

Harbormaster completed remote builds in B25058: Diff 24178.Nov 17 2021, 6:24 PM

Build is green

Patch application report for D6650 (id=24178)

Rebasing onto bd649ccb97...

Current branch diff-target is up to date.

Changes applied before test

commit c26a46f689288090930e22cf0e0f8378728bbe08
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Nov 17 18:18:06 2021 +0100

    git_bare: Fetch directories concurrently, using threads.
    
    This divides the total wall time by ~2.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/186/ for more details.

vlorentz requested review of this revision.Nov 17 2021, 6:24 PM

In general I'm a bit uneasy about parallelism for individual workers, as it's hidden and multiplies by the "exogeneous" parallelism of having multiple workers.

In that specific case, and considering the relatively small git-bare cooker workload, it's probably worth it though.

Could you consider adding this as a configuration option rather than a hardcoded constant? We may want to turn the parallelism off if we end up cooking a (large) batch of objects at once to reduce database load.

This revision is now accepted and ready to land.Nov 18 2021, 1:26 PM

vlorentz added a task: T3731: Try parallelizing directory_get_entries with an async swh-storage client.Nov 22 2021, 10:35 AM

Yeah I'm not a huge fan either, that's why I started with the async approach; but that's a lot of trouble so I'd rather do this.

make parallelism level configurable

Build is green

Patch application report for D6650 (id=24220)

Rebasing onto bd649ccb97...

Current branch diff-target is up to date.

Changes applied before test

commit 5fcbeb0be2e3af4a62028d8470b4f473f232d404
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Nov 17 18:18:06 2021 +0100

    git_bare: Fetch directories concurrently, using threads.
    
    This divides the total wall time by ~2.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/188/ for more details.

Harbormaster completed remote builds in B25096: Diff 24220.Nov 22 2021, 11:50 AM

Closed by commit rDVAU5fcbeb0be2e3: git_bare: Fetch directories concurrently, using threads. (authored by vlorentz). · Explain WhyNov 22 2021, 11:53 AM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDVAU5fcbeb0be2e3: git_bare: Fetch directories concurrently, using threads..