Page MenuHomeSoftware Heritage

new-loader: fix incremental load of multiple heads
ClosedPublic

Authored by Alphare on May 27 2021, 6:24 PM.

Details

Summary

Some branches have multiple heads for a single branch (they can even be
closed heads). The SWH data model does not yet handle this, so we fix
the issue by asking a more precise question to the repository by
including all locals heads that are already stored as revisions.

We also test that this resolves the issue where the new loader would
always see the additional head as missing.

Diff Detail

Repository
rDLDHG Mercurial loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D5793 (id=20704)

Rebasing onto 44fc0133da...

Current branch diff-target is up to date.
Changes applied before test
commit e7893d22339d32536a2911ad4c88bd57344e01a1
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Thu May 27 18:19:57 2021 +0200

    new-loader: fix incremental load of multiple heads
    
    Some branches have multiple heads for a single branch (they can even be
    closed heads). The SWH data model does not yet handle this, so we fix
    the issue by asking a more precise question to the repository by
    including all locals heads that are already stored as revisions.
    
    We also test that this resolves the issue where the new loader would
    always see the additional head as missing.

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/243/ for more details.

vlorentz added inline comments.
swh/loader/mercurial/from_disk.py
311–316

What about this instead, to batch RPC calls?

extids = self.storage.extid_get_from_extid(EXTID_TYPE, repo.heads())
known_heads = {extid.extid for extid in extids}
existing_heads.extend([repo[head].rev() for head in known_heads])

(not tested)

Alphare added inline comments.
swh/loader/mercurial/from_disk.py
311–316

Of course, what was I thinking!

Build is green

Patch application report for D5793 (id=20705)

Rebasing onto 44fc0133da...

Current branch diff-target is up to date.
Changes applied before test
commit 0f19c32fc0a511fa163fde2a0f22038ed3f84cf0
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Thu May 27 18:19:57 2021 +0200

    new-loader: fix incremental load of multiple heads
    
    Some branches have multiple heads for a single branch (they can even be
    closed heads). The SWH data model does not yet handle this, so we fix
    the issue by asking a more precise question to the repository by
    including all locals heads that are already stored as revisions.
    
    We also test that this resolves the issue where the new loader would
    always see the additional head as missing.

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/244/ for more details.

olasd added a subscriber: olasd.

Thanks!

This revision is now accepted and ready to land.May 28 2021, 11:07 AM
This revision was automatically updated to reflect the committed changes.
Alphare marked an inline comment as done.