Adapt extid filtering so it happens server side
ClosedPublic
Actions

Authored by ardumont on Sep 16 2021, 8:44 AM.

Details

Reviewers

anlambert
vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T3567: storage: Allow extid reading with filter on extid version

Commits

rDLDHG8e3b880ebcb8: Adapt extid filtering so it happens server side

Summary

This lifts the storage endpoint adaptations which allow to filter directly server side
server (storage > v0.37).

Related to T3567

Test Plan

tox (happy if v0.37 is released)

Diff Detail

Repository

rDLDHG Mercurial loader

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Sep 16 2021, 8:44 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptSep 16 2021, 8:44 AM

Build is green

Patch application report for D6275 (id=22726)

Could not rebase; Attempt merge onto bf39050051...

Updating bf39050..0d05077
Fast-forward
 requirements-swh.txt                         |   2 +-
 swh/loader/mercurial/from_disk.py            | 138 +++++++++++++++++++--------
 swh/loader/mercurial/tests/test_from_disk.py | 124 +++++++++++++++---------
 3 files changed, 180 insertions(+), 84 deletions(-)

Changes applied before test

commit 0d0507768ad5d19038e77bdcf48557e4c8158937
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Sep 16 08:40:49 2021 +0200

    Adapt extid filtering so it happens server side
    
    This lift the endpoint adaptations which allow the filtering directly in the
    server (storage > v0.37).
    
    Related to T3567

commit da1bd7c460b238edcbfdf8c7e931a4cf6fd9ab58
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Sep 15 14:53:24 2021 +0200

    Build snapshot on visits
    
    If an uneventful visit happens, then an empty snapshot should happen. When filtering
    revisions for speed, still the heads should appear in the snapshot.
    
    Related to T3571

commit 4c9eb38ec516de3f65c3996d937cd3328ded6261
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Sep 10 17:00:15 2021 +0200

    Add support to filter out already seen revisions across hg origins
    
    This uses the extid mapping from the storage to actually achieve this.
    
    Related to T3563

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/303/ for more details.

Harbormaster completed remote builds in B23659: Diff 22726.Sep 16 2021, 8:47 AM

ardumont requested review of this revision.Sep 16 2021, 8:47 AM

ardumont edited the summary of this revision. (Show Details)Sep 16 2021, 8:59 AM

ardumont edited the summary of this revision. (Show Details)

Looks good to me.

This revision is now accepted and ready to land.Sep 16 2021, 10:54 AM

vlorentz accepted this revision.Sep 16 2021, 11:10 AM

Rebase

Build is green

Patch application report for D6275 (id=22746)

Could not rebase; Attempt merge onto bf39050051...

Updating bf39050..8e3b880
Fast-forward
 requirements-swh.txt                         |   2 +-
 swh/loader/mercurial/from_disk.py            | 135 +++++++++++++++++++--------
 swh/loader/mercurial/tests/test_from_disk.py | 124 +++++++++++++++---------
 3 files changed, 178 insertions(+), 83 deletions(-)

Changes applied before test

commit 8e3b880ebcb85697a476aa002fa0808d0107d821
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Sep 16 08:40:49 2021 +0200

    Adapt extid filtering so it happens server side
    
    This lift the endpoint adaptations which allow the filtering directly in the
    server (storage > v0.37).
    
    Related to T3567

commit a0f8e9e856a4d659e9e73d5973f4c87884b79f6f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Sep 15 14:53:24 2021 +0200

    Build snapshot on visits
    
    If an uneventful visit happens, then an empty snapshot should happen. When filtering
    revisions for speed, still the heads should appear in the snapshot.
    
    Related to T3571

commit 765cb5aefa8bb370bc1343d990ad328188a4eba5
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Sep 10 17:00:15 2021 +0200

    Add support to filter out already seen revisions across hg origins
    
    This uses the extid mapping from the storage to actually achieve this.
    
    Related to T3563

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/306/ for more details.

Harbormaster completed remote builds in B23677: Diff 22746.Sep 16 2021, 1:52 PM

ardumont mentioned this in T3563: Analyze and make the bitbucket ingestion faster.Sep 16 2021, 5:15 PM

Closed by commit rDLDHG8e3b880ebcb8: Adapt extid filtering so it happens server side (authored by ardumont). · Explain WhySep 17 2021, 10:05 AM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDLDHG8e3b880ebcb8: Adapt extid filtering so it happens server side.