Differential D7027

replay: Handle recursive external definition
ClosedPublic
Actions

Authored by anlambert on Jan 24 2022, 3:38 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T3870: Analyze svn externals repository ingestion with loader.svn v1.0

Commits

rDLDSVNafdee6dbee9b: replay: Handle recursive external definition

Summary

It is possible that an external definition lead to a recursive subversion
export operation (https://issues.apache.org/jira/browse/SVN-1703).

As that recursive export will crash the loader and make the loading failed,
the only solution for that edge case is to stop processing externals while
a recursive external is detected in a revision.

We will then ignore externals when exporting a revision to check for divergence
with the recontructed filesystem.

Related to T3870#77491

Diff Detail

Repository

rDLDSVN Subversion (SVN) loader

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Jan 24 2022, 3:38 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 24 2022, 3:38 PM

Build is green

Patch application report for D7027 (id=25464)

Rebasing onto 3e4d5d7cc9...

Current branch diff-target is up to date.

Changes applied before test

commit bb606ed569a3f07d792f0550625560dcb47a6cd2
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Jan 24 15:32:02 2022 +0100

    replay: Handle recursive external definition
    
    It is possible that an external definition lead to a recursive subversion
    export operation (https://issues.apache.org/jira/browse/SVN-1703).
    
    As that recursive export will crash the loader and make the loading failed,
    the only solution for that edge case is to stop processing externals while
    a recursive external is detected in a revision.
    
    We will then ignore externals when exporting a revision to check for divergence
    with the recontructed filesystem.
    
    Related to T3870

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/267/ for more details.

Harbormaster completed remote builds in B26300: Diff 25464.Jan 24 2022, 3:40 PM

anlambert requested review of this revision.Jan 24 2022, 3:40 PM

Update: Fix recursive external detection on incremental loading.

Build is green

Patch application report for D7027 (id=25471)

Rebasing onto 3e4d5d7cc9...

Current branch diff-target is up to date.

Changes applied before test

commit 51caee0663dfcb8645a6c440084a69225dc0c4a0
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Jan 24 15:32:02 2022 +0100

    replay: Handle recursive external definition
    
    It is possible that an external definition lead to a recursive subversion
    export operation (https://issues.apache.org/jira/browse/SVN-1703).
    
    As that recursive export will crash the loader and make the loading failed,
    the only solution for that edge case is to stop processing externals while
    a recursive external is detected in a revision.
    
    We will then ignore externals when exporting a revision to check for divergence
    with the recontructed filesystem.
    
    Related to T3870

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/268/ for more details.

Harbormaster completed remote builds in B26306: Diff 25471.Jan 24 2022, 5:05 PM

ok, one suggestion inline

swh/loader/svn/replay.py
695–698	I'm not entirely sure i get all of that sentence. Is that better ^?

This revision is now accepted and ready to land.Jan 24 2022, 5:44 PM

Slightly rephrase comment

swh/loader/svn/replay.py
695–698	I'm not entirely sure i get all of that sentence. Is that better ^?
695–698	I have updated the comment and removed the indeed word.

Build is green

Patch application report for D7027 (id=25476)

Rebasing onto 3e4d5d7cc9...

Current branch diff-target is up to date.

Changes applied before test

commit afdee6dbee9be55f2eaf192dfb23733952977a2e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Jan 24 15:32:02 2022 +0100

    replay: Handle recursive external definition
    
    It is possible that an external definition lead to a recursive subversion
    export operation (https://issues.apache.org/jira/browse/SVN-1703).
    
    As that recursive export will crash the loader and make the loading failed,
    the only solution for that edge case is to stop processing externals while
    a recursive external is detected in a revision.
    
    We will then ignore externals when exporting a revision to check for divergence
    with the recontructed filesystem.
    
    Related to T3870

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/269/ for more details.

Harbormaster completed remote builds in B26311: Diff 25476.Jan 24 2022, 5:57 PM

ardumont accepted this revision.Jan 24 2022, 5:58 PM

Closed by commit rDLDSVNafdee6dbee9b: replay: Handle recursive external definition (authored by anlambert). · Explain WhyJan 24 2022, 5:58 PM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDLDSVNafdee6dbee9b: replay: Handle recursive external definition.

anlambert mentioned this in T3870: Analyze svn externals repository ingestion with loader.svn v1.0.Jan 24 2022, 6:11 PM