Differential D6648

loader: Create temp folder with pattern in SvnLoaderFromRemoteDump
ClosedPublic
Actions

Authored by anlambert on Nov 17 2021, 2:32 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T3719: loader svn fails to clean up properly disk when it crashes

Commits

rDLDSVNfdf526eceaea: loader: Create temp folder with pattern in SvnLoaderFromRemoteDump

Summary

It enables to properly remove remaining partial dump file resulting
from a celery task crash when a new svn loading task is executed
on the same worker.

Related to T3719

Diff Detail

Repository

rDLDSVN Subversion (SVN) loader

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Nov 17 2021, 2:32 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptNov 17 2021, 2:32 PM

Build is green

Patch application report for D6648 (id=24171)

Rebasing onto 49cd7cc4fa...

Current branch diff-target is up to date.

Changes applied before test

commit e1d2af937882e0e86e8ef257fd314535eb87e29d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Nov 17 14:25:48 2021 +0100

    loader: Create temp folder with pattern in SvnLoaderFromRemoteDump
    
    It enables to properly remove remaining partial dump file resulting
    from a celery task crash when a new svn loading task is executed
    on the same worker.
    
    Related to T3719

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/191/ for more details.

Harbormaster completed remote builds in B25052: Diff 24171.Nov 17 2021, 2:33 PM

anlambert requested review of this revision.Nov 17 2021, 2:33 PM

ardumont added a subscriber: ardumont.Nov 17 2021, 2:34 PM

ardumont added inline comments.

swh/loader/svn/loader.py
650	since we repeat it at least twice now in that loader, wondering whether we should not wrap this in an utility function.

Thanks.

That's what i was trying to fix last week without finding where (since i was unable to reproduce in tests).
I think you nailed it!

One non-blocking question inline.

This revision is now accepted and ready to land.Nov 17 2021, 2:38 PM

anlambert added inline comments.Nov 17 2021, 2:40 PM

swh/loader/svn/loader.py
650	ack, will update.

Update: Wrap temporary folder creation in a private method

Build is green

Patch application report for D6648 (id=24173)

Rebasing onto 49cd7cc4fa...

Current branch diff-target is up to date.

Changes applied before test

commit fdf526eceaea5bbd35e5642e47770c251242bb45
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Nov 17 14:25:48 2021 +0100

    loader: Create temp folder with pattern in SvnLoaderFromRemoteDump
    
    It enables to properly remove remaining partial dump file resulting
    from a celery task crash when a new svn loading task is executed
    on the same worker.
    
    Related to T3719

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/192/ for more details.

Harbormaster completed remote builds in B25054: Diff 24173.Nov 17 2021, 3:00 PM

Closed by commit rDLDSVNfdf526eceaea: loader: Create temp folder with pattern in SvnLoaderFromRemoteDump (authored by anlambert). · Explain WhyNov 17 2021, 3:04 PM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDLDSVNfdf526eceaea: loader: Create temp folder with pattern in SvnLoaderFromRemoteDump.