Differential D7586

loader: Fix rsync failures by retrying associated commands
ClosedPublic
Actions

Authored by anlambert on Apr 14 2022, 5:23 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDLDCVS9f5456b6f0c3: loader: Fix rsync failures by retrying associated commands

Summary

Fetching CVS repository data using rsync often fails with the following error
(especially with archived repositories hosted on sourceforge):

rsync error: some files/attrs were not transferred (see previous errors) (code 23)

It seems the only way to mitigate that issue is to retry the rsync command
until it succeeds.

(swh) anlambert@carnavalet:/tmp$ rsync -avz rsync://a.cvs.sourceforge.net/cvsroot/ojirc/ /tmp/cvsroot/ojirc/
receiving incremental file list
rsync: change_dir "/ojirc" (in cvsroot) failed: No such file or directory (2)

sent 8 bytes  received 99 bytes  71.33 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1819) [Receiver=3.2.3]
(swh) anlambert@carnavalet:/tmp$ echo $?
23
(swh) anlambert@carnavalet:/tmp$ rsync -avz rsync://a.cvs.sourceforge.net/cvsroot/ojirc/ /tmp/cvsroot/ojirc/
receiving incremental file list

sent 34 bytes  received 2,050 bytes  833.60 bytes/sec
total size is 487,994  speedup is 234.16
(swh) anlambert@carnavalet:/tmp$ echo $?
0

So add a rsync_retry decorator and apply it to a new method in the loader
wrapping the call to subprocess.run executing the rsync command.

Also use rsync option to compress file data during the transfer.

Fixes SWH-LOADER-CVS-D and SWH-LOADER-CVS-J.

Diff Detail

Repository

rDLDCVS CVS Loader

Branch

rsync-retry

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 28487
Build 44541: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 44540: arc lint + arc unit

Event Timeline

anlambert created this revision.Apr 14 2022, 5:23 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptApr 14 2022, 5:23 PM

Update commit message

Build is green

Patch application report for D7586 (id=27466)

Rebasing onto c82c326865...

Current branch diff-target is up to date.

Changes applied before test

commit 7d4f1bd2e444622eb00c74b4f85882cc9f8df5f9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Apr 14 17:12:53 2022 +0200

    loader: Fix rsync failures by retrying associated commands
    
    Fetching CVS repository data using rsync often fails with the following error
    (especially with archived repositories hosted on sourceforge):
    
    rsync error: some files/attrs were not transferred (see previous errors) (code 23)
    
    It seems the only way to mitigate that issue is to retry the rsync command
    until it succeeds.
    
    So add a rsync_retry decorator and apply it to a new method in the loader
    wrapping the call to subprocess.run executing the rsync command.

See https://jenkins.softwareheritage.org/job/DLDCVS/job/tests-on-diff/96/ for more details.

Harbormaster completed remote builds in B28486: Diff 27466.Apr 14 2022, 5:25 PM

anlambert edited the summary of this revision. (Show Details)Apr 14 2022, 5:26 PM

Remove line break in test

Build is green

Patch application report for D7586 (id=27467)

Rebasing onto c82c326865...

Current branch diff-target is up to date.

Changes applied before test

commit 8e737018fcefef79972588d47203a7500af3379f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Apr 14 17:12:53 2022 +0200

    loader: Fix rsync failures by retrying associated commands
    
    Fetching CVS repository data using rsync often fails with the following error
    (especially with archived repositories hosted on sourceforge):
    
    rsync error: some files/attrs were not transferred (see previous errors) (code 23)
    
    It seems the only way to mitigate that issue is to retry the rsync command
    until it succeeds.
    
    So add a rsync_retry decorator and apply it to a new method in the loader
    wrapping the call to subprocess.run executing the rsync command.
    
    Also use rsync option to compress file data during the transfer.

See https://jenkins.softwareheritage.org/job/DLDCVS/job/tests-on-diff/97/ for more details.

Harbormaster completed remote builds in B28487: Diff 27467.Apr 14 2022, 5:28 PM

Build is green

Patch application report for D7586 (id=27468)

Rebasing onto c82c326865...

Current branch diff-target is up to date.

Changes applied before test

commit 9f5456b6f0c3f4f57668cb61390bd5cfd86fd827
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Apr 14 17:12:53 2022 +0200

    loader: Fix rsync failures by retrying associated commands
    
    Fetching CVS repository data using rsync often fails with the following error
    (especially with archived repositories hosted on sourceforge):
    
    rsync error: some files/attrs were not transferred (see previous errors) (code 23)
    
    It seems the only way to mitigate that issue is to retry the rsync command
    until it succeeds.
    
    So add a rsync_retry decorator and apply it to a new method in the loader
    wrapping the call to subprocess.run executing the rsync command.
    
    Also use rsync option to compress file data during the transfer.

See https://jenkins.softwareheritage.org/job/DLDCVS/job/tests-on-diff/98/ for more details.

Harbormaster completed remote builds in B28488: Diff 27468.Apr 14 2022, 5:30 PM

anlambert requested review of this revision.Apr 14 2022, 5:30 PM

ardumont accepted this revision.Apr 14 2022, 5:35 PM

This revision is now accepted and ready to land.Apr 14 2022, 5:35 PM

Closed by commit rDLDCVS9f5456b6f0c3: loader: Fix rsync failures by retrying associated commands (authored by anlambert). · Explain WhyApr 14 2022, 5:40 PM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDLDCVS9f5456b6f0c3: loader: Fix rsync failures by retrying associated commands.

anlambert mentioned this in D8006: cvsclient: Retry pserver connection three times in case of failure.Jun 20 2022, 2:13 PM