Differential D8091

loader: Ensure to strip trailing slash from origin URL
ClosedPublic
Actions

Authored by anlambert on Jul 7 2022, 10:54 AM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDLDCVSe5215cf1983c: loader: Ensure to strip trailing slash from origin URL

Summary

It makes the loading process fail otherwise, see below:

rsync

(swh) ✔ ~/swh/swh-environment/swh-loader-cvs [master|⚑ 4] 
10:47 $ swh -l DEBUG loader -C ~/.config/swh/loader/cvs.yml run cvs rsync://a.cvs.sourceforge.net/cvsroot/smtpsvr/bbs/
WARNING:swh.core.cli:Could not load subcommand provenance: DistributionNotFound(Requirement.parse('zmq'), None)
DEBUG:swh.loader.cli:ctx: <click.core.Context object at 0x7f5a42b73280>
DEBUG:swh.core.config:Loading config file /home/anlambert/.config/swh/loader/cvs.yml
DEBUG:swh.loader.cli:config_file: /home/anlambert/.config/swh/loader/cvs.yml
DEBUG:swh.loader.cli:config: 
DEBUG:swh.loader.cli:kw: {}
DEBUG:swh.loader.cli:registry: {'task_modules': ['swh.loader.cvs.tasks'], 'loader': <class 'swh.loader.cvs.loader.CvsLoader'>}
DEBUG:swh.loader.cli:loader class: <class 'swh.loader.cvs.loader.CvsLoader'>
INFO:swh.loader.cvs.loader.CvsLoader:Load origin 'rsync://a.cvs.sourceforge.net/cvsroot/smtpsvr/bbs/' with type 'cvs'
DEBUG:swh.loader.cvs.loader.CvsLoader:lister_not provided, skipping extrinsic origin metadata
DEBUG:swh.core.statsd:Error submitting statsd packet. Dropping the packet and closing the socket.
DEBUG:swh.loader.cvs.loader.CvsLoader:prepare; origin_url=rsync://a.cvs.sourceforge.net/cvsroot/smtpsvr/bbs/ scheme=rsync path=/cvsroot/smtpsvr/bbs/
ERROR:swh.loader.cvs.loader.CvsLoader:Loading failure, updating to `failed` status
Traceback (most recent call last):
  File "/home/anlambert/swh/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 383, in load
    self.prepare()
  File "/home/anlambert/swh/swh-environment/swh-loader-cvs/swh/loader/cvs/loader.py", line 420, in prepare
    self.fetch_cvs_repo_with_rsync(url.hostname, url.path)
  File "/home/anlambert/swh/swh-environment/swh-loader-cvs/swh/loader/cvs/loader.py", line 358, in fetch_cvs_repo_with_rsync
    rsync = self.execute_rsync(
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 324, in wrapped_f
    return self(f, *args, **kw)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
    return fut.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 407, in __call__
    result = fn(*args, **kwargs)
  File "/home/anlambert/swh/swh-environment/swh-loader-cvs/swh/loader/cvs/loader.py", line 351, in execute_rsync
    rsync = subprocess.run(rsync_cmd, **run_opts)
  File "/usr/lib/python3.9/subprocess.py", line 507, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.9/subprocess.py", line 1134, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.9/subprocess.py", line 2019, in _communicate
    stdout = self._translate_newlines(stdout,
  File "/usr/lib/python3.9/subprocess.py", line 1011, in _translate_newlines
    data = data.decode(encoding, errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd6 in position 10738: ordinal not in range(128)
DEBUG:swh.core.statsd:Error submitting statsd packet. Dropping the packet and closing the socket.
DEBUG:swh.loader.cvs.loader.CvsLoader:cleanup
DEBUG:swh.core.statsd:Error submitting statsd packet. Dropping the packet and closing the socket.
{'status': 'failed'} for origin 'rsync://a.cvs.sourceforge.net/cvsroot/smtpsvr/bbs/'


(swh) ✔ ~/swh/swh-environment/swh-loader-cvs [master|⚑ 4] 
10:47 $ swh -l DEBUG loader -C ~/.config/swh/loader/cvs.yml run cvs rsync://a.cvs.sourceforge.net/cvsroot/smtpsvr/bbs
WARNING:swh.core.cli:Could not load subcommand provenance: DistributionNotFound(Requirement.parse('zmq'), None)
DEBUG:swh.loader.cli:ctx: <click.core.Context object at 0x7f9d503a0220>
DEBUG:swh.core.config:Loading config file /home/anlambert/.config/swh/loader/cvs.yml
DEBUG:swh.loader.cli:config_file: /home/anlambert/.config/swh/loader/cvs.yml
DEBUG:swh.loader.cli:config: 
DEBUG:swh.loader.cli:kw: {}
DEBUG:swh.loader.cli:registry: {'task_modules': ['swh.loader.cvs.tasks'], 'loader': <class 'swh.loader.cvs.loader.CvsLoader'>}
DEBUG:swh.loader.cli:loader class: <class 'swh.loader.cvs.loader.CvsLoader'>
INFO:swh.loader.cvs.loader.CvsLoader:Load origin 'rsync://a.cvs.sourceforge.net/cvsroot/smtpsvr/bbs' with type 'cvs'
DEBUG:swh.loader.cvs.loader.CvsLoader:lister_not provided, skipping extrinsic origin metadata
DEBUG:swh.core.statsd:Error submitting statsd packet. Dropping the packet and closing the socket.
DEBUG:swh.loader.cvs.loader.CvsLoader:prepare; origin_url=rsync://a.cvs.sourceforge.net/cvsroot/smtpsvr/bbs scheme=rsync path=/cvsroot/smtpsvr/bbs
DEBUG:swh.loader.cvs.loader.CvsLoader:rsync server: drwxr-xr-x              3 2011/09/14 10:24:48 .
DEBUG:swh.loader.cvs.loader.CvsLoader:rsync server: drwxr-xr-x             39 2011/09/14 10:24:48 CVSROOT
DEBUG:swh.loader.cvs.loader.CvsLoader:rsync server: drwxr-xr-x              2 2011/09/14 10:24:48 SMTPServer
DEBUG:swh.loader.cvs.loader.CvsLoader:rsync server: drwxr-xr-x            171 2011/09/14 10:24:48 bbs

pserver

(swh) ✔ ~/swh/swh-environment/swh-loader-cvs [master|⚑ 4] 
10:47 $ swh -l DEBUG loader -C ~/.config/swh/loader/cvs.yml run cvs pserver://anonymous@a.cvs.sourceforge.net/cvsroot/dynamicdraw/Doc/
WARNING:swh.core.cli:Could not load subcommand provenance: DistributionNotFound(Requirement.parse('zmq'), None)
DEBUG:swh.loader.cli:ctx: <click.core.Context object at 0x7f4f3776d820>
DEBUG:swh.core.config:Loading config file /home/anlambert/.config/swh/loader/cvs.yml
DEBUG:swh.loader.cli:config_file: /home/anlambert/.config/swh/loader/cvs.yml
DEBUG:swh.loader.cli:config: 
DEBUG:swh.loader.cli:kw: {}
DEBUG:swh.loader.cli:registry: {'task_modules': ['swh.loader.cvs.tasks'], 'loader': <class 'swh.loader.cvs.loader.CvsLoader'>}
DEBUG:swh.loader.cli:loader class: <class 'swh.loader.cvs.loader.CvsLoader'>
INFO:swh.loader.cvs.loader.CvsLoader:Load origin 'pserver://anonymous@a.cvs.sourceforge.net/cvsroot/dynamicdraw/Doc/' with type 'cvs'
DEBUG:swh.loader.cvs.loader.CvsLoader:lister_not provided, skipping extrinsic origin metadata
DEBUG:swh.core.statsd:Error submitting statsd packet. Dropping the packet and closing the socket.
DEBUG:swh.loader.cvs.loader.CvsLoader:prepare; origin_url=pserver://anonymous@a.cvs.sourceforge.net/cvsroot/dynamicdraw/Doc/ scheme=pserver path=/cvsroot/dynamicdraw/Doc/
Request: BEGIN AUTH REQUEST
/cvsroot/dynamicdraw/Doc
anonymous
A
END AUTH REQUEST


Request: BEGIN AUTH REQUEST
/cvsroot/dynamicdraw/Doc
anonymous
A
END AUTH REQUEST


Request: BEGIN AUTH REQUEST
/cvsroot/dynamicdraw/Doc
anonymous
A
END AUTH REQUEST


ERROR:swh.loader.cvs.loader.CvsLoader:Loading failure, updating to `not_found` status
Traceback (most recent call last):
  File "/home/anlambert/swh/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 383, in load
    self.prepare()
  File "/home/anlambert/swh/swh-environment/swh-loader-cvs/swh/loader/cvs/loader.py", line 492, in prepare
    self.cvsclient = CVSClient(url)
  File "/home/anlambert/swh/swh-environment/swh-loader-cvs/swh/loader/cvs/cvsclient.py", line 262, in __init__
    self.connect_pserver(url.hostname, url.port, url.username, url.password)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 324, in wrapped_f
    return self(f, *args, **kw)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter
    raise retry_exc.reraise()
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 193, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/tenacity/__init__.py", line 407, in __call__
    result = fn(*args, **kwargs)
  File "/home/anlambert/swh/swh-environment/swh-loader-cvs/swh/loader/cvs/cvsclient.py", line 150, in connect_pserver
    raise NotFound(
swh.loader.exception.NotFound: pserver authentication failed for a.cvs.sourceforge.net:2401: b'I HATE YOU\n'
DEBUG:swh.core.statsd:Error submitting statsd packet. Dropping the packet and closing the socket.
DEBUG:swh.loader.cvs.loader.CvsLoader:cleanup
DEBUG:swh.core.statsd:Error submitting statsd packet. Dropping the packet and closing the socket.
{'status': 'uneventful'} for origin 'pserver://anonymous@a.cvs.sourceforge.net/cvsroot/dynamicdraw/Doc/'


(swh) ✔ ~/swh/swh-environment/swh-loader-cvs [master|⚑ 4] 
10:48 $ swh -l DEBUG loader -C ~/.config/swh/loader/cvs.yml run cvs pserver://anonymous@a.cvs.sourceforge.net/cvsroot/dynamicdraw/Doc
WARNING:swh.core.cli:Could not load subcommand provenance: DistributionNotFound(Requirement.parse('zmq'), None)
DEBUG:swh.loader.cli:ctx: <click.core.Context object at 0x7fa1c81eb430>
DEBUG:swh.core.config:Loading config file /home/anlambert/.config/swh/loader/cvs.yml
DEBUG:swh.loader.cli:config_file: /home/anlambert/.config/swh/loader/cvs.yml
DEBUG:swh.loader.cli:config: 
DEBUG:swh.loader.cli:kw: {}
DEBUG:swh.loader.cli:registry: {'task_modules': ['swh.loader.cvs.tasks'], 'loader': <class 'swh.loader.cvs.loader.CvsLoader'>}
DEBUG:swh.loader.cli:loader class: <class 'swh.loader.cvs.loader.CvsLoader'>
INFO:swh.loader.cvs.loader.CvsLoader:Load origin 'pserver://anonymous@a.cvs.sourceforge.net/cvsroot/dynamicdraw/Doc' with type 'cvs'
DEBUG:swh.loader.cvs.loader.CvsLoader:lister_not provided, skipping extrinsic origin metadata
DEBUG:swh.core.statsd:Error submitting statsd packet. Dropping the packet and closing the socket.
DEBUG:swh.loader.cvs.loader.CvsLoader:prepare; origin_url=pserver://anonymous@a.cvs.sourceforge.net/cvsroot/dynamicdraw/Doc scheme=pserver path=/cvsroot/dynamicdraw/Doc
Request: BEGIN AUTH REQUEST
/cvsroot/dynamicdraw
anonymous
A
END AUTH REQUEST


Request: BEGIN AUTH REQUEST
/cvsroot/dynamicdraw
anonymous
A
END AUTH REQUEST


DEBUG:swh.loader.cvs.loader.CvsLoader:Fetching CVS rlog from a.cvs.sourceforge.net:/cvsroot/dynamicdraw/Doc
DEBUG:swh.loader.cvs.loader.CvsLoader:CVS changesets found for Doc: 32
DEBUG:swh.loader.cvs.loader.CvsLoader:changeset from jeu. 15 févr. 2007 06:56:52 by fukushirom on branch HEAD
DEBUG:swh.loader.cvs.loader.CvsLoader:rev 1.1 state Exp file b'/cvsroot/dynamicdraw/Doc/GraphLayout.mdpf,v'
DEBUG:swh.loader.cvs.loader.CvsLoader:checkout to b'/tmp/swh.loader.cvs.1jk17y96-899089/Doc/GraphLayout.mdpf'

Diff Detail

Repository

rDLDCVS CVS Loader

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Jul 7 2022, 10:54 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptJul 7 2022, 10:54 AM

anlambert edited the summary of this revision. (Show Details)Jul 7 2022, 10:55 AM

Build is green

Patch application report for D8091 (id=29202)

Rebasing onto d89f8d13e9...

Current branch diff-target is up to date.

Changes applied before test

commit e5215cf1983c9ca242c2e3226176935cd1db7ce5
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Jul 7 10:53:36 2022 +0200

    loader: Ensure to strip trailing slash from origin URL
    
    It makes the loading process fail otherwise.

See https://jenkins.softwareheritage.org/job/DLDCVS/job/tests-on-diff/118/ for more details.

Harbormaster completed remote builds in B30291: Diff 29202.Jul 7 2022, 10:57 AM

anlambert requested review of this revision.Jul 7 2022, 10:57 AM

ardumont accepted this revision.Jul 7 2022, 2:06 PM

This revision is now accepted and ready to land.Jul 7 2022, 2:06 PM

Closed by commit rDLDCVSe5215cf1983c: loader: Ensure to strip trailing slash from origin URL (authored by anlambert). · Explain WhyJul 7 2022, 2:09 PM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDLDCVSe5215cf1983c: loader: Ensure to strip trailing slash from origin URL.