Page MenuHomeSoftware Heritage
Differential D7996

loader: Bump dulwich and remove no longer valid comments
ClosedPublic
Actions

Authored by anlambert on Jun 16 2022, 1:47 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4311: Package and deploy dulwich 0.20.43 in production
Commits
rDLDG38537e49d37a: loader: Bump dulwich and remove no longer valid comments
Summary

Dulwich 0.20.43 dropped the double caching of HTTP responses so
we can now remove comments about that issue.

Related to T4311

Diff Detail

Repository
rDLDG Git loader
Branch
dulwich-bump
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 29900
Build 46739: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 46738: arc lint + arc unit

Event Timeline

anlambert created this revision.Jun 16 2022, 1:47 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 16 2022, 1:47 PM
swh-public-ci added a comment.Jun 16 2022, 1:50 PM

Build is green

Patch application report for D7996 (id=28814)

Rebasing onto 44e35e4e24...

Current branch diff-target is up to date.
Changes applied before test
commit 38537e49d37a3313a0afa90159f2cb290e6995b7
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Jun 16 13:45:39 2022 +0200

    loader: Bump dulwich and remove no longer valid comments
    
    Dulwich 0.20.43 dropped the double caching of HTTP responses so
    we can now remove comments about that issue.
    
    Related to T4311

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/225/ for more details.

Harbormaster completed remote builds in B29900: Diff 28814.Jun 16 2022, 1:50 PM
anlambert requested review of this revision.Jun 16 2022, 1:50 PM
ardumont accepted this revision.Jun 16 2022, 2:01 PM
This revision is now accepted and ready to land.Jun 16 2022, 2:01 PM
Closed by commit rDLDG38537e49d37a: loader: Bump dulwich and remove no longer valid comments (authored by anlambert). · Explain WhyJun 16 2022, 2:02 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDLDG38537e49d37a: loader: Bump dulwich and remove no longer valid comments.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
2 lines--
swh/
loader/
git/
20 lines85%
Loading...

Diff 28814

View Options

requirements.txt

Show First 20 Lines • Show All 240 Lines • ▼ Show 20 Linesclass MavenMapping(DictMapping, SingleFileMapping):
""" """
filename = b'pom.xml' filename = b'pom.xml'
mapping = CROSSWALK_TABLE['Java (Maven)'] mapping = CROSSWALK_TABLE['Java (Maven)']
def translate(self, content): def translate(self, content):
d = xmltodict.parse(content)['project'] d = xmltodict.parse(content)['project']
metadata = self.translate_dict(d, normalize=False) metadata = self.translate_dict(d, normalize=False)
metadata[SCHEMA_URI+'codeRepository'] = self.parse_repositories(d) metadata[SCHEMA_URI+'codeRepository'] = self.parse_repositories(d)
metadata[SCHEMA_URI+'license'] = self.parse_licenses(d)
return self.normalize_translation(metadata) return self.normalize_translation(metadata)
_default_repository = {'url': 'https://repo.maven.apache.org/maven2/'} _default_repository = {'url': 'https://repo.maven.apache.org/maven2/'}
def parse_repositories(self, d): def parse_repositories(self, d):
"""https://maven.apache.org/pom.html#Repositories""" """https://maven.apache.org/pom.html#Repositories"""
if 'repositories' not in d: if 'repositories' not in d:
return [self.parse_repository(d, self._default_repository)] return [self.parse_repository(d, self._default_repository)]
Show All 16 Lines def parse_repository(self, d, repo):
url = os.path.join(url, *d['groupId'].split('.')) url = os.path.join(url, *d['groupId'].split('.'))
if d['artifactId']: if d['artifactId']:
url = os.path.join(url, d['artifactId']) url = os.path.join(url, d['artifactId'])
return {"@id": url} return {"@id": url}
def normalize_groupId(self, id_): def normalize_groupId(self, id_):
return {"@id": id_} return {"@id": id_}
def parse_licenses(self, d):
"""https://maven.apache.org/pom.html#Licenses
zackUnsubmitted
Done
Inline Actions

What happens when multiple <license> elements are present?

It would be nice to have:

  • a dedicated test case for that, and
  • a description of the general semantics of how that is handled in the docstirng of this method
zack: What happens when multiple <license> elements are present? It would be nice to have: - a…
vlorentzAuthorUnsubmitted
Done
Inline Actions

This method always returns a list, so there's nothing interesting here. However, JSON-LD compaction happens later, so I added tests for that.

vlorentz: This method always returns a list, so there's nothing interesting here. However, JSON-LD…
zackUnsubmitted
Not Done
Inline Actions

The fact that I wanted documented (in the docstring, type annotation, whatever) was precisely that it always returns a list :-)

zack: The fact that I wanted documented (in the docstring, type annotation, whatever) was precisely…
vlorentzAuthorUnsubmitted
Done
Inline Actions

True. I added a task to remember to document all these methods further: T1403

vlorentz: True. I added a task to remember to document all these methods further: T1403
The origin XML has the form:
<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
</licenses>
Which was translated to a dict by xmltodict and is given as `d`:
>>> from pprint import pprint
>>> d = {
... # ...
... "licenses": {
... "license": {
... "name": "Apache License, Version 2.0",
... "url":
... "https://www.apache.org/licenses/LICENSE-2.0.txt"
... }
... }
... }
>>> MavenMapping().parse_licenses(d)
[{'@id': 'https://www.apache.org/licenses/LICENSE-2.0.txt'}]
>>> d = {
... # ...
... "licenses": {
... "license": [
... {
... "name": "Apache License, Version 2.0",
... "url":
... "https://www.apache.org/licenses/LICENSE-2.0.txt"
... },
... {
... "name": "MIT License, ",
... "url": "https://opensource.org/licenses/MIT"
... }
... ]
... }
... }
>>> pprint(MavenMapping().parse_licenses(d))
[{'@id': 'https://www.apache.org/licenses/LICENSE-2.0.txt'},
{'@id': 'https://opensource.org/licenses/MIT'}]
or, if there are more than one license:
"""
licenses = d.get('licenses', {}).get('license', [])
if isinstance(licenses, dict):
licenses = [licenses]
return [{"@id": license['url']} for license in licenses]
def main(): def main():
raw_content = """{"name": "test_name", "unknown_term": "ut"}""" raw_content = """{"name": "test_name", "unknown_term": "ut"}"""
raw_content1 = b"""{"name": "test_name", raw_content1 = b"""{"name": "test_name",
"unknown_term": "ut", "unknown_term": "ut",
"prerequisites" :"packageXYZ"}""" "prerequisites" :"packageXYZ"}"""
result = MAPPINGS["NpmMapping"].translate(raw_content) result = MAPPINGS["NpmMapping"].translate(raw_content)
result1 = MAPPINGS["MavenMapping"].translate(raw_content1) result1 = MAPPINGS["MavenMapping"].translate(raw_content1)
print(result) print(result)
print(result1) print(result1)
if __name__ == "__main__": if __name__ == "__main__":
main() main()
View Options

swh/loader/git/loader.py

Show First 20 Lines • Show All 240 Lines • ▼ Show 20 Linesclass MavenMapping(DictMapping, SingleFileMapping):
""" """
filename = b'pom.xml' filename = b'pom.xml'
mapping = CROSSWALK_TABLE['Java (Maven)'] mapping = CROSSWALK_TABLE['Java (Maven)']
def translate(self, content): def translate(self, content):
d = xmltodict.parse(content)['project'] d = xmltodict.parse(content)['project']
metadata = self.translate_dict(d, normalize=False) metadata = self.translate_dict(d, normalize=False)
metadata[SCHEMA_URI+'codeRepository'] = self.parse_repositories(d) metadata[SCHEMA_URI+'codeRepository'] = self.parse_repositories(d)
metadata[SCHEMA_URI+'license'] = self.parse_licenses(d)
return self.normalize_translation(metadata) return self.normalize_translation(metadata)
_default_repository = {'url': 'https://repo.maven.apache.org/maven2/'} _default_repository = {'url': 'https://repo.maven.apache.org/maven2/'}
def parse_repositories(self, d): def parse_repositories(self, d):
"""https://maven.apache.org/pom.html#Repositories""" """https://maven.apache.org/pom.html#Repositories"""
if 'repositories' not in d: if 'repositories' not in d:
return [self.parse_repository(d, self._default_repository)] return [self.parse_repository(d, self._default_repository)]
Show All 16 Lines def parse_repository(self, d, repo):
url = os.path.join(url, *d['groupId'].split('.')) url = os.path.join(url, *d['groupId'].split('.'))
if d['artifactId']: if d['artifactId']:
url = os.path.join(url, d['artifactId']) url = os.path.join(url, d['artifactId'])
return {"@id": url} return {"@id": url}
def normalize_groupId(self, id_): def normalize_groupId(self, id_):
return {"@id": id_} return {"@id": id_}
def parse_licenses(self, d):
"""https://maven.apache.org/pom.html#Licenses
zackUnsubmitted
Done
Inline Actions

What happens when multiple <license> elements are present?

It would be nice to have:

  • a dedicated test case for that, and
  • a description of the general semantics of how that is handled in the docstirng of this method
zack: What happens when multiple <license> elements are present? It would be nice to have: - a…
vlorentzAuthorUnsubmitted
Done
Inline Actions

This method always returns a list, so there's nothing interesting here. However, JSON-LD compaction happens later, so I added tests for that.

vlorentz: This method always returns a list, so there's nothing interesting here. However, JSON-LD…
zackUnsubmitted
Not Done
Inline Actions

The fact that I wanted documented (in the docstring, type annotation, whatever) was precisely that it always returns a list :-)

zack: The fact that I wanted documented (in the docstring, type annotation, whatever) was precisely…
vlorentzAuthorUnsubmitted
Done
Inline Actions

True. I added a task to remember to document all these methods further: T1403

vlorentz: True. I added a task to remember to document all these methods further: T1403
The origin XML has the form:
<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
</licenses>
Which was translated to a dict by xmltodict and is given as `d`:
>>> from pprint import pprint
>>> d = {
... # ...
... "licenses": {
... "license": {
... "name": "Apache License, Version 2.0",
... "url":
... "https://www.apache.org/licenses/LICENSE-2.0.txt"
... }
... }
... }
>>> MavenMapping().parse_licenses(d)
[{'@id': 'https://www.apache.org/licenses/LICENSE-2.0.txt'}]
>>> d = {
... # ...
... "licenses": {
... "license": [
... {
... "name": "Apache License, Version 2.0",
... "url":
... "https://www.apache.org/licenses/LICENSE-2.0.txt"
... },
... {
... "name": "MIT License, ",
... "url": "https://opensource.org/licenses/MIT"
... }
... ]
... }
... }
>>> pprint(MavenMapping().parse_licenses(d))
[{'@id': 'https://www.apache.org/licenses/LICENSE-2.0.txt'},
{'@id': 'https://opensource.org/licenses/MIT'}]
or, if there are more than one license:
"""
licenses = d.get('licenses', {}).get('license', [])
if isinstance(licenses, dict):
licenses = [licenses]
return [{"@id": license['url']} for license in licenses]
def main(): def main():
raw_content = """{"name": "test_name", "unknown_term": "ut"}""" raw_content = """{"name": "test_name", "unknown_term": "ut"}"""
raw_content1 = b"""{"name": "test_name", raw_content1 = b"""{"name": "test_name",
"unknown_term": "ut", "unknown_term": "ut",
"prerequisites" :"packageXYZ"}""" "prerequisites" :"packageXYZ"}"""
result = MAPPINGS["NpmMapping"].translate(raw_content) result = MAPPINGS["NpmMapping"].translate(raw_content)
result1 = MAPPINGS["MavenMapping"].translate(raw_content1) result1 = MAPPINGS["MavenMapping"].translate(raw_content1)
print(result) print(result)
print(result1) print(result1)
if __name__ == "__main__": if __name__ == "__main__":
main() main()
About · Developer information · Legal