Page MenuHomeSoftware Heritage
Differential D620

Translate from pom.xml and codemeta.json.
ClosedPublic
Actions

Authored by vlorentz on Oct 30 2018, 12:07 PM.

Details

Reviewers
moranegg
Maniphest Tasks
T1289: CodeMeta support in the metadata indexer
Commits
rDCIDX0a491f8d2781: Translate from pom.xml and codemeta.json.
Summary

Also changes slightly the output format to provide
a @type and make it more compact.

Resolves T1289.

Test Plan

tox

Diff Detail

Repository
rDCIDX Metadata indexer
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Oct 30 2018, 12:07 PM
Harbormaster completed remote builds in B1994: Diff 1906.Oct 30 2018, 12:07 PM
vlorentz added a parent revision: D619: Translate authors from package.json.Oct 30 2018, 12:08 PM
vlorentz edited the summary of this revision. (Show Details)Oct 30 2018, 1:25 PM
vlorentz added a task: T1289: CodeMeta support in the metadata indexer.
moranegg added a comment.Oct 31 2018, 4:00 PM

Add README or CITATION to data directory with the following:
Matthew B. Jones, Carl Boettiger, Abby Cabunoc Mayes, Arfon Smith, Peter Slaughter, Kyle Niemeyer, Yolanda Gil, Martin Fenner, Krzysztof Nowak, Mark Hahnel, Luke Coy, Alice Allen, Mercè Crosas, Ashley Sands, Neil Chue Hong, Patricia Cruse, Daniel S. Katz, Carole Goble. 2017. CodeMeta: an exchange schema for software metadata. Version 2.0. KNB Data Repository. doi:10.5063/schema/codemeta-2.0
swh:1:dir:39c509fd2002f9e531fb4b3a321ceb5e6994e54a;origin=https://github.com/codemeta/codemeta

swh/indexer/codemeta.py
65

The codemeta-V1 is the older version of codemeta.
The difficulty (not here but globaly) if we find a codemeta.json file, we need to use the @context attribute to see which version is used,
but this could also cause problems because of examples like this:
"@context": "https://raw.githubusercontent.com/codemeta/codemeta/master/codemeta.jsonld",

So here, the column codemeta-V1 shouldn't be the canonical name for codemeta.
The codemeta vocabulary is in the codemeta.csv table under property
and to facilitate things at present, I think when encountering a codemeta.json file, the vocabulary to check should be in the property column

vlorentz marked an inline comment as done.Oct 31 2018, 5:42 PM
vlorentz added inline comments.
swh/indexer/codemeta.py
65

Yeah I understood that later, there's a fix in D620. Adding support for v1 remains to be done though

moranegg requested changes to this revision.Nov 5 2018, 1:16 PM

There is no test a case of a revision with multiple 'metadata files' which is an intriguing case - this should be tested before accepting this diff.

Global comments that are not parts of this diff:

  1. in the revision_metadata output there isn't a list of file ids used for the translation, do we want to keep it?
  2. I would suggest before continuing to the next vocabulary, adding to the detected files list the AUTHOR/AUTHORS/CONTRIBUTORS file to keep in the authors property.
  3. Other ideas would be :
    • detecting LICENSE/ COPYING/... and using the result of the fossology_license indexer
    • detecting README and using a fixed portion of it in the description property
swh/indexer/metadata_detector.py
58

I see it does become a list when multiple values are given for the same term.
Would love to see it tested :-)

swh/indexer/tests/test_metadata.py
241–242

I kept a property called other in the content_metadata to regroup all metadata I wasn't able to translate to CodeMeta.
I see that you deleted this property, was it problematic with the type of output- keeping a codemeta.json output?

306

When running the indexation of a metadata file with a wrong @context, even with the same url with 1.0 at the end,
it fails on the following error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pyld/jsonld.py", line 4308, in _retrieve_context_urls
    remote_doc = load_document(url)
  File "/home/morane/Documents/code/swh-environment/swh-indexer/swh/indexer/codemeta.py", line 108, in _document_loader
    raise Exception(url)
Exception: https://doi.org/10.5063/schema/codemeta-1.0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pyld/jsonld.py", line 800, in expand
    input_, {}, options['documentLoader'], options['base'])
  File "/usr/lib/python3/dist-packages/pyld/jsonld.py", line 4315, in _retrieve_context_urls
    code='loading remote context failed', cause=cause)
pyld.jsonld.JsonLdError: <exception str() failed>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "metadata_dictionary.py", line 345, in <module>
    main()
  File "metadata_dictionary.py", line 335, in main
    result = MAPPINGS["CodemetaMapping"].translate(raw_content)
  File "metadata_dictionary.py", line 226, in translate
    return self.normalize_translation(expand(json.loads(content.decode())))
  File "/home/morane/Documents/code/swh-environment/swh-indexer/swh/indexer/codemeta.py", line 120, in expand
    options={'documentLoader': _document_loader})
  File "/usr/lib/python3/dist-packages/pyld/jsonld.py", line 171, in expand
    return JsonLdProcessor().expand(input_, options)
  File "/usr/lib/python3/dist-packages/pyld/jsonld.py", line 804, in expand
    'jsonld.ExpandError', cause=cause)
pyld.jsonld.JsonLdError: <exception str() failed>

This is an observation, I'm not saying we should fix it, but the usage of the DOI url might not be on all codemeta.json files we find.

461–462

I'm not sure where to write this comment, but when you have multiple detected files, what happens with the translated_metadata output?
does each property become a list?

swh/indexer/tests/test_origin_metadata.py
105

why the context name is here in the property?
looks like it's only a Mock test issue, but if so, you should change the tests to reflect exactly what output you are looking for.
Also, it should be consistent (on all properties).

swh/indexer/tests/test_utils.py
301–303

Here again the context is in the property, why?
is it because this property is inherited from schema.org?
or is it because in the content_get from the MockStorage it was already there

This revision now requires changes to proceed.Nov 5 2018, 1:16 PM
vlorentz marked 6 inline comments as done.Nov 5 2018, 1:25 PM
vlorentz added inline comments.
swh/indexer/metadata_detector.py
58

That's the same behavior as before; it's already tested in test_extract_minimal_metadata_dict.

swh/indexer/tests/test_metadata.py
241–242

It's just that it's not defined by codemeta's schema definition, so jsonld.compact drops it. I could add a new property with an absolute URI, though.

306

Unfortunately, it's either that or pulling untrusted schemas from the internet :/

461–462

does each property become a list?

Yes. Actually, they are all lists at the beginning, but JSON-LD compaction reduces them to their element.

swh/indexer/tests/test_origin_metadata.py
105

That's because codemeta defined it to be the @id: "issueTracker": { "@id":"codemeta:issueTracker", "@type": "@id" },

swh/indexer/tests/test_utils.py
301–303

is it because this property is inherited from schema.org?

On the contrary, it's because codemeta:author is not the same as schema:author (because they defined it as "author": { "@id": "schema:author", "@container": "@list" }, instead of "author": { "@id": "schema:author" },), so referring only to author is ambiguous.

vlorentz updated this revision to Diff 1936.Nov 5 2018, 4:04 PM
  • Rebase
Harbormaster failed remote builds in B2039: Diff 1936!Nov 5 2018, 4:08 PM
swh-public-ci added a subscriber: swh-public-ci.Nov 5 2018, 4:08 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/4/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/4/console

vlorentz updated this revision to Diff 1940.Nov 5 2018, 4:12 PM
  • Rebase
swh-public-ci added a comment.Nov 5 2018, 4:20 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/8/ for more details.

Harbormaster completed remote builds in B2043: Diff 1940.Nov 5 2018, 4:20 PM
moranegg accepted this revision.Nov 5 2018, 5:30 PM
This revision is now accepted and ready to land.Nov 5 2018, 5:30 PM
moranegg mentioned this in D617: Always output valid JSON-LD..Nov 5 2018, 5:31 PM
moranegg mentioned this in D619: Translate authors from package.json.
Closed by commit rDCIDX0a491f8d2781: Translate from pom.xml and codemeta.json. (authored by vlorentz). · Explain WhyNov 5 2018, 5:37 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
1 line
swh/
indexer/
27 lines
6 lines
100 lines
tests/
272 lines
16 lines
19 lines

Diff 1946

View Options

requirements.txt

Loading...
View Options

swh/indexer/codemeta.py

Loading...
View Options

swh/indexer/metadata_detector.py

Loading...
View Options

swh/indexer/metadata_dictionary.py

Loading...
View Options

swh/indexer/tests/test_metadata.py

Loading...
View Options

swh/indexer/tests/test_origin_metadata.py

Loading...
View Options

swh/indexer/tests/test_utils.py

Loading...
About · Developer information · Legal