Page MenuHomeSoftware Heritage
Differential D591

Make mappings into a hierarchy of classes that can be easily extended.
ClosedPublic
Actions

Authored by vlorentz on Oct 24 2018, 6:37 PM.

Details

Reviewers
moranegg
Group Reviewers
Reviewers
Maniphest Tasks
T1235: Refactor metadata detector tool to add easily new file names to detect
Commits
rDCIDX86695b3f6464: Make mappings into a hierarchy of classes that can be easily extended.
Summary

Second part of T1236, and also resolves T1235.

Diff Detail

Repository
rDCIDX Metadata indexer
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Oct 24 2018, 6:37 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 24 2018, 6:37 PM
vlorentz planned changes to this revision.Oct 24 2018, 6:37 PM
Harbormaster completed remote builds in B1885: Diff 1806.Oct 24 2018, 6:37 PM
vlorentz retitled this revision from Summary: to Implement all mappings.Oct 24 2018, 6:37 PM
vlorentz added a parent revision: D587: Include and load codemeta's crosswalk table..
vlorentz retitled this revision from Implement all mappings to Make mappings into a hierarchy of classes that can be easily extended..
vlorentz edited the summary of this revision. (Show Details)Oct 24 2018, 6:39 PM
vlorentz added a task: T1235: Refactor metadata detector tool to add easily new file names to detect.
vlorentz updated this revision to Diff 1809.Oct 24 2018, 6:44 PM
  • Comments.
Harbormaster completed remote builds in B1889: Diff 1809.Oct 24 2018, 6:44 PM
vlorentz removed a parent revision: D587: Include and load codemeta's crosswalk table..Oct 24 2018, 6:46 PM
vlorentz added a parent revision: D587: Include and load codemeta's crosswalk table..
vlorentz added a reviewer: moranegg.
moranegg added a comment.Oct 25 2018, 11:44 AM

I need some more time to play with this diff before accepting it

swh/indexer/data/codemeta/crosswalk.csv
1 ↗(On Diff #1809)

We should reference this file with a citation, maybe in the docs.
I should add that to my task T794

swh/indexer/metadata_dictionary.py
71

This is a really nice way to do it

vlorentz updated this revision to Diff 1813.Oct 25 2018, 11:44 AM
This comment was removed by vlorentz.
Harbormaster completed remote builds in B1892: Diff 1813.Oct 25 2018, 11:44 AM
vlorentz updated this revision to Diff 1814.Oct 25 2018, 11:45 AM
This comment was removed by vlorentz.
Harbormaster completed remote builds in B1893: Diff 1814.Oct 25 2018, 11:45 AM
vlorentz marked 2 inline comments as done.Oct 25 2018, 11:46 AM
vlorentz added inline comments.
swh/indexer/data/codemeta/crosswalk.csv
1 ↗(On Diff #1809)

Let's add a new file swh/indexer/data/codemeta/README?

moranegg mentioned this in P323 Codemeta test.Oct 25 2018, 12:10 PM
moranegg accepted this revision.Oct 25 2018, 1:51 PM
moranegg added a subscriber: ardumont.

I'm accepting this diff- it looks great.
Nevertheless, I think there should be at least one test for CodeMeta (even without the json-ld resolution)
and I added a comment about the tool version.

Concerning the citation/documentation of CodeMeta, I will add this information in a README, in the data directory as you suggested, next week.

Anyway I think you are ready for deployment, which is better to do today than tomorrow, especially because @ardumont is on leave next week.

swh/indexer/sql/50-swh-data.sql
21

the version in the diff is 0.0.2
as we never deployed the metadata indexers in production, I think we don't need to keep entries of '0.0.1'

This revision is now accepted and ready to land.Oct 25 2018, 1:51 PM
moranegg mentioned this in T1237: Update mappings to CodeMeta crosswalk table .Oct 25 2018, 1:55 PM
vlorentz updated this revision to Diff 1824.Oct 25 2018, 2:21 PM
vlorentz marked an inline comment as done.
  • Rebase
Harbormaster completed remote builds in B1908: Diff 1824.Oct 25 2018, 2:21 PM
Closed by commit rDCIDX86695b3f6464: Make mappings into a hierarchy of classes that can be easily extended. (authored by vlorentz). · Explain WhyOct 25 2018, 2:22 PM
This revision was automatically updated to reflect the committed changes.
vlorentz mentioned this in T1236: Refactor metadata translator to parse different types of files.Oct 25 2018, 2:23 PM

Revision Contents
Changeset List

PathSize
swh/
indexer/
9 lines
18 lines
187 lines
sql/
4 lines
tests/
storage/
2 lines
13 lines
6 lines

Diff 1825

View Options

swh/indexer/metadata.py

Loading...
View Options

swh/indexer/metadata_detector.py

Loading...
View Options

swh/indexer/metadata_dictionary.py

Loading...
View Options

swh/indexer/sql/50-swh-data.sql

Loading...
View Options

swh/indexer/tests/storage/test_storage.py

Loading...
View Options

swh/indexer/tests/test_metadata.py

Loading...
View Options

swh/indexer/tests/test_utils.py

Loading...
About · Developer information · Legal