Page MenuHomeSoftware Heritage
Differential D1010

Make metadata indexers store the mappings used to translate metadata.
ClosedPublic
Actions

Authored by vlorentz on Jan 25 2019, 3:38 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T1483: Store the type of intrinsic metadata that were extracted
Commits
rDCIDX8644756acd42: Make metadata indexers store the mappings used to translate metadata.
rDCIDXd0574b513ae0: Add 'mappings' column to the idx storage.
Summary

Depends on D986.
Resolves T1483.

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
store-metadata-source
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 3788
Build 4948: tox-on-jenkinsJenkins
Build 4947: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jan 25 2019, 3:38 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 25 2019, 3:38 PM
swh-public-ci added a subscriber: swh-public-ci.Jan 25 2019, 3:42 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/274/ for more details.

Harbormaster completed remote builds in B3729: Diff 3197.Jan 25 2019, 3:42 PM
ardumont requested changes to this revision.Jan 28 2019, 3:17 PM
ardumont added a subscriber: ardumont.

Aside from the 2 remarks, this sounds good to go.

swh/indexer/sql/30-swh-schema.sql
135

please, update the schema version and add the migration script that alters the schema.

swh/indexer/storage/__init__.py
618

please remove those print statements.

This revision now requires changes to proceed.Jan 28 2019, 3:17 PM
vlorentz marked 2 inline comments as done.Jan 28 2019, 3:38 PM
vlorentz updated this revision to Diff 3217.Jan 28 2019, 3:38 PM
  • Remove prints.
  • Add schema migration.
Harbormaster failed remote builds in B3754: Diff 3217!Jan 28 2019, 3:39 PM
swh-public-ci added a comment.Jan 28 2019, 3:39 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/277/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/277/console

ardumont accepted this revision.EditedJan 28 2019, 3:50 PM

beware the db version is wrong in the schema migration script.

sql/upgrades/119.sql
7

119 ;)

This revision is now accepted and ready to land.Jan 28 2019, 3:50 PM
vlorentz updated this revision to Diff 3218.Jan 28 2019, 3:58 PM
  • fix typo
vlorentz marked an inline comment as done.Jan 28 2019, 3:59 PM
Harbormaster failed remote builds in B3755: Diff 3218!Jan 28 2019, 3:59 PM
swh-public-ci added a comment.Jan 28 2019, 3:59 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/278/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/278/console

vlorentz mentioned this in T1483: Store the type of intrinsic metadata that were extracted.Jan 29 2019, 10:58 AM
vlorentz updated this revision to Diff 3243.Jan 29 2019, 12:20 PM
  • Rebase
Closed by commit rDCIDXd0574b513ae0: Add 'mappings' column to the idx storage. (authored by vlorentz). · Explain WhyJan 29 2019, 12:20 PM
This revision was automatically updated to reflect the committed changes.
Harbormaster failed remote builds in B3788: Diff 3243!Jan 29 2019, 12:20 PM
swh-public-ci added a comment.Jan 29 2019, 12:20 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/289/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/289/console

Revision Contents
Changeset List

PathSize
sql/
upgrades/
19 lines
swh/
indexer/
16 lines
16 lines
sql/
8 lines
16 lines
storage/
15 lines
7 lines
tests/
storage/
27 lines
3 lines
2 lines

Diff 3243

View Options

sql/upgrades/119.sql

Loading...
View Options

swh/indexer/metadata.py

Loading...
View Options

swh/indexer/metadata_dictionary.py

Loading...
View Options

swh/indexer/sql/30-swh-schema.sql

Loading...
View Options

swh/indexer/sql/40-swh-func.sql

Loading...
View Options

swh/indexer/storage/__init__.py

Loading...
View Options

swh/indexer/storage/db.py

Loading...
View Options

swh/indexer/tests/storage/test_storage.py

Loading...
View Options

swh/indexer/tests/test_metadata.py

Loading...
View Options

swh/indexer/tests/test_origin_metadata.py

Loading...
About · Developer information · Legal