Page MenuHomeSoftware Heritage
Differential D1226

Fix heterogeneity of names in metadata tables
ClosedPublic
Actions

Authored by haltode on Mar 7 2019, 8:16 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T1561: Fix heterogeneity of names in metadata tables
Commits
rDCIDX4f6ab3c9ab17: Fix heterogeneity of names in metadata tables
Summary

Fixes T1561:

  • revision_metadata renamed to revision_intrinsic_metadata,
  • revision_metadata.translated_metadata renamed to revision_metadata.metadata,
  • content_metadata.translated_metadata renamed to content_metadata.metadata,
  • origin_intrinsic_metadata.origin_id renamed to origin_intrinsic_metadata.id.
  • rename/update functions and affected tests
  • add database migration script

Diff Detail

Repository
rDCIDX Metadata indexer
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

haltode created this revision.Mar 7 2019, 8:16 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 7 2019, 8:16 AM
Harbormaster failed remote builds in B4552: Diff 3841!Mar 7 2019, 8:16 AM
swh-public-ci added a subscriber: swh-public-ci.Mar 7 2019, 8:16 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/485/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/485/console

vlorentz added a subscriber: vlorentz.Mar 7 2019, 9:14 AM

all of which because of the same error).

Looks like Jenkins could not apply your patch. You should rebase your changes on the latest master and arc diff --update D1226

haltode updated this revision to Diff 3842.Mar 7 2019, 9:46 AM

Rebasing on master

swh-public-ci added a comment.Mar 7 2019, 9:49 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/487/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/487/console

Harbormaster failed remote builds in B4553: Diff 3842!Mar 7 2019, 9:49 AM
haltode updated this revision to Diff 3890.EditedMar 12 2019, 5:06 AM

Fix failing tests and rename content_metadata.translated_metadata column.

haltode retitled this revision from [WIP][DO NOT MERGE] Fix heterogeneity of names in metadata tables to Fix heterogeneity of names in metadata tables.Mar 12 2019, 5:08 AM
haltode edited the summary of this revision. (Show Details)
haltode added a task: T1561: Fix heterogeneity of names in metadata tables.
swh-public-ci added a comment.Mar 12 2019, 5:09 AM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/490/ for more details.

Harbormaster completed remote builds in B4595: Diff 3890.Mar 12 2019, 5:09 AM
vlorentz added a comment.Mar 12 2019, 10:04 AM

Cool, thanks!

There are some little things to change (see inline comments), but other than that it's good!

sql/upgrades/123.sql
13

You should add migrations for the functions as well (just copy-paste the affected create or replace function statements).

swh/indexer/storage/__init__.py
726–727

You can change _check_duplicates to remove the second argument, it's no longer needed now.

swh/indexer/storage/db.py
380–383

Can be rewritten as a generator expression, which I find more readable:

keys = (convert_key(col, 'oim') for col in self.origin_intrinsic_metadata_cols)

(same below)

381

debugging leftover? ^^

swh/indexer/storage/in_memory.py
620–621

That line (and other similar ones) can be dropped now

haltode updated this revision to Diff 3893.Mar 12 2019, 12:09 PM

Updating D1226 concerning comments

swh-public-ci added a comment.Mar 12 2019, 12:14 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/492/ for more details.

Harbormaster completed remote builds in B4598: Diff 3893.Mar 12 2019, 12:14 PM
vlorentz added inline comments.Mar 12 2019, 12:25 PM
sql/upgrades/123.sql
9–10

The code of these functions must be updated as well

swh/indexer/storage/in_memory.py
620–621

Sorry, one more nitpick: the for loop can be replaced by: yield from self._origin_intrinsic_metadata.get(ids)

haltode updated this revision to Diff 3894.Mar 12 2019, 3:41 PM
haltode marked 6 inline comments as done.

Update functions code during migration + More pythonic yield statement

swh-public-ci added a comment.Mar 12 2019, 3:46 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/493/ for more details.

Harbormaster completed remote builds in B4599: Diff 3894.Mar 12 2019, 3:46 PM
vlorentz added a comment.Mar 12 2019, 4:39 PM

Migration of swh_content_metadata_add and swh_origin_intrinsic_metadata_add are still missing. Oh, and I just noticed that the name change of index revision_intrinsic_metadata_pkey is also missing

haltode updated this revision to Diff 3897.Mar 13 2019, 3:38 AM
haltode marked 2 inline comments as done.

Add missing functions/index migrations.

swh-public-ci added a comment.Mar 13 2019, 3:42 AM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/494/ for more details.

Harbormaster completed remote builds in B4612: Diff 3897.Mar 13 2019, 3:42 AM
haltode edited the summary of this revision. (Show Details)Mar 13 2019, 4:47 AM
vlorentz accepted this revision.Mar 13 2019, 9:48 AM
This revision is now accepted and ready to land.Mar 13 2019, 9:48 AM
haltode updated this revision to Diff 3907.Mar 13 2019, 1:21 PM

Rebasing on master

Closed by commit rDCIDX4f6ab3c9ab17: Fix heterogeneity of names in metadata tables (authored by haltode). · Explain WhyMar 13 2019, 1:22 PM
This revision was automatically updated to reflect the committed changes.
swh-public-ci added a comment.Mar 13 2019, 1:26 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/495/ for more details.

Harbormaster completed remote builds in B4622: Diff 3907.Mar 13 2019, 1:26 PM

Revision Contents
Changeset List

PathSize
sql/
upgrades/
103 lines
swh/
indexer/
61 lines
sql/
28 lines
59 lines
14 lines
storage/
87 lines
58 lines
57 lines
tests/
2 lines
storage/
4 lines
162 lines
2 lines
4 lines
13 lines
2 lines
29 lines

Diff 3909

View Options

sql/upgrades/123.sql

Loading...
View Options

swh/indexer/metadata.py

Loading...
View Options

swh/indexer/sql/30-swh-schema.sql

Loading...
View Options

swh/indexer/sql/40-swh-func.sql

Loading...
View Options

swh/indexer/sql/60-swh-indexes.sql

Loading...
View Options

swh/indexer/storage/__init__.py

Loading...
View Options

swh/indexer/storage/db.py

Loading...
View Options

swh/indexer/storage/in_memory.py

Loading...
View Options

swh/indexer/tests/conftest.py

Loading...
View Options

swh/indexer/tests/storage/test_converters.py

Loading...
View Options

swh/indexer/tests/storage/test_storage.py

Loading...
View Options

swh/indexer/tests/tasks.py

Loading...
View Options

swh/indexer/tests/test_cli.py

Loading...
View Options

swh/indexer/tests/test_metadata.py

Loading...
View Options

swh/indexer/tests/test_origin_head.py

Loading...
View Options

swh/indexer/tests/test_origin_metadata.py

Loading...
About · Developer information · Legal