Page MenuHomeSoftware Heritage

Fix heterogeneity of names in metadata tables
ClosedPublic

Authored by haltode on Mar 7 2019, 8:16 AM.

Details

Summary

Fixes T1561:

  • revision_metadata renamed to revision_intrinsic_metadata,
  • revision_metadata.translated_metadata renamed to revision_metadata.metadata,
  • content_metadata.translated_metadata renamed to content_metadata.metadata,
  • origin_intrinsic_metadata.origin_id renamed to origin_intrinsic_metadata.id.
  • rename/update functions and affected tests
  • add database migration script

Diff Detail

Repository
rDCIDX Metadata indexer
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

haltode created this revision.Mar 7 2019, 8:16 AM

all of which because of the same error).

Looks like Jenkins could not apply your patch. You should rebase your changes on the latest master and arc diff --update D1226

haltode updated this revision to Diff 3842.Mar 7 2019, 9:46 AM

Rebasing on master

haltode updated this revision to Diff 3890.EditedMar 12 2019, 5:06 AM

Fix failing tests and rename content_metadata.translated_metadata column.

haltode retitled this revision from [WIP][DO NOT MERGE] Fix heterogeneity of names in metadata tables to Fix heterogeneity of names in metadata tables.Mar 12 2019, 5:08 AM
haltode edited the summary of this revision. (Show Details)

Cool, thanks!

There are some little things to change (see inline comments), but other than that it's good!

sql/upgrades/123.sql
14

You should add migrations for the functions as well (just copy-paste the affected create or replace function statements).

swh/indexer/storage/__init__.py
726–727

You can change _check_duplicates to remove the second argument, it's no longer needed now.

swh/indexer/storage/db.py
380–383

Can be rewritten as a generator expression, which I find more readable:

keys = (convert_key(col, 'oim') for col in self.origin_intrinsic_metadata_cols)

(same below)

382

debugging leftover? ^^

swh/indexer/storage/in_memory.py
620–621

That line (and other similar ones) can be dropped now

haltode updated this revision to Diff 3893.Mar 12 2019, 12:09 PM

Updating D1226 concerning comments

vlorentz added inline comments.Mar 12 2019, 12:25 PM
sql/upgrades/123.sql
10–11

The code of these functions must be updated as well

swh/indexer/storage/in_memory.py
620–621

Sorry, one more nitpick: the for loop can be replaced by: yield from self._origin_intrinsic_metadata.get(ids)

haltode updated this revision to Diff 3894.Mar 12 2019, 3:41 PM
haltode marked 6 inline comments as done.

Update functions code during migration + More pythonic yield statement

Migration of swh_content_metadata_add and swh_origin_intrinsic_metadata_add are still missing. Oh, and I just noticed that the name change of index revision_intrinsic_metadata_pkey is also missing

haltode updated this revision to Diff 3897.Mar 13 2019, 3:38 AM
haltode marked 2 inline comments as done.

Add missing functions/index migrations.

haltode edited the summary of this revision. (Show Details)Mar 13 2019, 4:47 AM
vlorentz accepted this revision.Mar 13 2019, 9:48 AM
This revision is now accepted and ready to land.Mar 13 2019, 9:48 AM
haltode updated this revision to Diff 3907.Mar 13 2019, 1:21 PM

Rebasing on master

This revision was automatically updated to reflect the committed changes.