Page MenuHomeSoftware Heritage

Add context columns to object_metadata table and object_metadata_{add,get}.
ClosedPublic

Authored by vlorentz on Thu, Jun 25, 5:54 PM.

Details

Summary

Partially replaces D3247 (part 2/3)

Not used/tested yet; will be used when I introduce content_metadata_{get,add}.

Depends on D3355.

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Thu, Jun 25, 5:54 PM
vlorentz edited the summary of this revision. (Show Details)Thu, Jun 25, 5:55 PM

Build is green

Patch application report for D3356 (id=11905)

Could not rebase; Attempt merge onto b991e69707...

Merge made by the 'recursive' strategy.
 sql/upgrades/156.sql               |  42 ++++++++++
 swh/storage/cassandra/cql.py       |  87 +++++++++++--------
 swh/storage/cassandra/schema.py    |  46 ++++++++--
 swh/storage/cassandra/storage.py   | 116 ++++++++++++++++++-------
 swh/storage/db.py                  | 132 ++++++++++++++++++-----------
 swh/storage/extrinsic_metadata.py  |  57 +++++++++++++
 swh/storage/in_memory.py           | 101 ++++++++++++++++++----
 swh/storage/interface.py           |   2 +-
 swh/storage/sql/30-swh-schema.sql  |  48 +++++++----
 swh/storage/sql/60-swh-indexes.sql |  18 ++--
 swh/storage/storage.py             | 168 ++++++++++++++++++++++++++++---------
 11 files changed, 606 insertions(+), 211 deletions(-)
 create mode 100644 sql/upgrades/156.sql
 create mode 100644 swh/storage/extrinsic_metadata.py
Changes applied before test
commit a53431bcb197ef936c7139d2e713fbe2b5e25673
Merge: b991e697 9a3d5885
Author: Jenkins user <jenkins@localhost>
Date:   Thu Jun 25 15:55:10 2020 +0000

    Merge branch 'diff-target' into HEAD

commit 9a3d58855cd07a321699bbb72eeb440d219e0daf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:53:06 2020 +0200

    Add context columns to object_metadata table and object_metadata_{add,get}.
    
    Not used/tested yet; will be used when I introduce content_metadata_{get,add}.

commit feb1b53057b02cc794f04e95f12c5b57c9b1ae5b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:42:31 2020 +0200

    Generalize origin_metadata to allow support for other object types in the future.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/360/ for more details.

vlorentz updated this revision to Diff 11907.Thu, Jun 25, 6:05 PM

add missing migration code

Build is green

Patch application report for D3356 (id=11907)

Could not rebase; Attempt merge onto b991e69707...

Merge made by the 'recursive' strategy.
 sql/upgrades/156.sql               |  59 +++++++++++++
 swh/storage/cassandra/cql.py       |  87 +++++++++++--------
 swh/storage/cassandra/schema.py    |  46 ++++++++--
 swh/storage/cassandra/storage.py   | 116 ++++++++++++++++++-------
 swh/storage/db.py                  | 132 ++++++++++++++++++-----------
 swh/storage/extrinsic_metadata.py  |  57 +++++++++++++
 swh/storage/in_memory.py           | 101 ++++++++++++++++++----
 swh/storage/interface.py           |   2 +-
 swh/storage/sql/30-swh-schema.sql  |  48 +++++++----
 swh/storage/sql/60-swh-indexes.sql |  18 ++--
 swh/storage/storage.py             | 168 ++++++++++++++++++++++++++++---------
 11 files changed, 623 insertions(+), 211 deletions(-)
 create mode 100644 sql/upgrades/156.sql
 create mode 100644 swh/storage/extrinsic_metadata.py
Changes applied before test
commit bf96ef7f0883007ffc1fc043b6266f1476d467ec
Merge: b991e697 e6b8f236
Author: Jenkins user <jenkins@localhost>
Date:   Thu Jun 25 16:09:21 2020 +0000

    Merge branch 'diff-target' into HEAD

commit e6b8f236863c8f7b9010d47ab4875d949b78aa7d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:53:06 2020 +0200

    Add context columns to object_metadata table and object_metadata_{add,get}.
    
    Not used/tested yet; will be used when I introduce content_metadata_{get,add}.

commit feb1b53057b02cc794f04e95f12c5b57c9b1ae5b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:42:31 2020 +0200

    Generalize origin_metadata to allow support for other object types in the future.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/362/ for more details.

Build is green

Patch application report for D3356 (id=11949)

Could not rebase; Attempt merge onto 10443b8a17...

Auto-merging swh/storage/storage.py
Auto-merging swh/storage/interface.py
Auto-merging swh/storage/in_memory.py
Auto-merging swh/storage/db.py
Auto-merging swh/storage/cassandra/storage.py
Auto-merging swh/storage/cassandra/cql.py
Merge made by the 'recursive' strategy.
 sql/upgrades/156.sql               |  59 ++++++++++++++
 swh/storage/cassandra/cql.py       | 104 ++++++++++++++----------
 swh/storage/cassandra/schema.py    |  46 ++++++++---
 swh/storage/cassandra/storage.py   | 118 ++++++++++++++++++++-------
 swh/storage/db.py                  | 132 ++++++++++++++++++------------
 swh/storage/extrinsic_metadata.py  |  57 +++++++++++++
 swh/storage/in_memory.py           | 102 ++++++++++++++++++-----
 swh/storage/interface.py           |   2 +-
 swh/storage/sql/30-swh-schema.sql  |  48 +++++++----
 swh/storage/sql/60-swh-indexes.sql |  18 ++---
 swh/storage/storage.py             | 160 ++++++++++++++++++++++++++++---------
 11 files changed, 628 insertions(+), 218 deletions(-)
 create mode 100644 sql/upgrades/156.sql
 create mode 100644 swh/storage/extrinsic_metadata.py
Changes applied before test
commit 7a6b1998f8b4d1c9a6a650e3152d784fdcf8949b
Merge: 10443b8a a7156c75
Author: Jenkins user <jenkins@localhost>
Date:   Mon Jun 29 09:51:39 2020 +0000

    Merge branch 'diff-target' into HEAD

commit a7156c7592472221582c8ccc60630a56fd8767e0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:53:06 2020 +0200

    Add context columns to object_metadata table and object_metadata_{add,get}.
    
    Not used/tested yet; will be used when I introduce content_metadata_{get,add}.

commit 35f7ef913b42e5ac15eb19be2ff9ec99fa07f548
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:42:31 2020 +0200

    Generalize origin_metadata to allow support for other object types in the future.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/382/ for more details.

Build has FAILED

Patch application report for D3356 (id=11964)

Could not rebase; Attempt merge onto 10443b8a17...

Updating 10443b8a..b8c98650
Fast-forward
 sql/upgrades/156.sql               |  63 +++++++++++++++
 swh/storage/cassandra/cql.py       | 104 ++++++++++++++----------
 swh/storage/cassandra/schema.py    |  46 ++++++++---
 swh/storage/cassandra/storage.py   | 118 ++++++++++++++++++++-------
 swh/storage/db.py                  | 132 ++++++++++++++++++------------
 swh/storage/extrinsic_metadata.py  |  57 +++++++++++++
 swh/storage/in_memory.py           | 102 ++++++++++++++++++-----
 swh/storage/interface.py           |   2 +-
 swh/storage/sql/30-swh-schema.sql  |  49 ++++++++----
 swh/storage/sql/60-swh-indexes.sql |  18 ++---
 swh/storage/storage.py             | 160 ++++++++++++++++++++++++++++---------
 11 files changed, 633 insertions(+), 218 deletions(-)
 create mode 100644 sql/upgrades/156.sql
 create mode 100644 swh/storage/extrinsic_metadata.py
Changes applied before test
commit b8c986507d0d2de2290d69908140cc713c828211
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:53:06 2020 +0200

    Add context columns to object_metadata table and object_metadata_{add,get}.
    
    Not used/tested yet; will be used when I introduce content_metadata_{get,add}.

commit 595d5462e14f010b6acae3e87ff6aa7ce75ae8ce
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:42:31 2020 +0200

    Generalize origin_metadata to allow support for other object types in the future.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/388/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/388/console

This looks sound.

swh/storage/extrinsic_metadata.py
25

That check variable is not easy to read. It also relies on the current
setup (content uses context directory's keys which uses the context revision's
... up to the top).

I guess it's fine if the current setup does not change over time. Or that it
changes but we make sure it does not break the "cascading" setup style (through
a future test i guess)

ardumont accepted this revision.Tue, Jun 30, 10:18 AM
This revision is now accepted and ready to land.Tue, Jun 30, 10:18 AM
vlorentz updated this revision to Diff 11981.Tue, Jun 30, 12:33 PM

rebase + remove ALL_CONTEXT_KEYS

vlorentz marked an inline comment as done.Tue, Jun 30, 12:34 PM
vlorentz added inline comments.
swh/storage/extrinsic_metadata.py
25

I removed it, it wasn't necessary

Build is green

Patch application report for D3356 (id=11981)

Could not rebase; Attempt merge onto 1f0e25615e...

Updating 1f0e2561..869679a8
Fast-forward
 sql/upgrades/157.sql               |  63 +++++++++++++++
 swh/storage/cassandra/cql.py       | 104 ++++++++++++++----------
 swh/storage/cassandra/schema.py    |  46 ++++++++---
 swh/storage/cassandra/storage.py   | 118 ++++++++++++++++++++-------
 swh/storage/db.py                  | 132 +++++++++++++++++++-----------
 swh/storage/extrinsic_metadata.py  |  55 +++++++++++++
 swh/storage/in_memory.py           | 102 ++++++++++++++++++-----
 swh/storage/interface.py           |   2 +-
 swh/storage/sql/30-swh-schema.sql  |  49 +++++++----
 swh/storage/sql/60-swh-indexes.sql |  18 ++---
 swh/storage/storage.py             | 161 ++++++++++++++++++++++++++++---------
 11 files changed, 632 insertions(+), 218 deletions(-)
 create mode 100644 sql/upgrades/157.sql
 create mode 100644 swh/storage/extrinsic_metadata.py
Changes applied before test
commit 869679a85c55c07811c6cb156fac0eba226a6be8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:53:06 2020 +0200

    Add context columns to object_metadata table and object_metadata_{add,get}.
    
    Not used/tested yet; will be used when I introduce content_metadata_{get,add}.

commit 27e942621cef91cea6a98ffcc7beed5935714fe8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 25 17:42:31 2020 +0200

    Generalize origin_metadata to allow support for other object types in the future.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/393/ for more details.

This revision was automatically updated to reflect the committed changes.
vlorentz marked an inline comment as done.