Page MenuHomeSoftware Heritage

pg-storage: Adapt internal implementations to use origin visit status model representation
ClosedPublic

Authored by ardumont on Apr 1 2020, 12:10 PM.

Details

Summary

Related to T2310
Related to D2879

Depends on D2937

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Branch
make-origin-visit-immutable
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 11553
Build 17512: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 17511: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D2938 (id=10449)

Could not rebase; Attempt merge onto 377e6a8072...

Updating 377e6a8..0cdd6df
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/db.py                  | 173 ++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 185 ++++++++++++++++++++++++++-----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 137 ++++++++++++++++++++++-----
 6 files changed, 435 insertions(+), 145 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 0cdd6dfa9c5b038f1ad2033c4b0baa908d48adc4
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit b29327fd7cb7316ae6ba3ad27f53971d696d324f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/44/ for more details.

Rebase and Amend commit with pg-storage adaptations (merged in wrong commit as
well)

Build is green

Patch application report for D2938 (id=10455)

Could not rebase; Attempt merge onto 377e6a8072...

Updating 377e6a8..0151a20
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/db.py                  | 173 ++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 186 ++++++++++++++++++++++++++-----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 137 ++++++++++++++++++++++-----
 6 files changed, 436 insertions(+), 145 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 0151a209ccb80b6fc44b77109fe01ad736ebafd9
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit c0b05980ca3a19afa30fba6e75d067aaeac2cc96
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/47/ for more details.

Build is green

Patch application report for D2938 (id=10460)

Could not rebase; Attempt merge onto df3207a6b7...

Updating df3207a..acfd83b
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/db.py                  | 173 ++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 186 ++++++++++++++++++++++++++-----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 137 ++++++++++++++++++++++-----
 6 files changed, 436 insertions(+), 145 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit acfd83b8414c591ae25fc2d178db176074a7e8c7
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit c7ca0d04ff2d74150a1acde201b278d03d5a94d1
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/49/ for more details.

Rebase on latest master and blackify

Build is green

Patch application report for D2938 (id=10642)

Could not rebase; Attempt merge onto b0b0313c96...

Updating b0b0313..6020bf5
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 6 files changed, 455 insertions(+), 142 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 6020bf50b613c3dd9d2967ae5efdf985cd8097c7
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 65a4044601d1c8259585a127c28a7b5c23a3d1c6
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/82/ for more details.

  • rebase to latest master
  • update to use the latest OriginVisitStatus model object
ardumont retitled this revision from pg-storage: Adapt internal implementations to use origin visit update model representation to pg-storage: Adapt internal implementations to use origin visit status model representation.Apr 10 2020, 5:58 PM

Build is green

Patch application report for D2938 (id=10693)

Could not rebase; Attempt merge onto ddac3d27e3...

Updating ddac3d2..666018f
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  68 ++++++++++++++
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  25 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 8 files changed, 476 insertions(+), 146 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 666018f9d82714a18a084ab5aace0308140ee36e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 88fa1b0e22dd1996afeee99ffcbccd0c8c3be110
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/91/ for more details.

Build is green

Patch application report for D2938 (id=10820)

Could not rebase; Attempt merge onto bca643acab...

Updating bca643a..8758745
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  68 ++++++++++++++
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  25 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 8 files changed, 476 insertions(+), 146 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 87587457bd1d5e6ca2e5c6e416e6003f4671dacc
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 13fc814a0c0bef9487321138cf472b0fdbe48a28
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/109/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
sql/upgrades/147.sql
16–21

why about alter type origin_visit_status rename to origin_visit_state?

23–26

is this solved?

did you check it works in staging/docker?

45

not a very helpful comment...

46

isn't the metadata about the visit, instead of the origin?

47

possibly partial

swh/storage/sql/30-swh-schema.sql
286–287

??

This revision now requires changes to proceed.Apr 28 2020, 1:25 PM
sql/upgrades/147.sql
23–26

I checked my test with a transaction in staging indeed.
But no it's not really satisfying yet.

I'll check with your alter statement proposal which sounds nice.

46

yes.

swh/storage/sql/30-swh-schema.sql
286–287

i don't get the question ;)

the enum got renamed to origin_visit_state?

swh/storage/sql/30-swh-schema.sql
286–287

But why do we still have this field on origin_visit?

swh/storage/sql/30-swh-schema.sql
286–287

ah ok, because that's our initial work plan [1]

[1] we initially said we'd remove it when we are done migrating
T2310 (i can't link to the "work plan" paragraph :/)

swh/storage/sql/30-swh-schema.sql
286–287

hmm, yeah, ok

sql/upgrades/147.sql
16–21

Perfect, that works (tested on staging) and this simplify the migration below.

  • Rebase on latest master
  • Adapt according to review (alter rename old enum, improve column comments...)
This revision is now accepted and ready to land.Apr 28 2020, 2:45 PM

Build is green

Patch application report for D2938 (id=10933)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..af54589
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 182 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 8 files changed, 474 insertions(+), 148 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit af5458904fbab3383a2ce175c78db05c14ed7824
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 9cc80f8f92f37ecdd948179b6bf954b3109382d5
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/138/ for more details.

Build is green

Patch application report for D2938 (id=10934)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..d0b1b1c
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 182 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 8 files changed, 474 insertions(+), 148 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit d0b1b1c1e5de8eb8e0811d77f78ca23198cb5ed3
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 9cc80f8f92f37ecdd948179b6bf954b3109382d5
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/139/ for more details.

Rename visit "update" into visit "status" where it makes sense

Build is green

Patch application report for D2938 (id=10937)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..3029a2c
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 8 files changed, 468 insertions(+), 148 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 3029a2c7300ac518d024de24aaeecb1ef964399e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit d3e97b9d9cfa636b7e4a079ee3dcf0851ee34dba
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/142/ for more details.

Fix missing comment adaptation

Build is green

Patch application report for D2938 (id=10939)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..a8383ad
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 8 files changed, 468 insertions(+), 148 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit a8383ad26d53ea50ab26c1c33bc80967ae8635e3
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit d3e97b9d9cfa636b7e4a079ee3dcf0851ee34dba
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/144/ for more details.

Fix migration script (it fully works now)

Build is green

Patch application report for D2938 (id=10947)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..a720cae
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 8 files changed, 468 insertions(+), 148 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit a720caed6eebbb68a9f9b5be554a52859aa052d6
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit ead80889fb44d7c0e929b913f3e852ab851c0a3d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/147/ for more details.