Page MenuHomeSoftware Heritage
Differential D2939

cassandra storage: Adapt internal implementations to use origin visit status model representation
ClosedPublic
Actions

Authored by ardumont on Apr 1 2020, 12:11 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2310: Make origin visits immutable
Commits
rDSTOe3e76c42b43a: cassandra: Adapt internal implementations to use origin visit update
Summary

Related to T2310
Related to D2879
Depends on D2938

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Apr 1 2020, 12:11 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 1 2020, 12:11 PM
ardumont mentioned this in D2879: storage: Refactor internal implementations to use "origin visit update" model representation.Apr 1 2020, 12:12 PM
swh-public-ci added a comment.Apr 1 2020, 12:26 PM

Build is green

Patch application report for D2939 (id=10450)

Could not rebase; Attempt merge onto 377e6a8072...

Updating 377e6a8..400dd86
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/cassandra/cql.py       |  91 +++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 163 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 173 ++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 185 ++++++++++++++++++++++++++-----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 137 ++++++++++++++++++++++-----
 9 files changed, 627 insertions(+), 219 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 400dd86e25e4b64558e957f004a60a699caff87d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 0cdd6dfa9c5b038f1ad2033c4b0baa908d48adc4
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit b29327fd7cb7316ae6ba3ad27f53971d696d324f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/45/ for more details.

Harbormaster completed remote builds in B11535: Diff 10450.Apr 1 2020, 12:26 PM
ardumont updated this revision to Diff 10461.Apr 1 2020, 3:18 PM

Rebase on latest master

swh-public-ci added a comment.Apr 1 2020, 3:33 PM

Build is green

Patch application report for D2939 (id=10461)

Could not rebase; Attempt merge onto df3207a6b7...

Updating df3207a..0861c6f
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/cassandra/cql.py       |  91 +++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 163 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 173 ++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 186 ++++++++++++++++++++++++++-----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 137 ++++++++++++++++++++++-----
 9 files changed, 628 insertions(+), 219 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 0861c6f773211fcabf6ce438f8ff16ccf7203ca0
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit acfd83b8414c591ae25fc2d178db176074a7e8c7
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit c7ca0d04ff2d74150a1acde201b278d03d5a94d1
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/50/ for more details.

Harbormaster completed remote builds in B11554: Diff 10461.Apr 1 2020, 3:33 PM
ardumont mentioned this in T2310: Make origin visits immutable.Apr 1 2020, 4:40 PM
ardumont updated this revision to Diff 10643.Apr 9 2020, 4:16 PM

Rebase on latest master and blackify

swh-public-ci added a comment.Apr 9 2020, 4:31 PM

Build is green

Patch application report for D2939 (id=10643)

Could not rebase; Attempt merge onto b0b0313c96...

Updating b0b0313..042cc07
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 161 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 9 files changed, 655 insertions(+), 216 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 042cc078dac0d295541865cf4212aad4a9c5a075
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 6020bf50b613c3dd9d2967ae5efdf985cd8097c7
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 65a4044601d1c8259585a127c28a7b5c23a3d1c6
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/83/ for more details.

Harbormaster completed remote builds in B11763: Diff 10643.Apr 9 2020, 4:31 PM
ardumont mentioned this in D3001: model: Rename OriginVisitUpdate to OriginVisitStatus.Apr 10 2020, 11:20 AM
ardumont updated this revision to Diff 10694.Apr 10 2020, 5:49 PM
  • rebase to latest master
  • update to use the latest OriginVisitStatus model object
ardumont retitled this revision from cassandra storage: Adapt internal implementations to use origin visit update model representation to cassandra storage: Adapt internal implementations to use origin visit status model representation.Apr 10 2020, 5:58 PM
swh-public-ci added a comment.Apr 10 2020, 6:05 PM

Build is green

Patch application report for D2939 (id=10694)

Could not rebase; Attempt merge onto ddac3d27e3...

Updating ddac3d2..659e4b3
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  68 ++++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 161 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  25 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 11 files changed, 676 insertions(+), 220 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 659e4b3bf5efba7412ad79c1867e6d57b47cc28f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 666018f9d82714a18a084ab5aace0308140ee36e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 88fa1b0e22dd1996afeee99ffcbccd0c8c3be110
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/92/ for more details.

Harbormaster completed remote builds in B11815: Diff 10694.Apr 10 2020, 6:05 PM
ardumont added a task: T2310: Make origin visits immutable.Apr 14 2020, 10:47 AM
ardumont updated this revision to Diff 10821.Apr 22 2020, 2:38 PM

Rebase on latest master

swh-public-ci added a comment.Apr 22 2020, 2:54 PM

Build is green

Patch application report for D2939 (id=10821)

Could not rebase; Attempt merge onto bca643acab...

Updating bca643a..8a984f1
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  68 ++++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 161 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  25 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 11 files changed, 676 insertions(+), 220 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 8a984f187dd839fe2427b7d417355bf0b98d06b0
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 87587457bd1d5e6ca2e5c6e416e6003f4671dacc
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 13fc814a0c0bef9487321138cf472b0fdbe48a28
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/110/ for more details.

Harbormaster completed remote builds in B11975: Diff 10821.Apr 22 2020, 2:54 PM
vlorentz requested changes to this revision.Apr 28 2020, 2:24 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/storage/cassandra/storage.py
925–926

I think this should be an assert, and the function not return Optional

This revision now requires changes to proceed.Apr 28 2020, 2:24 PM
ardumont updated this revision to Diff 10935.Apr 28 2020, 2:52 PM
  • Rebase to latest master
  • Adapt according to review
ardumont updated this revision to Diff 10938.Apr 28 2020, 3:00 PM

Rebase

swh-public-ci added a comment.Apr 28 2020, 3:04 PM

Build is green

Patch application report for D2939 (id=10935)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..5b97b5a
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 158 +++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 182 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 11 files changed, 671 insertions(+), 222 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 5b97b5a6d97da79fc341c9d65be78e5d35e0a02e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit d0b1b1c1e5de8eb8e0811d77f78ca23198cb5ed3
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 9cc80f8f92f37ecdd948179b6bf954b3109382d5
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/140/ for more details.

Harbormaster completed remote builds in B12089: Diff 10935.Apr 28 2020, 3:04 PM
ardumont updated this revision to Diff 10940.Apr 28 2020, 3:18 PM

Rebase

/me sighs

swh-public-ci added a comment.Apr 28 2020, 3:22 PM

Build is green

Patch application report for D2939 (id=10938)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..b405a60
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 157 +++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 11 files changed, 664 insertions(+), 222 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit b405a60d5d7483a775bdfa58f40af98649b6d0bd
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 3029a2c7300ac518d024de24aaeecb1ef964399e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit d3e97b9d9cfa636b7e4a079ee3dcf0851ee34dba
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/143/ for more details.

Harbormaster completed remote builds in B12092: Diff 10938.Apr 28 2020, 3:22 PM
swh-public-ci added a comment.Apr 28 2020, 3:34 PM

Build is green

Patch application report for D2939 (id=10940)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..b1b443e
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 157 +++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 11 files changed, 664 insertions(+), 222 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit b1b443ea465337830887aa22305a420f5c161d6f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit a8383ad26d53ea50ab26c1c33bc80967ae8635e3
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit d3e97b9d9cfa636b7e4a079ee3dcf0851ee34dba
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/145/ for more details.

Harbormaster completed remote builds in B12094: Diff 10940.Apr 28 2020, 3:34 PM
vlorentz accepted this revision.Apr 28 2020, 4:12 PM
This revision is now accepted and ready to land.Apr 28 2020, 4:12 PM
ardumont updated this revision to Diff 10948.Apr 28 2020, 4:48 PM

Rebase

swh-public-ci added a comment.Apr 28 2020, 5:06 PM

Build is green

Patch application report for D2939 (id=10948)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..e3e76c4
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 157 +++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 11 files changed, 664 insertions(+), 222 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit e3e76c42b43a5634f612b0f6acf19468abd1eb3a
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit a720caed6eebbb68a9f9b5be554a52859aa052d6
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit ead80889fb44d7c0e929b913f3e852ab851c0a3d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/148/ for more details.

Harbormaster completed remote builds in B12104: Diff 10948.Apr 28 2020, 5:06 PM
Closed by commit rDSTOe3e76c42b43a: cassandra: Adapt internal implementations to use origin visit update (authored by ardumont). · Explain WhyApr 28 2020, 5:10 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSTOe3e76c42b43a: cassandra: Adapt internal implementations to use origin visit update.
ardumont mentioned this in D3101: pg-storage: Adapt implem to write both origin visit updates and status.Apr 30 2020, 12:03 PM
ardumont mentioned this in rDSTOb0b767b91ca0: pg: Write both origin visit updates & status, read from origin_visit.Apr 30 2020, 2:42 PM

Revision Contents
Changeset List

PathSize
swh/
storage/
cassandra/
101 lines
12 lines
157 lines

Diff 10953

View Options

swh/storage/cassandra/cql.py

Loading...
View Options

swh/storage/cassandra/schema.py

Loading...
View Options

swh/storage/cassandra/storage.py

Loading...
About · Developer information · Legal