Page MenuHomeSoftware Heritage
Differential D2939

cassandra storage: Adapt internal implementations to use origin visit status model representation
ClosedPublic
Actions

Authored by ardumont on Apr 1 2020, 12:11 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2310: Make origin visits immutable
Commits
rDSTOe3e76c42b43a: cassandra: Adapt internal implementations to use origin visit update
Summary

Related to T2310
Related to D2879
Depends on D2938

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Branch
make-origin-visit-immutable
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 12089
Build 18342: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 18341: arc lint + arc unit

Event Timeline

ardumont created this revision.Apr 1 2020, 12:11 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 1 2020, 12:11 PM
ardumont mentioned this in D2879: storage: Refactor internal implementations to use "origin visit update" model representation.Apr 1 2020, 12:12 PM
swh-public-ci added a comment.Apr 1 2020, 12:26 PM

Build is green

Patch application report for D2939 (id=10450)

Could not rebase; Attempt merge onto 377e6a8072...

Updating 377e6a8..400dd86
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/cassandra/cql.py       |  91 +++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 163 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 173 ++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 185 ++++++++++++++++++++++++++-----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 137 ++++++++++++++++++++++-----
 9 files changed, 627 insertions(+), 219 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 400dd86e25e4b64558e957f004a60a699caff87d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 0cdd6dfa9c5b038f1ad2033c4b0baa908d48adc4
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit b29327fd7cb7316ae6ba3ad27f53971d696d324f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/45/ for more details.

Harbormaster completed remote builds in B11535: Diff 10450.Apr 1 2020, 12:26 PM
ardumont updated this revision to Diff 10461.Apr 1 2020, 3:18 PM

Rebase on latest master

swh-public-ci added a comment.Apr 1 2020, 3:33 PM

Build is green

Patch application report for D2939 (id=10461)

Could not rebase; Attempt merge onto df3207a6b7...

Updating df3207a..0861c6f
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/cassandra/cql.py       |  91 +++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 163 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 173 ++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 186 ++++++++++++++++++++++++++-----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 137 ++++++++++++++++++++++-----
 9 files changed, 628 insertions(+), 219 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 0861c6f773211fcabf6ce438f8ff16ccf7203ca0
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit acfd83b8414c591ae25fc2d178db176074a7e8c7
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit c7ca0d04ff2d74150a1acde201b278d03d5a94d1
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/50/ for more details.

Harbormaster completed remote builds in B11554: Diff 10461.Apr 1 2020, 3:33 PM
ardumont mentioned this in T2310: Make origin visits immutable.Apr 1 2020, 4:40 PM
ardumont updated this revision to Diff 10643.Apr 9 2020, 4:16 PM

Rebase on latest master and blackify

swh-public-ci added a comment.Apr 9 2020, 4:31 PM

Build is green

Patch application report for D2939 (id=10643)

Could not rebase; Attempt merge onto b0b0313c96...

Updating b0b0313..042cc07
Fast-forward
 sql/upgrades/147.sql               |  51 ++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 161 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/30-swh-schema.sql  |  23 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 9 files changed, 655 insertions(+), 216 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 042cc078dac0d295541865cf4212aad4a9c5a075
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 6020bf50b613c3dd9d2967ae5efdf985cd8097c7
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 65a4044601d1c8259585a127c28a7b5c23a3d1c6
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/83/ for more details.

Harbormaster completed remote builds in B11763: Diff 10643.Apr 9 2020, 4:31 PM
ardumont mentioned this in D3001: model: Rename OriginVisitUpdate to OriginVisitStatus.Apr 10 2020, 11:20 AM
ardumont updated this revision to Diff 10694.Apr 10 2020, 5:49 PM
  • rebase to latest master
  • update to use the latest OriginVisitStatus model object
ardumont retitled this revision from cassandra storage: Adapt internal implementations to use origin visit update model representation to cassandra storage: Adapt internal implementations to use origin visit status model representation.Apr 10 2020, 5:58 PM
swh-public-ci added a comment.Apr 10 2020, 6:05 PM

Build is green

Patch application report for D2939 (id=10694)

Could not rebase; Attempt merge onto ddac3d27e3...

Updating ddac3d2..659e4b3
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  68 ++++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 161 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  25 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 11 files changed, 676 insertions(+), 220 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 659e4b3bf5efba7412ad79c1867e6d57b47cc28f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 666018f9d82714a18a084ab5aace0308140ee36e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 88fa1b0e22dd1996afeee99ffcbccd0c8c3be110
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/92/ for more details.

Harbormaster completed remote builds in B11815: Diff 10694.Apr 10 2020, 6:05 PM
ardumont added a task: T2310: Make origin visits immutable.Apr 14 2020, 10:47 AM
ardumont updated this revision to Diff 10821.Apr 22 2020, 2:38 PM

Rebase on latest master

swh-public-ci added a comment.Apr 22 2020, 2:54 PM

Build is green

Patch application report for D2939 (id=10821)

Could not rebase; Attempt merge onto bca643acab...

Updating bca643a..8a984f1
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  68 ++++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 161 ++++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 184 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  25 ++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 11 files changed, 676 insertions(+), 220 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 8a984f187dd839fe2427b7d417355bf0b98d06b0
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 87587457bd1d5e6ca2e5c6e416e6003f4671dacc
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 13fc814a0c0bef9487321138cf472b0fdbe48a28
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/110/ for more details.

Harbormaster completed remote builds in B11975: Diff 10821.Apr 22 2020, 2:54 PM
vlorentz requested changes to this revision.Apr 28 2020, 2:24 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/storage/cassandra/storage.py
926–927

I think this should be an assert, and the function not return Optional

This revision now requires changes to proceed.Apr 28 2020, 2:24 PM
ardumont updated this revision to Diff 10935.Apr 28 2020, 2:52 PM
  • Rebase to latest master
  • Adapt according to review
ardumont updated this revision to Diff 10938.Apr 28 2020, 3:00 PM

Rebase

swh-public-ci added a comment.Apr 28 2020, 3:04 PM

Build is green

Patch application report for D2939 (id=10935)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..5b97b5a
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 158 +++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 182 ++++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 143 +++++++++++++++++++++++-----
 11 files changed, 671 insertions(+), 222 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit 5b97b5a6d97da79fc341c9d65be78e5d35e0a02e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit d0b1b1c1e5de8eb8e0811d77f78ca23198cb5ed3
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 9cc80f8f92f37ecdd948179b6bf954b3109382d5
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/140/ for more details.

Harbormaster completed remote builds in B12089: Diff 10935.Apr 28 2020, 3:04 PM
ardumont updated this revision to Diff 10940.Apr 28 2020, 3:18 PM

Rebase

/me sighs

swh-public-ci added a comment.Apr 28 2020, 3:22 PM

Build is green

Patch application report for D2939 (id=10938)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..b405a60
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 157 +++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 11 files changed, 664 insertions(+), 222 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit b405a60d5d7483a775bdfa58f40af98649b6d0bd
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 3029a2c7300ac518d024de24aaeecb1ef964399e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit d3e97b9d9cfa636b7e4a079ee3dcf0851ee34dba
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/143/ for more details.

Harbormaster completed remote builds in B12092: Diff 10938.Apr 28 2020, 3:22 PM
swh-public-ci added a comment.Apr 28 2020, 3:34 PM

Build is green

Patch application report for D2939 (id=10940)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..b1b443e
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 157 +++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 11 files changed, 664 insertions(+), 222 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit b1b443ea465337830887aa22305a420f5c161d6f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit a8383ad26d53ea50ab26c1c33bc80967ae8635e3
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit d3e97b9d9cfa636b7e4a079ee3dcf0851ee34dba
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/145/ for more details.

Harbormaster completed remote builds in B12094: Diff 10940.Apr 28 2020, 3:34 PM
vlorentz accepted this revision.Apr 28 2020, 4:12 PM
This revision is now accepted and ready to land.Apr 28 2020, 4:12 PM
ardumont updated this revision to Diff 10948.Apr 28 2020, 4:48 PM

Rebase

swh-public-ci added a comment.Apr 28 2020, 5:06 PM

Build is green

Patch application report for D2939 (id=10948)

Could not rebase; Attempt merge onto 49109d173f...

Updating 49109d1..e3e76c4
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/147.sql               |  64 +++++++++++++
 swh/storage/cassandra/cql.py       | 101 +++++++++++---------
 swh/storage/cassandra/schema.py    |  12 ++-
 swh/storage/cassandra/storage.py   | 157 +++++++++++++++++++++++++------
 swh/storage/db.py                  | 185 +++++++++++++++++++++++--------------
 swh/storage/in_memory.py           | 179 +++++++++++++++++++++++++----------
 swh/storage/sql/20-swh-enums.sql   |   4 +-
 swh/storage/sql/30-swh-schema.sql  |  31 ++++++-
 swh/storage/sql/60-swh-indexes.sql |  11 +++
 swh/storage/storage.py             | 140 +++++++++++++++++++++++-----
 11 files changed, 664 insertions(+), 222 deletions(-)
 create mode 100644 sql/upgrades/147.sql
Changes applied before test
commit e3e76c42b43a5634f612b0f6acf19468abd1eb3a
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit a720caed6eebbb68a9f9b5be554a52859aa052d6
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit ead80889fb44d7c0e929b913f3e852ab851c0a3d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/148/ for more details.

Harbormaster completed remote builds in B12104: Diff 10948.Apr 28 2020, 5:06 PM
Closed by commit rDSTOe3e76c42b43a: cassandra: Adapt internal implementations to use origin visit update (authored by ardumont). · Explain WhyApr 28 2020, 5:10 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSTOe3e76c42b43a: cassandra: Adapt internal implementations to use origin visit update.
ardumont mentioned this in D3101: pg-storage: Adapt implem to write both origin visit updates and status.Apr 30 2020, 12:03 PM
ardumont mentioned this in rDSTOb0b767b91ca0: pg: Write both origin visit updates & status, read from origin_visit.Apr 30 2020, 2:42 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
storage/
cassandra/
101 lines98%
Loading...
12 lines100%
Loading...
158 lines93%
Loading...

Diff 10935

View Options

swh/storage/cassandra/cql.py

Loading...
View Options

swh/storage/cassandra/schema.py

Loading...
View Options

swh/storage/cassandra/storage.py

Loading...
About · Developer information · Legal