Page MenuHomeSoftware Heritage

storage: Refactor internal implementations to use "origin visit update" model representation
AbandonedPublic

Authored by ardumont on Mar 25 2020, 5:58 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

(origin visit update objects as first-citizen objects in swh model)

(pairing with @vlorentz)

currently:

  • in-memory implementation
  • pg storage implementation
  • cassandra implementation
  • sql migration script
  • Land D2880 (to make those tests go green here)
  • rework the perimeter to only include the origin-visit-update changes
  • split diff into multiple diffs (1 per backend, D2937, D2938, D2939)

Related to T2310
Related to D2880

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Branch
make-origin-visit-immutable
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 11343
Build 17165: tox-on-jenkinsJenkins
Build 17164: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ardumont updated this revision to Diff 10397.Mar 30 2020, 5:54 PM
  • Ensure visit id is set in origin_visit_upsert before kafka write
  • wip: cassandra: Implement origin_visit_update internally

cassandra implementation is not complete yet

Build has FAILED

Patch application report for D2879 (id=10397)

Rebasing onto 623a1b75cb...

First, rewinding head to replay your work on top of it...
Applying: storage: Define a now() function
Applying: storage*: Align origin_visit_update interface and implementations
Applying: storage*: Add types to origin_visit_get
Applying: in_memory: Adapt internal implementations to use origin visit update
Applying: pg-storage: Start using origin_visit_update
Applying: Add missing type annotations on origin_visit_get* endpoints
Applying: pg-storage: Implement origin visit update
Using index info to reconstruct a base tree...
M	swh/storage/tests/test_storage.py
M	swh/storage/validate.py
Falling back to patching base and 3-way merge...
Auto-merging swh/storage/validate.py
CONFLICT (content): Merge conflict in swh/storage/validate.py
Auto-merging swh/storage/tests/test_storage.py
Patch failed at 0007 pg-storage: Implement origin visit update

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Rebase failed (ret=1)!

Could not rebase; Attempt merge onto 623a1b75cb...

Already up to date.
Changes applied before test
commit 62e7718868af8715d110703b9182130ba04bdc2d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 17:01:13 2020 +0200

    Ensure visit id is set in origin_visit_upsert before kafka write

commit 0a0953de12d1b63681a29f1622943434f7f04c02
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    wip: cassandra: Implement origin_visit_update internally
    
    methods:
    - origin_visit_add
    - origin_visit_update
    - origin_visit_upsert
    - origin_visit_get
    - origin_visit_find_by_date
    - origin_visit_get_by
    - origin_visit_get_latest

commit dd54c6a2ad96fec391252bd393e6d30ec6cc0030
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Implement origin visit update

commit f6ad731889ed78a656b294a85723a4c55e7bdcac
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit ffff91e1b08856ab0af1d4b0c307795bf60c4c72
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit f07b0edc98f992a1707b9cf2ae9842ddd0b2c5bc
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

commit 0961d931edc1c68171e48e9a96e56026ec48b8ea
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 14:40:36 2020 +0100

    storage*: Add types to origin_visit_get

commit 5859dc794a1d8fa1a43049506ef9711d225a0b09
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 08:57:18 2020 +0100

    storage*: Align origin_visit_update interface and implementations

commit dd1dd1c62b2bed4492130894265f8b2ed22d9daa
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 15:08:35 2020 +0100

    storage: Define a now() function

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/29/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/29/console

ardumont edited the summary of this revision. (Show Details)Mar 30 2020, 6:54 PM
ardumont planned changes to this revision.Mar 30 2020, 6:57 PM
ardumont retitled this revision from wip: storage: Refactor internal implementations to use origin visit update to wip: storage: Refactor internal implementations to use origin visit update representation.
ardumont edited the summary of this revision. (Show Details)
ardumont updated this revision to Diff 10401.Mar 30 2020, 8:39 PM

Finish cassandra implementation

Build has FAILED

Patch application report for D2879 (id=10401)

Rebasing onto 623a1b75cb...

First, rewinding head to replay your work on top of it...
Applying: storage: Define a now() function
Applying: storage*: Align origin_visit_update interface and implementations
Applying: storage*: Add types to origin_visit_get
Applying: cassandra/cql: Simplify type using Iterator
Applying: in_memory: Adapt internal implementations to use origin visit update
Applying: pg-storage: Start using origin_visit_update
Applying: Add missing type annotations on origin_visit_get* endpoints
Applying: pg-storage: Adapt internal implementations to use origin visit update
Using index info to reconstruct a base tree...
M	swh/storage/tests/test_storage.py
M	swh/storage/validate.py
Falling back to patching base and 3-way merge...
Auto-merging swh/storage/validate.py
CONFLICT (content): Merge conflict in swh/storage/validate.py
Auto-merging swh/storage/tests/test_storage.py
Patch failed at 0008 pg-storage: Adapt internal implementations to use origin visit update

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Rebase failed (ret=1)!

Could not rebase; Attempt merge onto 623a1b75cb...

Already up to date.
Changes applied before test
commit 9fa8a16610ecd60d8fd76406f1248378af9e0a3d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 17:01:13 2020 +0200

    Ensure visit id is set in origin_visit_upsert before kafka write

commit b41430fac90e15c6482719a4f025748c6a9059f2
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit 28ba789262abec36e55cd2f67e6a25d6fe9a1902
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 5b765bd879c3d783f44a5e7121586799da401886
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit 8a66356234f3925e7b2ef5658e3f12dbebc2d6a8
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit 46d7ebfabda7b6169eb8f1219e8a02d215f6355d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

commit f3824d3ccc1a178f7d09c2036ce91a4343cecc6c
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 20:30:44 2020 +0200

    cassandra/cql: Simplify type using Iterator

commit 0961d931edc1c68171e48e9a96e56026ec48b8ea
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 14:40:36 2020 +0100

    storage*: Add types to origin_visit_get

commit 5859dc794a1d8fa1a43049506ef9711d225a0b09
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 08:57:18 2020 +0100

    storage*: Align origin_visit_update interface and implementations

commit dd1dd1c62b2bed4492130894265f8b2ed22d9daa
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 15:08:35 2020 +0100

    storage: Define a now() function

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/30/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/30/console

ardumont edited the summary of this revision. (Show Details)Mar 30 2020, 8:40 PM
ardumont edited the test plan for this revision. (Show Details)
ardumont updated this revision to Diff 10402.Mar 31 2020, 10:09 AM

Add migration script (schema, data)

Build has FAILED

Patch application report for D2879 (id=10402)

Rebasing onto 623a1b75cb...

First, rewinding head to replay your work on top of it...
Applying: storage: Define a now() function
Applying: storage*: Align origin_visit_update interface and implementations
Applying: storage*: Add types to origin_visit_get
Applying: cassandra/cql: Simplify type using Iterator
Applying: in_memory: Adapt internal implementations to use origin visit update
Applying: pg-storage: Start using origin_visit_update
Applying: Add missing type annotations on origin_visit_get* endpoints
Applying: pg-storage: Adapt internal implementations to use origin visit update
Using index info to reconstruct a base tree...
M	swh/storage/tests/test_storage.py
M	swh/storage/validate.py
Falling back to patching base and 3-way merge...
Auto-merging swh/storage/validate.py
CONFLICT (content): Merge conflict in swh/storage/validate.py
Auto-merging swh/storage/tests/test_storage.py
Patch failed at 0008 pg-storage: Adapt internal implementations to use origin visit update

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Rebase failed (ret=1)!

Could not rebase; Attempt merge onto 623a1b75cb...

Already up to date.
Changes applied before test
commit 72d2928e96ebcfd4e73640a70ef063fb5650dc9e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 17:01:13 2020 +0200

    Ensure visit id is set in origin_visit_upsert before kafka write

commit 1b7e7c795e837f70ca0c45ea58d61a1981f5387e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit 6959e96817a49eb2955d7be148edaeb878993d6b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 5b765bd879c3d783f44a5e7121586799da401886
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit 8a66356234f3925e7b2ef5658e3f12dbebc2d6a8
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit 46d7ebfabda7b6169eb8f1219e8a02d215f6355d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

commit f3824d3ccc1a178f7d09c2036ce91a4343cecc6c
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 20:30:44 2020 +0200

    cassandra/cql: Simplify type using Iterator

commit 0961d931edc1c68171e48e9a96e56026ec48b8ea
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 14:40:36 2020 +0100

    storage*: Add types to origin_visit_get

commit 5859dc794a1d8fa1a43049506ef9711d225a0b09
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 08:57:18 2020 +0100

    storage*: Align origin_visit_update interface and implementations

commit dd1dd1c62b2bed4492130894265f8b2ed22d9daa
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 15:08:35 2020 +0100

    storage: Define a now() function

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/31/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/31/console

ardumont planned changes to this revision.Mar 31 2020, 10:50 AM
ardumont edited the summary of this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)
ardumont updated this revision to Diff 10415.Mar 31 2020, 1:07 PM
  • test_retry: Use datetime instead of string

Build has FAILED

Patch application report for D2879 (id=10415)

Rebasing onto 623a1b75cb...

First, rewinding head to replay your work on top of it...
Applying: storage: Define a now() function
Applying: storage*: Align origin_visit_update interface and implementations
Applying: storage*: Add types to origin_visit_get
Applying: cassandra/cql: Simplify type using Iterator
Applying: in_memory: Adapt internal implementations to use origin visit update
Applying: pg-storage: Start using origin_visit_update
Applying: Add missing type annotations on origin_visit_get* endpoints
Applying: pg-storage: Adapt internal implementations to use origin visit update
Using index info to reconstruct a base tree...
M	swh/storage/tests/test_storage.py
M	swh/storage/validate.py
Falling back to patching base and 3-way merge...
Auto-merging swh/storage/validate.py
CONFLICT (content): Merge conflict in swh/storage/validate.py
Auto-merging swh/storage/tests/test_storage.py
Patch failed at 0008 pg-storage: Adapt internal implementations to use origin visit update

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Rebase failed (ret=1)!

Could not rebase; Attempt merge onto 623a1b75cb...

Already up to date.
Changes applied before test
commit 05ed7d143ad0794831ee2031d4a7cc7a1e89a89e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Tue Mar 31 12:51:26 2020 +0200

    test_retry: Use datetime instead of string

commit 72d2928e96ebcfd4e73640a70ef063fb5650dc9e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 17:01:13 2020 +0200

    Ensure visit id is set in origin_visit_upsert before kafka write

commit 1b7e7c795e837f70ca0c45ea58d61a1981f5387e
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit 6959e96817a49eb2955d7be148edaeb878993d6b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 5b765bd879c3d783f44a5e7121586799da401886
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit 8a66356234f3925e7b2ef5658e3f12dbebc2d6a8
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit 46d7ebfabda7b6169eb8f1219e8a02d215f6355d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

commit f3824d3ccc1a178f7d09c2036ce91a4343cecc6c
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 20:30:44 2020 +0200

    cassandra/cql: Simplify type using Iterator

commit 0961d931edc1c68171e48e9a96e56026ec48b8ea
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 14:40:36 2020 +0100

    storage*: Add types to origin_visit_get

commit 5859dc794a1d8fa1a43049506ef9711d225a0b09
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 08:57:18 2020 +0100

    storage*: Align origin_visit_update interface and implementations

commit dd1dd1c62b2bed4492130894265f8b2ed22d9daa
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 15:08:35 2020 +0100

    storage: Define a now() function

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/32/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/32/console

ardumont planned changes to this revision.Mar 31 2020, 3:08 PM
ardumont updated this revision to Diff 10416.Mar 31 2020, 3:11 PM

Rebase on latest master

Build has FAILED

Patch application report for D2879 (id=10416)

Rebasing onto eb82792892...

Current branch diff-target is up to date.
Changes applied before test
commit f2ed18484eae3626b548458f72f4040f62ae8dbd
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 17:01:13 2020 +0200

    Ensure visit id is set in origin_visit_upsert before kafka write

commit 4b84881e7168277a897b0b02cc87d2cb3b203283
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit d587db9d766d9b61d257ea91df254e75e7a5d5dc
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 8454f679a36dbb1d6dd106ecb4e547ded732f74c
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit f6620104020f40db13996ec495305375653c8941
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit 3f7841fee6d89e02c762cea3f8631fa8edea4f27
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/33/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/33/console

ardumont updated this revision to Diff 10425.Mar 31 2020, 4:18 PM

Rebase on latest master

Build has FAILED

Patch application report for D2879 (id=10425)

Rebasing onto acf057e98b...

Current branch diff-target is up to date.
Changes applied before test
commit f1c6105735fd161bec8d37e59d9132e8cdbde8f9
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 17:01:13 2020 +0200

    Ensure visit id is set in origin_visit_upsert before kafka write

commit 449657549e02f58f77303d7f5220a6ac6ce2879d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit 845af24072a4aa68e5613be7b737d6632177920a
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit df518ebd2e6318ed75e7300a4fade9df2129b5c2
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit b99977ad2d41ca2ae5932a54843c1c656bb9935f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit 46e52aa36d9e54b4104144872a534f28f77a491b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/34/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/34/console

ardumont updated this revision to Diff 10427.Mar 31 2020, 4:37 PM

Rebase on latest origin/master

Build has FAILED

Patch application report for D2879 (id=10427)

Rebasing onto 2856004350...

Current branch diff-target is up to date.
Changes applied before test
commit 15ff5ea00e159ed83b2ee908c0ac7e78044c7cf9
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit 2ea654151bae2b6f737926941fb1ae1c1aadf6a4
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 0ce9d8059cba22228635d4ee274529486f577cf3
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit fb61a5956ec06180e3b148eafb24f45627e858a8
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit b124ae6d52577068e2cd312e031ffe9a1d7c1aa9
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/35/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/35/console

ardumont updated this revision to Diff 10428.Mar 31 2020, 4:45 PM

sql: Drop update_id as it's not used
sql: migrate data: on conflict do nothing

Build has FAILED

Patch application report for D2879 (id=10428)

Rebasing onto 2856004350...

Current branch diff-target is up to date.
Changes applied before test
commit 9c396754c1a9c1503c360c39a1197a7c4d5f70ac
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit d756b832cf0316a3a00809488d063a8ff5edfe77
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 0ce9d8059cba22228635d4ee274529486f577cf3
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit fb61a5956ec06180e3b148eafb24f45627e858a8
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit b124ae6d52577068e2cd312e031ffe9a1d7c1aa9
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/36/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/36/console

Could you check code coverage status, to find if some other functions become dead code?

swh/storage/cassandra/cql.py
610–628 ↗(On Diff #10428)

You should be able to remove this method now

Could you check code coverage status, to find if some other functions become dead code?

I'm sad, i'm wondering why the new coverage report does not show up here :/

swh/storage/cassandra/cql.py
610–628 ↗(On Diff #10428)

yeah, i'll check.

I'm sad, i'm wondering why the new coverage report does not show up here :/

oh maybe because the tests are red because they need D2880...

ardumont updated this revision to Diff 10432.Mar 31 2020, 5:47 PM

Drop some dead code:

  • db.origin_visit_update
  • cql_runner.origin_visit_update

Build has FAILED

Patch application report for D2879 (id=10432)

Rebasing onto 2856004350...

Current branch diff-target is up to date.
Changes applied before test
commit 097abef82786c6127c640b8b3b9e79c9b30ca476
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit d98876b53d2ff29338be845b686cf81e38b32ea0
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 384e44ed97b78735473c2cc6561e00c9a110f626
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit 7dfc90c2ff1d8eb1277949867688fddd3e816edf
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit d5b700455dda0bcec38a815fa7e7c12b14c6387d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/37/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/37/console

ardumont edited the summary of this revision. (Show Details)Mar 31 2020, 5:50 PM
ardumont added a comment.EditedMar 31 2020, 5:54 PM

First:

$ cd swh-model
$ arc patch D2880

then:

$ cd swh-storage
$ ./.tox/py3/bin/pip install -e ../swh-model

Here we go, for tomorrow:

$ tox -- -x -s
----------- coverage: platform linux, python 3.7.3-final-0 -----------
Name                                                                                    Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------------------------------------------------------------
.tox/py3/lib/python3.7/site-packages/swh/storage/__init__.py                               37      0     28      1    98%
.tox/py3/lib/python3.7/site-packages/swh/storage/algos/__init__.py                          0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/algos/diff.py                            131     20     60     13    82%
.tox/py3/lib/python3.7/site-packages/swh/storage/algos/dir_iterators.py                   130      2     26      1    98%
.tox/py3/lib/python3.7/site-packages/swh/storage/algos/origin.py                           15      1      8      1    91%
.tox/py3/lib/python3.7/site-packages/swh/storage/algos/revisions_walker.py                162     50     72      5    62%
.tox/py3/lib/python3.7/site-packages/swh/storage/algos/snapshot.py                         10      1      4      1    86%
.tox/py3/lib/python3.7/site-packages/swh/storage/api/__init__.py                            0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/api/client.py                             28      2      4      1    91%
.tox/py3/lib/python3.7/site-packages/swh/storage/api/serializers.py                         9      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/api/server.py                             51     10     12      2    78%
.tox/py3/lib/python3.7/site-packages/swh/storage/buffer.py                                 50      1     26      2    96%
.tox/py3/lib/python3.7/site-packages/swh/storage/cassandra/__init__.py                      3      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/cassandra/common.py                        6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/cassandra/converters.py                   33      0      6      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/cassandra/cql.py                         327      6     64      4    97%
.tox/py3/lib/python3.7/site-packages/swh/storage/cassandra/schema.py                        8      0      4      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/cassandra/storage.py                     577     36    302     24    92%
.tox/py3/lib/python3.7/site-packages/swh/storage/cli.py                                    23      9      4      1    56%
.tox/py3/lib/python3.7/site-packages/swh/storage/common.py                                  1      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/converters.py                             84      2     34      4    95%
.tox/py3/lib/python3.7/site-packages/swh/storage/db.py                                    370     12     99      5    96%
.tox/py3/lib/python3.7/site-packages/swh/storage/exc.py                                    20      3      4      0    88%
.tox/py3/lib/python3.7/site-packages/swh/storage/filter.py                                 56      1     28      1    98%
.tox/py3/lib/python3.7/site-packages/swh/storage/in_memory.py                             687     30    376     25    94%
.tox/py3/lib/python3.7/site-packages/swh/storage/interface.py                             183     67      0      0    63%
.tox/py3/lib/python3.7/site-packages/swh/storage/metrics.py                                33      0      8      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/objstorage.py                             27      3      8      1    89%
.tox/py3/lib/python3.7/site-packages/swh/storage/retry.py                                  78      8     10      2    86%
.tox/py3/lib/python3.7/site-packages/swh/storage/storage.py                               702     48    270     23    91%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/__init__.py                          3      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/algos/__init__.py                    0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/algos/test_diff.py                 101      0      4      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/algos/test_dir_iterator.py          80      1     26      1    98%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/algos/test_origin.py                27      0      4      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/algos/test_revisions_walker.py      38      0     10      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/algos/test_snapshot.py              17      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/conftest.py                         93      1     22      2    97%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/generate_data_test.py               11      6      2      0    38%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/storage_data.py                     53      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_api_client.py                  29      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_api_client_dicts.py            28      1      0      0    96%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_buffer.py                     123      0      6      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_cassandra.py                  173     30     24      5    81%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_cassandra_converters.py        12      0      4      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_converters.py                  19      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_db.py                          10      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_exception.py                   12      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_filter.py                      61      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_in_memory.py                    4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_init.py                        38      0      4      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_metrics.py                     20      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_retry.py                      388      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_server.py                      32      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_storage.py                   1768     32    236     28    97%
.tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_utils.py                       20      0      6      0   100%
.tox/py3/lib/python3.7/site-packages/swh/storage/utils.py                                  30      2     10      2    90%
.tox/py3/lib/python3.7/site-packages/swh/storage/validate.py                               59      1     18      1    97%
.tox/py3/lib/python3.7/site-packages/swh/storage/writer.py                                 62      5     28      3    91%
-------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                    7152    391   1871    159    93%
ardumont edited the summary of this revision. (Show Details)Mar 31 2020, 5:57 PM
ardumont edited the test plan for this revision. (Show Details)

Build has FAILED

Patch application report for D2879 (id=10432)

Rebasing onto 2856004350...

Current branch diff-target is up to date.
Changes applied before test
commit 097abef82786c6127c640b8b3b9e79c9b30ca476
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit d98876b53d2ff29338be845b686cf81e38b32ea0
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 384e44ed97b78735473c2cc6561e00c9a110f626
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit 7dfc90c2ff1d8eb1277949867688fddd3e816edf
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit d5b700455dda0bcec38a815fa7e7c12b14c6387d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/38/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/38/console

ardumont edited the summary of this revision. (Show Details)Mar 31 2020, 6:04 PM
ardumont edited the summary of this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)Apr 1 2020, 10:17 AM
ardumont retitled this revision from wip: storage: Refactor internal implementations to use origin visit update representation to storage: Refactor internal implementations to use "origin visit update" model representation.

Build has FAILED

Patch application report for D2879 (id=10432)

Rebasing onto 2856004350...

Current branch diff-target is up to date.
Changes applied before test
commit 097abef82786c6127c640b8b3b9e79c9b30ca476
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit d98876b53d2ff29338be845b686cf81e38b32ea0
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit 384e44ed97b78735473c2cc6561e00c9a110f626
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit 7dfc90c2ff1d8eb1277949867688fddd3e816edf
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit d5b700455dda0bcec38a815fa7e7c12b14c6387d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/39/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/39/console

ardumont planned changes to this revision.Apr 1 2020, 10:20 AM

Build has FAILED

triggered the build too soon

ardumont updated this revision to Diff 10439.Apr 1 2020, 10:27 AM
ardumont edited the test plan for this revision. (Show Details)

Fix flake8 violations (due to dangling leftovers after rebase)

Build is green

Patch application report for D2879 (id=10439)

Rebasing onto 2856004350...

Current branch diff-target is up to date.
Changes applied before test
commit e98bda144538cb1a1dff5c8c020a6b20fd25f5f6
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update

commit f2313e5bde0d4315874f8b4a38ab30ceb3ff10a6
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:38:58 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update

commit a4ab716ab6183059d069eeeae861e5e09394362c
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Fri Mar 27 17:28:56 2020 +0100

    Add missing type annotations on origin_visit_get* endpoints

commit c46998bed64e905b7bf4ef73d8bb828da0527618
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Start using origin_visit_update

commit f7e56d4153acb78cb0a7ee6efdaaa618370349ce
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/40/ for more details.

ardumont updated this revision to Diff 10446.Apr 1 2020, 11:27 AM

Rebase on latest master, Split into 3 commits (1 per backend implem)

  • in_memory: Adapt internal implementations to use origin visit update
  • pg-storage: Adapt internal implementations to use origin visit update
  • cassandra: Adapt internal implementations to use origin visit update

Build is green

Patch application report for D2879 (id=10446)

Rebasing onto 9c221569de...

Current branch diff-target is up to date.
Changes applied before test
commit 3ed0be2cce3d58d6afa0063ee31850a2e1e9e72b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 4af691a68afa326d72d75ec378c19461a07df438
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 0ce207d5f309a1daee172a57aee118717f527eba
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/41/ for more details.

ardumont edited the summary of this revision. (Show Details)Apr 1 2020, 11:33 AM
ardumont added a project: Storage manager.
ardumont edited the summary of this revision. (Show Details)Apr 1 2020, 12:00 PM
ardumont edited the summary of this revision. (Show Details)
ardumont updated this revision to Diff 10447.Apr 1 2020, 12:05 PM

Rebase on latest master

Build is green

Patch application report for D2879 (id=10447)

Rebasing onto 377e6a8072...

Current branch diff-target is up to date.
Changes applied before test
commit 400dd86e25e4b64558e957f004a60a699caff87d
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Mon Mar 30 13:08:32 2020 +0200

    cassandra: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit 0cdd6dfa9c5b038f1ad2033c4b0baa908d48adc4
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Mar 26 14:15:17 2020 +0100

    pg-storage: Adapt internal implementations to use origin visit update
    
    Related to T2310

commit b29327fd7cb7316ae6ba3ad27f53971d696d324f
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Mar 25 17:53:48 2020 +0100

    in_memory: Adapt internal implementations to use origin visit update
    
    (pairing with @vlorentz)
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/42/ for more details.

ardumont edited the summary of this revision. (Show Details)Apr 1 2020, 12:12 PM
ardumont edited the summary of this revision. (Show Details)
ardumont abandoned this revision.Apr 1 2020, 12:18 PM

Closed in favor of D2937, D2938, D2939