Differential D3276

storage*: Remove origin-visit-update endpoint
ClosedPublic
Actions

Authored by ardumont on Jun 12 2020, 6:55 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T2310: Make origin visits immutable

Commits

rDSTOc7f3060461a5: storage*: Remove origin-visit-update endpoint

Summary

Api clients migrated to use origin-visit-status-add already.

This diff rewrote the tests to use origin-visit-status-add, then drop the endpoint implementations altogether.

Depends on D3275
Related to T2310

Test Plan

tox

Diff Detail

Repository

rDSTO Storage manager

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Jun 12 2020, 6:55 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 12 2020, 6:55 PM

ardumont edited the summary of this revision. (Show Details)Jun 12 2020, 6:57 PM

Build is green

Patch application report for D3276 (id=11617)

Could not rebase; Attempt merge onto 37c45300bc...

Updating 37c4530..64c1e56
Fast-forward
 swh/storage/cassandra/storage.py       |  44 ---
 swh/storage/db.py                      |  25 --
 swh/storage/in_memory.py               |  50 ---
 swh/storage/interface.py               |  27 --
 swh/storage/storage.py                 |  58 ---
 swh/storage/tests/test_kafka_writer.py |  14 +-
 swh/storage/tests/test_retry.py        |  81 ----
 swh/storage/tests/test_storage.py      | 672 ++++++++++++++-------------------
 swh/storage/writer.py                  |   3 -
 9 files changed, 298 insertions(+), 676 deletions(-)

Changes applied before test

commit 64c1e56e35cdc6d98acc51a00adb642184a0050b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 12 18:51:25 2020 +0200

    storage*: Remove origin-visit-update endpoint
    
    Related to T2310

commit 33efdb0dd488cfb6f8d189a696863dd540fd3734
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 12 18:51:47 2020 +0200

    storage*: Do not write twice origin-visit-status in journal
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/267/ for more details.

Harbormaster completed remote builds in B12824: Diff 11617.Jun 12 2020, 7:08 PM

ardumont mentioned this in T2310: Make origin visits immutable.Jun 12 2020, 7:15 PM

ardumont added inline comments.Jun 12 2020, 7:26 PM

swh/storage/tests/test_kafka_writer.py
59	question ;)

ardumont mentioned this in D3273: replay: Replay origin-visit and origin-visit-status.Jun 13 2020, 8:51 AM

Drop last origin-visit-update reference

Depends on D3273

ardumont added inline comments.Jun 13 2020, 9:12 AM

swh/storage/tests/test_kafka_writer.py
59	I removed the unneeded part (in red to the left)

Build is green

Patch application report for D3276 (id=11623)

Could not rebase; Attempt merge onto 33efdb0dd4...

Updating 33efdb0..4873955
Fast-forward
 swh/storage/cassandra/storage.py       |  43 --
 swh/storage/db.py                      |  25 --
 swh/storage/in_memory.py               |  56 +--
 swh/storage/interface.py               |  27 --
 swh/storage/replay.py                  |  15 +-
 swh/storage/storage.py                 |  58 ---
 swh/storage/tests/test_kafka_writer.py |   8 +-
 swh/storage/tests/test_replay.py       |   7 +-
 swh/storage/tests/test_retry.py        |  81 ----
 swh/storage/tests/test_storage.py      | 714 +++++++++++++++------------------
 swh/storage/validate.py                |  32 +-
 swh/storage/writer.py                  |   3 -
 12 files changed, 361 insertions(+), 708 deletions(-)

Changes applied before test

commit 487395580de80b1d6f0af858ebf32f0622a659ca
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 12 18:51:25 2020 +0200

    storage*: Remove origin-visit-update endpoint
    
    Related to T2310

commit 645f104bd401dc0399bbdc61349210ad81300f88
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 11 17:16:29 2020 +0200

    replay: Replay origin-visit and origin-visit-status
    
    This now uses the respective origin-visit-add and origin-visit-status-add
    endpoints.
    
    Related to T2310

commit 1d8a54ba8312d15c9fd843efe3754691576398af
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jun 13 08:40:29 2020 +0200

    in_memory: Make origin-visit-status-add do on conflict ignore
    
    Aligning its behavior with the other backend.
    
    Related to T2310

commit 874da2dee1957ac5e9f5b17a81e11d9d9e8690f0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jun 13 08:37:57 2020 +0200

    Start migrating the validate proxy toward using BaseModel objects
    
    This will allow to progress incrementally towards removing it.
    
    When it allows to use BaseModel objects everywhere (and tests in test_storage
    are adapted to use this property), it will be time to remove it entirely (as
    it's only used in test).
    
    It's preparatory work for future diffs.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/272/ for more details.

Harbormaster completed remote builds in B12830: Diff 11623.Jun 13 2020, 9:17 AM

vlorentz accepted this revision.Jun 15 2020, 11:05 AM

This revision is now accepted and ready to land.Jun 15 2020, 11:05 AM

Rebase on latest D3273

Build is green

Patch application report for D3276 (id=11638)

Could not rebase; Attempt merge onto 0183fec91d...

Updating 0183fec..c7f3060
Fast-forward
 swh/storage/cassandra/storage.py       |  43 ---
 swh/storage/db.py                      |  25 --
 swh/storage/in_memory.py               |  50 ---
 swh/storage/interface.py               |  27 --
 swh/storage/replay.py                  |  15 +-
 swh/storage/storage.py                 |  58 ---
 swh/storage/tests/test_kafka_writer.py |   8 +-
 swh/storage/tests/test_replay.py       |   7 +-
 swh/storage/tests/test_retry.py        |  81 ----
 swh/storage/tests/test_storage.py      | 672 ++++++++++++++-------------------
 swh/storage/writer.py                  |   3 -
 11 files changed, 303 insertions(+), 686 deletions(-)

Changes applied before test

commit c7f3060461a53480f9803c099489f7f27441eee5
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 12 18:51:25 2020 +0200

    storage*: Remove origin-visit-update endpoint
    
    Related to T2310

commit 2bcbc82b708550415ce23a8a91830ad2b75f848c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 11 17:16:29 2020 +0200

    replay: Replay origin-visit and origin-visit-status
    
    This now uses the respective origin-visit-add and origin-visit-status-add
    endpoints.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/280/ for more details.

Harbormaster completed remote builds in B12845: Diff 11638.Jun 15 2020, 2:25 PM

ardumont added a child revision: D3281: storage*: Drop origin_visit_upsert endpoint.Jun 15 2020, 2:27 PM

Closed by commit rDSTOc7f3060461a5: storage*: Remove origin-visit-update endpoint (authored by ardumont). · Explain WhyJun 15 2020, 3:05 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSTOc7f3060461a5: storage*: Remove origin-visit-update endpoint.