Page MenuHomeSoftware Heritage
Differential D4330

Allow vault configuration without the args indirection
ClosedPublic
Actions

Authored by tenma on Oct 22 2020, 2:15 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDVAUcf6aa9558f02: Refactor vault configuration without the args indirection
Summary

Keep the old behavior to avoid breakage on part not yet migrated.
Detach configuration checking from instanciation in cookers similarly to most swh components.

Test Plan

tox

Diff Detail

Unit TestsFailed
View All

TimeTest
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_init_cookers::test_get_cooker_nominal[config_ok2]
config_ok = {'storage': {'cls': 'remote', 'url': 'mock://storage-url'}, 'vault': {'cls': 'remote', 'url': 'mock://vault-backend'}} tmp_path = PosixPath('/tmp/pytest-of-jenkins/pytest-0/test_get_cooker_nominal_config2') monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f1d2dbd7d68>
57 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_server::test_check_config_missing_key[cache]
missing_key = 'cache' swh_vault_config = {'cache': {'args': {'allow_delete': True, 'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_1', '...oot': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_1', 'slicing': '0:1/1:5'}, 'cls': 'pathslicing'}}}
57 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_server::test_check_config_missing_key[scheduler]
missing_key = 'scheduler' swh_vault_config = {'cache': {'args': {'allow_delete': True, 'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_2', '...oot': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_2', 'slicing': '0:1/1:5'}, 'cls': 'pathslicing'}}}
59 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_server::test_check_config_missing_key[storage]
missing_key = 'storage' swh_vault_config = {'cache': {'args': {'allow_delete': True, 'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_0', '...oot': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_0', 'slicing': '0:1/1:5'}, 'cls': 'pathslicing'}}}
54 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_server::test_check_config_ok[cache]
missing_key = 'cache' swh_vault_config = {'cache': {'args': {'allow_delete': True, 'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_ok_cache_0', 'sli...{'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_ok_cache_0', 'slicing': '0:1/1:5'}, 'cls': 'pathslicing'}}}
View Full Test Results (12 Failed · 56 Passed)

Event Timeline

ardumont created this revision.Oct 22 2020, 2:15 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 22 2020, 2:15 PM
swh-public-ci added a comment.Oct 22 2020, 2:16 PM

Build is green

Patch application report for D4330 (id=15323)

Could not rebase; Attempt merge onto 97a860997b...

Updating 97a8609..6b84f80
Fast-forward
 requirements.txt                     |   2 +-
 swh/vault/api/client.py              |  48 +------
 swh/vault/api/server.py              | 236 ++++++++++-------------------------
 swh/vault/backend.py                 | 176 +++++++++++++++++---------
 swh/vault/cookers/__init__.py        |  24 +++-
 swh/vault/cookers/base.py            |   4 +-
 swh/vault/interface.py               |  68 ++++++++++
 swh/vault/tests/conftest.py          |  17 ++-
 swh/vault/tests/test_backend.py      |  22 +++-
 swh/vault/tests/test_cookers_base.py |   6 +-
 swh/vault/tests/test_init_cookers.py | 112 +++++++++++++++++
 swh/vault/tests/test_server.py       | 128 ++++++++++++++++---
 12 files changed, 529 insertions(+), 314 deletions(-)
 create mode 100644 swh/vault/interface.py
 create mode 100644 swh/vault/tests/test_init_cookers.py
Changes applied before test
commit 6b84f803c48a16afc172de869c87924fd421b886
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

commit c14a0f2d6139a55208da6dee53e29d56ce3c7543
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 18:12:18 2020 +0200

    vault.server: Introduce typed VaultInterface
    
    This allows to drop redundant code in between rpc server and clients using the
    base class from swh.core.api.async module.

commit eb868d0e690e9890172b699184bcffb80696fe89
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 20 15:15:52 2020 +0200

    Add tests on current configuration check for cooker instantiation

commit e5e428481c76966cf253d914ca1276c7c9749e69
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:06:00 2020 +0200

    api.server: Add types and tests on configuration checks

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/35/ for more details.

Harbormaster completed remote builds in B16551: Diff 15323.Oct 22 2020, 2:16 PM
ardumont updated this revision to Diff 15335.Oct 22 2020, 4:13 PM

Rebase

swh-public-ci added a comment.Oct 22 2020, 4:15 PM

Build is green

Patch application report for D4330 (id=15335)

Could not rebase; Attempt merge onto eb868d0e69...

Updating eb868d0..d547c5a
Fast-forward
 requirements.txt                     |   2 +-
 swh/vault/api/client.py              |  48 +------
 swh/vault/api/server.py              | 252 +++++++++--------------------------
 swh/vault/backend.py                 | 176 +++++++++++++++---------
 swh/vault/cookers/__init__.py        |   3 +-
 swh/vault/cookers/base.py            |   2 +-
 swh/vault/interface.py               |  68 ++++++++++
 swh/vault/tests/conftest.py          |  20 ++-
 swh/vault/tests/test_backend.py      |  22 ++-
 swh/vault/tests/test_cookers_base.py |   6 +-
 swh/vault/tests/test_init_cookers.py |   5 +-
 swh/vault/tests/test_server.py       |  83 +++++++++---
 12 files changed, 359 insertions(+), 328 deletions(-)
 create mode 100644 swh/vault/interface.py
Changes applied before test
commit d547c5a73ef816c7f20c1a10986db4c8be4d171a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

commit 144c1f10c4b7d1a3bb1d2d87109b3f27a38174b3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 18:12:18 2020 +0200

    vault.server: Introduce typed VaultInterface
    
    This allows to drop redundant code in between rpc server and clients using the
    base class from swh.core.api.async module.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/37/ for more details.

Harbormaster completed remote builds in B16562: Diff 15335.Oct 22 2020, 4:15 PM
ardumont updated this revision to Diff 15341.Oct 22 2020, 4:43 PM

Rebase

swh-public-ci added a comment.Oct 22 2020, 4:45 PM

Build is green

Patch application report for D4330 (id=15341)

Could not rebase; Attempt merge onto eb868d0e69...

Updating eb868d0..61ed8a6
Fast-forward
 requirements.txt                     |   2 +-
 swh/vault/api/client.py              |  48 +------
 swh/vault/api/server.py              | 261 ++++++++++-------------------------
 swh/vault/backend.py                 | 176 ++++++++++++++---------
 swh/vault/cookers/__init__.py        |   3 +-
 swh/vault/cookers/base.py            |   2 +-
 swh/vault/interface.py               |  68 +++++++++
 swh/vault/tests/conftest.py          |  20 ++-
 swh/vault/tests/test_backend.py      |  22 ++-
 swh/vault/tests/test_cookers_base.py |   6 +-
 swh/vault/tests/test_init_cookers.py |   5 +-
 swh/vault/tests/test_server.py       | 114 ++++++++++++---
 12 files changed, 400 insertions(+), 327 deletions(-)
 create mode 100644 swh/vault/interface.py
Changes applied before test
commit 61ed8a6ed41be92b6c38a74b094c926371ee87df
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

commit 4d49f567fde289b3316236a1a6ed65f3bc95fff4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 18:12:18 2020 +0200

    vault.server: Introduce typed VaultInterface
    
    This allows to drop redundant code in between rpc server and clients using the
    base class from swh.core.api.async module.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/39/ for more details.

Harbormaster completed remote builds in B16568: Diff 15341.Oct 22 2020, 4:45 PM
vlorentz added a subscriber: vlorentz.Oct 23 2020, 11:54 AM
vlorentz added inline comments.
swh/vault/api/server.py
71–77

alternatively:

args = {**vcfg, **vcfg.get("args")}

and undo the other changes.

swh/vault/cookers/__init__.py
52–54

with the same change as above, you can remove that if block

vlorentz added inline comments.Oct 23 2020, 11:55 AM
swh/vault/cookers/__init__.py
52–54

er, forget the part about this if

ardumont updated this revision to Diff 15393.Oct 23 2020, 6:55 PM

Adapt according to review

swh-public-ci added a comment.Oct 23 2020, 6:57 PM

Build has FAILED

Patch application report for D4330 (id=15393)

Rebasing onto 08c35ed0f1...

Current branch diff-target is up to date.
Changes applied before test
commit 84f52cae1200af662253fd5ad2772db6ebee3e5b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/46/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/46/console

Harbormaster failed remote builds in B16620: Diff 15393!Oct 23 2020, 6:57 PM
ardumont updated this revision to Diff 15395.Oct 23 2020, 7:02 PM

Revert to the working initial implementation

swh-public-ci added a comment.Oct 23 2020, 7:03 PM

Build is green

Patch application report for D4330 (id=15395)

Rebasing onto 08c35ed0f1...

Current branch diff-target is up to date.
Changes applied before test
commit 945ebc9bdb1f5089355d19b1957cd1f09babc43c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/47/ for more details.

Harbormaster completed remote builds in B16622: Diff 15395.Oct 23 2020, 7:03 PM
ardumont added inline comments.Oct 23 2020, 7:04 PM
swh/vault/api/server.py
71–77

That does not work, see the previous run which failed.
I've reverted to my naive original implementation which works.

tenma added a subscriber: tenma.Oct 23 2020, 7:04 PM
tenma added inline comments.
swh/vault/api/server.py
71

hum .get() may return None -> crash, bad vlorentz
easiest is to change to .get('args', {})

swh/vault/cookers/__init__.py
53

same as supra

tenma commandeered this revision.Oct 27 2020, 2:27 PM
tenma added a reviewer: ardumont.
tenma updated this revision to Diff 15442.Oct 27 2020, 4:24 PM

Refactored configuration checking in vault server+worker

swh-public-ci added a comment.Oct 27 2020, 4:26 PM

Build is green

Patch application report for D4330 (id=15442)

Rebasing onto 35438cadcd...

Current branch diff-target is up to date.
Changes applied before test
commit 8e4777204e38cad525760d13b58906b66b52972a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 27 14:27:59 2020 +0100

    Refactor vault configuration without the args indirection
    
    Keep the old behavior to avoid breakage on part not yet migrated.
    Detach configuration checking from instanciation in cookers similarly
    to most swh components.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/48/ for more details.

Harbormaster completed remote builds in B16670: Diff 15442.Oct 27 2020, 4:26 PM
tenma edited the summary of this revision. (Show Details)Oct 27 2020, 5:36 PM
ardumont accepted this revision.Oct 27 2020, 6:23 PM
This revision is now accepted and ready to land.Oct 27 2020, 6:23 PM
tenma updated this revision to Diff 15466.Oct 28 2020, 10:03 AM

Reset author

swh-public-ci added a comment.Oct 28 2020, 10:04 AM

Build is green

Patch application report for D4330 (id=15466)

Rebasing onto 35438cadcd...

Current branch diff-target is up to date.
Changes applied before test
commit cf6aa9558f0258f7aaec733f65d46a08b7d4803c
Author: tenma <tenma+swh@mailbox.org>
Date:   Wed Oct 28 09:58:30 2020 +0100

    Refactor vault configuration without the args indirection
    
    Keep the old behavior to avoid breakage on part not yet migrated.
    Detach configuration checking from instanciation in cookers similarly
    to most swh components.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/50/ for more details.

Harbormaster completed remote builds in B16690: Diff 15466.Oct 28 2020, 10:04 AM
Closed by commit rDVAUcf6aa9558f02: Refactor vault configuration without the args indirection (authored by tenma). · Explain WhyOct 28 2020, 10:11 AM
This revision was automatically updated to reflect the committed changes.
tenma added a commit: rDVAUcf6aa9558f02: Refactor vault configuration without the args indirection.

Revision Contents
Changeset List

PathSize
swh/
vault/
api/
2 lines
cookers/
3 lines
tests/
2 lines
5 lines
6 lines

Diff 15393

View Options

swh/vault/api/server.py

Loading...
View Options

swh/vault/cookers/__init__.py

Loading...
View Options

swh/vault/tests/conftest.py

Loading...
View Options

swh/vault/tests/test_init_cookers.py

Loading...
View Options

swh/vault/tests/test_server.py

Loading...
About · Developer information · Legal