Page MenuHomeSoftware Heritage

Allow vault configuration without the args indirection
ClosedPublic

Authored by tenma on Oct 22 2020, 2:15 PM.

Details

Summary

Keep the old behavior to avoid breakage on part not yet migrated.
Detach configuration checking from instanciation in cookers similarly to most swh components.

Test Plan

tox

Diff Detail

Unit TestsFailed

TimeTest
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_init_cookers::test_get_cooker_nominal[config_ok2]
config_ok = {'storage': {'cls': 'remote', 'url': 'mock://storage-url'}, 'vault': {'cls': 'remote', 'url': 'mock://vault-backend'}} tmp_path = PosixPath('/tmp/pytest-of-jenkins/pytest-0/test_get_cooker_nominal_config2') monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f1d2dbd7d68>
57 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_server::test_check_config_missing_key[cache]
missing_key = 'cache' swh_vault_config = {'cache': {'args': {'allow_delete': True, 'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_1', '...oot': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_1', 'slicing': '0:1/1:5'}, 'cls': 'pathslicing'}}}
57 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_server::test_check_config_missing_key[scheduler]
missing_key = 'scheduler' swh_vault_config = {'cache': {'args': {'allow_delete': True, 'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_2', '...oot': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_2', 'slicing': '0:1/1:5'}, 'cls': 'pathslicing'}}}
59 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_server::test_check_config_missing_key[storage]
missing_key = 'storage' swh_vault_config = {'cache': {'args': {'allow_delete': True, 'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_0', '...oot': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_missing_key_0', 'slicing': '0:1/1:5'}, 'cls': 'pathslicing'}}}
54 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.vault.tests.test_server::test_check_config_ok[cache]
missing_key = 'cache' swh_vault_config = {'cache': {'args': {'allow_delete': True, 'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_ok_cache_0', 'sli...{'root': '/tmp/pytest-of-jenkins/pytest-0/test_check_config_ok_cache_0', 'slicing': '0:1/1:5'}, 'cls': 'pathslicing'}}}
View Full Test Results (12 Failed · 56 Passed)

Event Timeline

Build is green

Patch application report for D4330 (id=15323)

Could not rebase; Attempt merge onto 97a860997b...

Updating 97a8609..6b84f80
Fast-forward
 requirements.txt                     |   2 +-
 swh/vault/api/client.py              |  48 +------
 swh/vault/api/server.py              | 236 ++++++++++-------------------------
 swh/vault/backend.py                 | 176 +++++++++++++++++---------
 swh/vault/cookers/__init__.py        |  24 +++-
 swh/vault/cookers/base.py            |   4 +-
 swh/vault/interface.py               |  68 ++++++++++
 swh/vault/tests/conftest.py          |  17 ++-
 swh/vault/tests/test_backend.py      |  22 +++-
 swh/vault/tests/test_cookers_base.py |   6 +-
 swh/vault/tests/test_init_cookers.py | 112 +++++++++++++++++
 swh/vault/tests/test_server.py       | 128 ++++++++++++++++---
 12 files changed, 529 insertions(+), 314 deletions(-)
 create mode 100644 swh/vault/interface.py
 create mode 100644 swh/vault/tests/test_init_cookers.py
Changes applied before test
commit 6b84f803c48a16afc172de869c87924fd421b886
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

commit c14a0f2d6139a55208da6dee53e29d56ce3c7543
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 18:12:18 2020 +0200

    vault.server: Introduce typed VaultInterface
    
    This allows to drop redundant code in between rpc server and clients using the
    base class from swh.core.api.async module.

commit eb868d0e690e9890172b699184bcffb80696fe89
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 20 15:15:52 2020 +0200

    Add tests on current configuration check for cooker instantiation

commit e5e428481c76966cf253d914ca1276c7c9749e69
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:06:00 2020 +0200

    api.server: Add types and tests on configuration checks

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/35/ for more details.

Build is green

Patch application report for D4330 (id=15335)

Could not rebase; Attempt merge onto eb868d0e69...

Updating eb868d0..d547c5a
Fast-forward
 requirements.txt                     |   2 +-
 swh/vault/api/client.py              |  48 +------
 swh/vault/api/server.py              | 252 +++++++++--------------------------
 swh/vault/backend.py                 | 176 +++++++++++++++---------
 swh/vault/cookers/__init__.py        |   3 +-
 swh/vault/cookers/base.py            |   2 +-
 swh/vault/interface.py               |  68 ++++++++++
 swh/vault/tests/conftest.py          |  20 ++-
 swh/vault/tests/test_backend.py      |  22 ++-
 swh/vault/tests/test_cookers_base.py |   6 +-
 swh/vault/tests/test_init_cookers.py |   5 +-
 swh/vault/tests/test_server.py       |  83 +++++++++---
 12 files changed, 359 insertions(+), 328 deletions(-)
 create mode 100644 swh/vault/interface.py
Changes applied before test
commit d547c5a73ef816c7f20c1a10986db4c8be4d171a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

commit 144c1f10c4b7d1a3bb1d2d87109b3f27a38174b3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 18:12:18 2020 +0200

    vault.server: Introduce typed VaultInterface
    
    This allows to drop redundant code in between rpc server and clients using the
    base class from swh.core.api.async module.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/37/ for more details.

Build is green

Patch application report for D4330 (id=15341)

Could not rebase; Attempt merge onto eb868d0e69...

Updating eb868d0..61ed8a6
Fast-forward
 requirements.txt                     |   2 +-
 swh/vault/api/client.py              |  48 +------
 swh/vault/api/server.py              | 261 ++++++++++-------------------------
 swh/vault/backend.py                 | 176 ++++++++++++++---------
 swh/vault/cookers/__init__.py        |   3 +-
 swh/vault/cookers/base.py            |   2 +-
 swh/vault/interface.py               |  68 +++++++++
 swh/vault/tests/conftest.py          |  20 ++-
 swh/vault/tests/test_backend.py      |  22 ++-
 swh/vault/tests/test_cookers_base.py |   6 +-
 swh/vault/tests/test_init_cookers.py |   5 +-
 swh/vault/tests/test_server.py       | 114 ++++++++++++---
 12 files changed, 400 insertions(+), 327 deletions(-)
 create mode 100644 swh/vault/interface.py
Changes applied before test
commit 61ed8a6ed41be92b6c38a74b094c926371ee87df
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

commit 4d49f567fde289b3316236a1a6ed65f3bc95fff4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 18:12:18 2020 +0200

    vault.server: Introduce typed VaultInterface
    
    This allows to drop redundant code in between rpc server and clients using the
    base class from swh.core.api.async module.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/39/ for more details.

vlorentz added inline comments.
swh/vault/api/server.py
71–77

alternatively:

args = {**vcfg, **vcfg.get("args")}

and undo the other changes.

swh/vault/cookers/__init__.py
52–54

with the same change as above, you can remove that if block

swh/vault/cookers/__init__.py
52–54

er, forget the part about this if

Build has FAILED

Patch application report for D4330 (id=15393)

Rebasing onto 08c35ed0f1...

Current branch diff-target is up to date.
Changes applied before test
commit 84f52cae1200af662253fd5ad2772db6ebee3e5b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/46/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/46/console

Revert to the working initial implementation

Build is green

Patch application report for D4330 (id=15395)

Rebasing onto 08c35ed0f1...

Current branch diff-target is up to date.
Changes applied before test
commit 945ebc9bdb1f5089355d19b1957cd1f09babc43c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 16 20:13:37 2020 +0200

    Allow vault configuration without the args indirection
    
    This keeps the old behavior to avoid breakage on part not yet migrated

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/47/ for more details.

swh/vault/api/server.py
71–77

That does not work, see the previous run which failed.
I've reverted to my naive original implementation which works.

tenma added inline comments.
swh/vault/api/server.py
71

hum .get() may return None -> crash, bad vlorentz
easiest is to change to .get('args', {})

swh/vault/cookers/__init__.py
53

same as supra

tenma added a reviewer: ardumont.

Refactored configuration checking in vault server+worker

Build is green

Patch application report for D4330 (id=15442)

Rebasing onto 35438cadcd...

Current branch diff-target is up to date.
Changes applied before test
commit 8e4777204e38cad525760d13b58906b66b52972a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 27 14:27:59 2020 +0100

    Refactor vault configuration without the args indirection
    
    Keep the old behavior to avoid breakage on part not yet migrated.
    Detach configuration checking from instanciation in cookers similarly
    to most swh components.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/48/ for more details.

This revision is now accepted and ready to land.Oct 27 2020, 6:23 PM

Build is green

Patch application report for D4330 (id=15466)

Rebasing onto 35438cadcd...

Current branch diff-target is up to date.
Changes applied before test
commit cf6aa9558f0258f7aaec733f65d46a08b7d4803c
Author: tenma <tenma+swh@mailbox.org>
Date:   Wed Oct 28 09:58:30 2020 +0100

    Refactor vault configuration without the args indirection
    
    Keep the old behavior to avoid breakage on part not yet migrated.
    Detach configuration checking from instanciation in cookers similarly
    to most swh components.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/50/ for more details.