Page MenuHomeSoftware Heritage

swh.indexer.storage: Unify get_indexer_storage function with others
ClosedPublic

Authored by ardumont on Oct 15 2020, 3:46 PM.

Details

Summary

Trying to sense a bit better on how to converge with what's described in the hackmd
discussion [1]

(also, we have those kinds of tests on at least the storage and search).

Impacts [2]:

[1] https://hackmd.io/8hxTL4XMQoO2RVKtqFqM2g?view#Synthesis-of-discussion-tenmadouarddaolasd-2020-10-12

[2]

$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/cli.py:    from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/cli.py:        get_indexer_storage, ctx.obj["config"], "indexer_storage", indexer_storage_url
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/indexer.py:from swh.indexer.storage import INDEXER_CFG_KEY, PagedResult, Sha1, get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/indexer.py:        self.idx_storage = get_indexer_storage(**idx_storage)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/conftest.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/conftest.py:    idx_storage = get_indexer_storage("memory")
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_api_client.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_api_client.py:    server.storage = get_indexer_storage("local", db=swh_indexer_storage_postgresql.dsn)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/conftest.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/conftest.py:    return get_indexer_storage("local", db=swh_indexer_storage_postgresql.dsn)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:from swh.indexer.storage import IndexerStorage, get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:def test_init_get_indexer_storage_failure():
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:        get_indexer_storage("unknown-idx-storage")
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:def test_init_get_indexer_storage(class_name, expected_class, kwargs, mock_psycopg2):
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:        concrete_idx_storage = get_indexer_storage(class_name, **kwargs)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:        concrete_idx_storage = get_indexer_storage(class_name)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:def test_init_get_indexer_storage_deprecation_warning(
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:        concrete_idx_storage = get_indexer_storage(class_name, args=kwargs)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_in_memory.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_in_memory.py:    return get_indexer_storage("memory")
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/storage/api/server.py:from swh.indexer.storage import INDEXER_CFG_KEY, get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/storage/api/server.py:        storage = get_indexer_storage(**app.config[INDEXER_CFG_KEY])
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/storage/__init__.py:def get_indexer_storage(cls: str, **kwargs) -> IndexerStorageInterface:
$SWH_ENVIRONMENT_HOME/swh-web/swh/web/config.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-web/swh/web/config.py:        swhweb_config["indexer_storage"] = get_indexer_storage(
$SWH_ENVIRONMENT_HOME/swh-web/swh/web/tests/data.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-web/swh/web/tests/data.py:    idx_storage = get_indexer_storage("memory")

Related to T1410

Test Plan

tox

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16342
Build 25161: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 25160: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D4272 (id=15109)

Rebasing onto c6afd9d798...

Current branch diff-target is up to date.
Changes applied before test
commit 255c1961d615049ee2b19e819793a7b9a20d3230
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Oct 15 15:42:58 2020 +0200

    swh.indexer.storage: Unify get_indexer_storage function with others
    
    Related to T1410

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/88/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/88/console

Fix one instantiation (remains the test-types to fix)

Build has FAILED

Patch application report for D4272 (id=15111)

Rebasing onto c6afd9d798...

Current branch diff-target is up to date.
Changes applied before test
commit fe522f0782e5c18f74e2b6add5b29ef8d0e5ff95
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Oct 15 15:42:58 2020 +0200

    swh.indexer.storage: Unify get_indexer_storage function with others
    
    Related to T1410

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/90/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/90/console

This revision is now accepted and ready to land.Oct 15 2020, 4:35 PM
  • Add some more assertions
  • Simplify instantiation with the new code

TODO:

  • Remains the test_type to fix (no clue so far)

Build has FAILED

Patch application report for D4272 (id=15115)

Rebasing onto c6afd9d798...

Current branch diff-target is up to date.
Changes applied before test
commit db4af8e674ece6b00df0cb747a6accd8d26aa550
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Oct 15 15:42:58 2020 +0200

    swh.indexer.storage: Unify get_indexer_storage function with others
    
    Related to T1410

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/91/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/91/console

  • Move test_types into test_init test module
  • Fix that test

Build is green

Patch application report for D4272 (id=15117)

Rebasing onto c6afd9d798...

Current branch diff-target is up to date.
Changes applied before test
commit 07c96743d726b668d96d847f1f3297680c780bb7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Oct 15 15:42:58 2020 +0200

    swh.indexer.storage: Unify get_indexer_storage function with others
    
    Related to T1410

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/92/ for more details.