Page MenuHomeSoftware Heritage
Differential D4272

swh.indexer.storage: Unify get_indexer_storage function with others
ClosedPublic
Actions

Authored by ardumont on Oct 15 2020, 3:46 PM.

Details

Reviewers
vlorentz
tenma
Group Reviewers
Reviewers
Maniphest Tasks
T1410: Kill implicit configuration: new configuration scheme
Commits
rDCIDX07c96743d726: swh.indexer.storage: Unify get_indexer_storage function with others
Summary

Trying to sense a bit better on how to converge with what's described in the hackmd
discussion [1]

(also, we have those kinds of tests on at least the storage and search).

Impacts [2]:

[1] https://hackmd.io/8hxTL4XMQoO2RVKtqFqM2g?view#Synthesis-of-discussion-tenmadouarddaolasd-2020-10-12

[2]

$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/cli.py:    from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/cli.py:        get_indexer_storage, ctx.obj["config"], "indexer_storage", indexer_storage_url
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/indexer.py:from swh.indexer.storage import INDEXER_CFG_KEY, PagedResult, Sha1, get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/indexer.py:        self.idx_storage = get_indexer_storage(**idx_storage)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/conftest.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/conftest.py:    idx_storage = get_indexer_storage("memory")
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_api_client.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_api_client.py:    server.storage = get_indexer_storage("local", db=swh_indexer_storage_postgresql.dsn)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/conftest.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/conftest.py:    return get_indexer_storage("local", db=swh_indexer_storage_postgresql.dsn)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:from swh.indexer.storage import IndexerStorage, get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:def test_init_get_indexer_storage_failure():
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:        get_indexer_storage("unknown-idx-storage")
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:def test_init_get_indexer_storage(class_name, expected_class, kwargs, mock_psycopg2):
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:        concrete_idx_storage = get_indexer_storage(class_name, **kwargs)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:        concrete_idx_storage = get_indexer_storage(class_name)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:def test_init_get_indexer_storage_deprecation_warning(
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_init.py:        concrete_idx_storage = get_indexer_storage(class_name, args=kwargs)
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_in_memory.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/tests/storage/test_in_memory.py:    return get_indexer_storage("memory")
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/storage/api/server.py:from swh.indexer.storage import INDEXER_CFG_KEY, get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/storage/api/server.py:        storage = get_indexer_storage(**app.config[INDEXER_CFG_KEY])
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/storage/__init__.py:def get_indexer_storage(cls: str, **kwargs) -> IndexerStorageInterface:
$SWH_ENVIRONMENT_HOME/swh-web/swh/web/config.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-web/swh/web/config.py:        swhweb_config["indexer_storage"] = get_indexer_storage(
$SWH_ENVIRONMENT_HOME/swh-web/swh/web/tests/data.py:from swh.indexer.storage import get_indexer_storage
$SWH_ENVIRONMENT_HOME/swh-web/swh/web/tests/data.py:    idx_storage = get_indexer_storage("memory")

Related to T1410

Test Plan

tox

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16342
Build 25161: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 25160: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
41 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.storage.test_storage::test_types
swh_indexer_storage = <swh.indexer.storage.IndexerStorage object at 0x7fd8e0e9f278> def test_types(swh_indexer_storage) -> None:
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.codemeta::swh.indexer.codemeta.merge_values
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.normalize_groupId
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.parse_licenses
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.parse_repositories
View Full Test Results (1 Failed · 340 Passed · 15 Skipped)

Event Timeline

ardumont created this revision.Oct 15 2020, 3:46 PM
ardumont added a child revision: D4273: indexer.storage: Add tests on signature type for all indexer storage implems.
swh-public-ci added a comment.Oct 15 2020, 3:47 PM

Build has FAILED

Patch application report for D4272 (id=15109)

Rebasing onto c6afd9d798...

Current branch diff-target is up to date.
Changes applied before test
commit 255c1961d615049ee2b19e819793a7b9a20d3230
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Oct 15 15:42:58 2020 +0200

    swh.indexer.storage: Unify get_indexer_storage function with others
    
    Related to T1410

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/88/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/88/console

Harbormaster failed remote builds in B16335: Diff 15109!Oct 15 2020, 3:47 PM
ardumont edited the summary of this revision. (Show Details)Oct 15 2020, 3:48 PM
ardumont updated this revision to Diff 15111.Oct 15 2020, 3:58 PM

Fix one instantiation (remains the test-types to fix)

swh-public-ci added a comment.Oct 15 2020, 4:00 PM

Build has FAILED

Patch application report for D4272 (id=15111)

Rebasing onto c6afd9d798...

Current branch diff-target is up to date.
Changes applied before test
commit fe522f0782e5c18f74e2b6add5b29ef8d0e5ff95
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Oct 15 15:42:58 2020 +0200

    swh.indexer.storage: Unify get_indexer_storage function with others
    
    Related to T1410

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/90/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/90/console

Harbormaster failed remote builds in B16338: Diff 15111!Oct 15 2020, 4:00 PM
vlorentz accepted this revision.Oct 15 2020, 4:35 PM
This revision is now accepted and ready to land.Oct 15 2020, 4:35 PM
ardumont edited the summary of this revision. (Show Details)Oct 15 2020, 4:51 PM
ardumont updated this revision to Diff 15115.Oct 15 2020, 5:30 PM
  • Add some more assertions
  • Simplify instantiation with the new code

TODO:

  • Remains the test_type to fix (no clue so far)
swh-public-ci added a comment.Oct 15 2020, 5:32 PM

Build has FAILED

Patch application report for D4272 (id=15115)

Rebasing onto c6afd9d798...

Current branch diff-target is up to date.
Changes applied before test
commit db4af8e674ece6b00df0cb747a6accd8d26aa550
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Oct 15 15:42:58 2020 +0200

    swh.indexer.storage: Unify get_indexer_storage function with others
    
    Related to T1410

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/91/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/91/console

Harbormaster failed remote builds in B16342: Diff 15115!Oct 15 2020, 5:32 PM
ardumont updated this revision to Diff 15117.Oct 15 2020, 5:38 PM
  • Move test_types into test_init test module
  • Fix that test
swh-public-ci added a comment.Oct 15 2020, 5:39 PM

Build is green

Patch application report for D4272 (id=15117)

Rebasing onto c6afd9d798...

Current branch diff-target is up to date.
Changes applied before test
commit 07c96743d726b668d96d847f1f3297680c780bb7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Oct 15 15:42:58 2020 +0200

    swh.indexer.storage: Unify get_indexer_storage function with others
    
    Related to T1410

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/92/ for more details.

Harbormaster completed remote builds in B16344: Diff 15117.Oct 15 2020, 5:39 PM
ardumont edited the summary of this revision. (Show Details)Oct 15 2020, 5:49 PM
ardumont mentioned this in D4275: swh.web.tests: Adapt get_indexer_storage to latest version.Oct 15 2020, 5:54 PM
ardumont edited the summary of this revision. (Show Details)
Closed by commit rDCIDX07c96743d726: swh.indexer.storage: Unify get_indexer_storage function with others (authored by ardumont). · Explain WhyOct 15 2020, 6:15 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDCIDX07c96743d726: swh.indexer.storage: Unify get_indexer_storage function with others.
ardumont edited the summary of this revision. (Show Details)Oct 15 2020, 6:44 PM
tenma mentioned this in T1410: Kill implicit configuration: new configuration scheme.Oct 16 2020, 10:13 AM
ardumont mentioned this in D4280: storage.api.server: Align configuration structure.Oct 16 2020, 10:33 AM

Revision Contents
Changeset List

PathSize
1 line
swh/
indexer/
storage/
62 lines
5 lines
tests/
2 lines
storage/
6 lines
6 lines
6 lines
52 lines
6 lines

Diff 15115

View Options

requirements.txt

Loading...
View Options

swh/indexer/storage/__init__.py

Loading...
View Options

swh/indexer/storage/interface.py

Loading...
View Options

swh/indexer/tests/conftest.py

Loading...
View Options

swh/indexer/tests/storage/conftest.py

Loading...
View Options

swh/indexer/tests/storage/test_api_client.py

Loading...
View Options

swh/indexer/tests/storage/test_in_memory.py

Loading...
View Options

swh/indexer/tests/storage/test_init.py

Loading...
View Options

swh/indexer/tests/storage/test_storage.py

Loading...
About · Developer information · Legal