Paths

Table of Contentst

Differential D3683

storage.in_memory: Fix origin_list implementation
AbandonedPublic
Actions

Authored by ardumont on Aug 1 2020, 11:16 AM.

Details

Reviewers

None

Group Reviewers

Reviewers

Maniphest Tasks

T645: Type swh-storage endpoints with swh.model objects

Summary

This kept on creating unrelated errors on other modules (see D3682 for
the most recent but not only)

Related to T645

Test Plan

tox

Diff Detail

Repository

rDSTO Storage manager

Branch

drop-origin-get-range

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 14207
Build 21840: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 21839: arc lint + arc unit

Event Timeline

ardumont created this revision.Aug 1 2020, 11:16 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptAug 1 2020, 11:16 AM

Build is green

Patch application report for D3683 (id=12966)

Could not rebase; Attempt merge onto 92f1183de0...

Updating 92f1183d..e2e6a1b8
Fast-forward
 swh/storage/algos/origin.py            | 32 ++++---------
 swh/storage/cassandra/storage.py       | 12 ++++-
 swh/storage/in_memory.py               | 44 ++++++++++--------
 swh/storage/interface.py               | 30 +++++-------
 swh/storage/storage.py                 | 14 +++++-
 swh/storage/tests/algos/test_origin.py | 84 +++++-----------------------------
 swh/storage/tests/test_cassandra.py    |  8 ----
 swh/storage/tests/test_storage.py      | 62 +++++++++----------------
 8 files changed, 102 insertions(+), 184 deletions(-)

Changes applied before test

commit e2e6a1b825b25f58de35ca5c118801f4a87c8e2a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 11:15:56 2020 +0200

    storage.in_memory: Fix origin_list implementation
    
    This kept on creating unrelated errors on other modules.
    
    Related to T645

commit 77757a0dccb429bd84154df11ad58c8442d75b79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

commit 87c5ba223508b212ed6939496a6380a771b05e43
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 09:07:12 2020 +0200

    storage*: Do not allow unknown visit status in origin_visit*_get_latest
    
    That makes some storage (pg-storage) fail without that filtering.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/648/ for more details.

Harbormaster completed remote builds in B14204: Diff 12966.Aug 1 2020, 11:24 AM

Add scenario with page-token

tests need storage > v0.11.8 to pass

Build has FAILED

Patch application report for D3683 (id=12967)

Rebasing onto 849d063758...

Current branch diff-target is up to date.

Changes applied before test

commit 95882a4212324f828a13c19ddc0ed8284e93362b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 10:03:23 2020 +0200

    cli.task: Migrate scheduler cli to latest storage change on iter_origins
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/52/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/52/console

Harbormaster failed remote builds in B14205: Diff 12967!Aug 1 2020, 11:28 AM

Drop empty line

Build is green

Patch application report for D3683 (id=12969)

Could not rebase; Attempt merge onto 92f1183de0...

Updating 92f1183d..6524b2ab
Fast-forward
 swh/storage/algos/origin.py            | 32 ++++---------
 swh/storage/cassandra/storage.py       | 12 ++++-
 swh/storage/in_memory.py               | 45 ++++++++++--------
 swh/storage/interface.py               | 30 +++++-------
 swh/storage/storage.py                 | 14 +++++-
 swh/storage/tests/algos/test_origin.py | 84 +++++-----------------------------
 swh/storage/tests/test_cassandra.py    |  8 ----
 swh/storage/tests/test_storage.py      | 62 +++++++++----------------
 8 files changed, 102 insertions(+), 185 deletions(-)

Changes applied before test

commit 6524b2ab578733d49601b80bc7d9ce30cef27b39
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 11:15:56 2020 +0200

    storage.in_memory: Fix origin_list implementation
    
    This kept on creating unrelated errors on other modules.
    
    Related to T645

commit 77757a0dccb429bd84154df11ad58c8442d75b79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

commit 87c5ba223508b212ed6939496a6380a771b05e43
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 09:07:12 2020 +0200

    storage*: Do not allow unknown visit status in origin_visit*_get_latest
    
    That makes some storage (pg-storage) fail without that filtering.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/649/ for more details.

Harbormaster completed remote builds in B14207: Diff 12969.Aug 1 2020, 11:36 AM

I don't understand what the issue was. Could you add a test that triggers it?

I don't understand what the issue was. Could you add a test that triggers it?

ack, it's an order issue which is hidden in the current test as it's sorting out both the input and the output prior to the comparison.

Replace [1] by a bare:

assert returned_origins == swh_origins

And the implementation without this diff will fail [2]

[1] https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/tests/test_storage.py$4012

[2]

$ pytest -vv -x -s swh/storage/tests/test_in_memory.py -k test_origin_list
============================================================================================================= test session starts =============================================================================================================
platform linux -- Python 3.7.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /home/tony/.virtualenvs/swh/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tony/work/inria/repo/swh/swh-environment/swh-storage/.hypothesis/examples')
rootdir: /home/tony/work/inria/repo/swh/swh-environment/swh-storage, configfile: pytest.ini
plugins: xdist-1.32.0, aiohttp-0.3.0, celery-4.4.6, testinfra-5.2.1, asyncio-0.14.0, postgresql-2.4.0, forked-1.2.0, requests-mock-1.8.0, mock-3.1.1, dash-1.13.1, django-3.9.0, flask-1.0.0, hypothesis-5.19.0, kafka-0.4.0, swh.journal-0.4.0, swh.core-0.2.2
collected 164 items / 159 deselected / 5 selected

swh/storage/tests/test_in_memory.py::TestStorageGeneratedData::test_origin_list[1] <- swh/storage/tests/test_storage.py FAILED

================================================================================================================== FAILURES ===================================================================================================================
________________________________________________________________________________________________ TestStorageGeneratedData.test_origin_list[1] _________________________________________________________________________________________________

self = <swh.storage.tests.test_storage.TestStorageGeneratedData object at 0x7f60bb876c88>, swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7f60bb876eb8>
swh_origins = [Origin(url='http://xn--n28h.tld/path/with/anchor#id=42'), Origin(url='http://example.com/path/with/qargs?q=1&b'), Ori...tld/path/to/resource'), Origin(url='deb://example.com/'), Origin(url='http://example.com/path/with/anchor#id=42'), ...]
limit = 1

    @pytest.mark.parametrize("limit", [1, 7, 10, 100, 1000])
    def test_origin_list(self, swh_storage, swh_origins, limit):
        returned_origins = []

        page_token = None
        i = 0
        while True:
            actual_page = swh_storage.origin_list(page_token=page_token, limit=limit)
            assert len(actual_page.results) <= limit

            returned_origins.extend(actual_page.results)

            i += 1
            page_token = actual_page.next_page_token

            if page_token is None:
                assert i * limit >= len(swh_origins)
                break
            else:
                assert len(actual_page.results) == limit

>       assert returned_origins == swh_origins
E       AssertionError: assert [Origin(url='deb://example.com'),\n Origin(url='deb://example.com/'),\n Origin(url='deb://example.com/path/to/resource'),\n Origin(url='deb://example.com/path/with/anchor#id=42'),\n Origin(url='deb://example.com/path/with/qargs?q=1&b'),\n Origin(url='deb://example.com/stuff'),\n Origin(url='deb://some.long.host.name'),\n Origin(url='deb://some.long.host.name/'),\n Origin(url='deb://some.long.host.name/path/to/resource'),\n Origin(url='deb://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='deb://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='deb://some.long.host.name/stuff/'),\n Origin(url='deb://xn--n28h.tld/'),\n Origin(url='deb://xn--n28h.tld/path/to/resource'),\n Origin(url='deb://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='deb://xn--n28h.tld/stuff'),\n Origin(url='git://example.com'),\n Origin(url='git://example.com/path/to/resource'),\n Origin(url='git://example.com/path/with/anchor#id=42'),\n Origin(url='git://example.com/stuff'),\n Origin(url='git://example.com/stuff/'),\n Origin(url='git://some.long.host.name/'),\n Origin(url='git://some.long.host.name/path/to/resource'),\n Origin(url='git://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='git://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='git://some.long.host.name/stuff'),\n Origin(url='git://some.long.host.name/stuff/'),\n Origin(url='git://xn--n28h.tld'),\n Origin(url='git://xn--n28h.tld/'),\n Origin(url='git://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='git://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='git://xn--n28h.tld/stuff'),\n Origin(url='git://xn--n28h.tld/stuff/'),\n Origin(url='http://example.com'),\n Origin(url='http://example.com/path/to/resource'),\n Origin(url='http://example.com/path/with/anchor#id=42'),\n Origin(url='http://example.com/path/with/qargs?q=1&b'),\n Origin(url='http://example.com/stuff'),\n Origin(url='http://example.com/stuff/'),\n Origin(url='http://some.long.host.name/'),\n Origin(url='http://some.long.host.name/path/to/resource'),\n Origin(url='http://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='http://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='http://some.long.host.name/stuff'),\n Origin(url='http://some.long.host.name/stuff/'),\n Origin(url='http://xn--n28h.tld'),\n Origin(url='http://xn--n28h.tld/'),\n Origin(url='http://xn--n28h.tld/path/to/resource'),\n Origin(url='http://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='http://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='http://xn--n28h.tld/stuff'),\n Origin(url='http://xn--n28h.tld/stuff/'),\n Origin(url='https://example.com'),\n Origin(url='https://example.com/path/to/resource'),\n Origin(url='https://example.com/path/with/anchor#id=42'),\n Origin(url='https://example.com/path/with/qargs?q=1&b'),\n Origin(url='https://example.com/stuff'),\n Origin(url='https://example.com/stuff/'),\n Origin(url='https://some.long.host.name/path/to/resource'),\n Origin(url='https://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='https://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='https://some.long.host.name/stuff'),\n Origin(url='https://some.long.host.name/stuff/'),\n Origin(url='https://xn--n28h.tld'),\n Origin(url='https://xn--n28h.tld/'),\n Origin(url='https://xn--n28h.tld/path/to/resource'),\n Origin(url='https://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='https://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='https://xn--n28h.tld/stuff'),\n Origin(url='mock://example.com'),\n Origin(url='mock://example.com/'),\n Origin(url='mock://example.com/path/to/resource'),\n Origin(url='mock://example.com/path/with/qargs?q=1&b'),\n Origin(url='mock://example.com/stuff/'),\n Origin(url='mock://some.long.host.name'),\n Origin(url='mock://some.long.host.name/'),\n Origin(url='mock://some.long.host.name/path/to/resource'),\n Origin(url='mock://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='mock://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='mock://some.long.host.name/stuff'),\n Origin(url='mock://xn--n28h.tld'),\n Origin(url='mock://xn--n28h.tld/path/to/resource'),\n Origin(url='mock://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='mock://xn--n28h.tld/stuff'),\n Origin(url='svn://example.com'),\n Origin(url='svn://example.com/'),\n Origin(url='svn://example.com/path/to/resource'),\n Origin(url='svn://example.com/path/with/anchor#id=42'),\n Origin(url='svn://example.com/path/with/qargs?q=1&b'),\n Origin(url='svn://example.com/stuff/'),\n Origin(url='svn://some.long.host.name/'),\n Origin(url='svn://some.long.host.name/path/to/resource'),\n Origin(url='svn://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='svn://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='svn://some.long.host.name/stuff'),\n Origin(url='svn://xn--n28h.tld/'),\n Origin(url='svn://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='svn://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='svn://xn--n28h.tld/stuff'),\n Origin(url='svn://xn--n28h.tld/stuff/')] == [Origin(url='http://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='http://example.com/path/with/qargs?q=1&b'),\n Origin(url='https://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='deb://xn--n28h.tld/path/to/resource'),\n Origin(url='deb://example.com/'),\n Origin(url='http://example.com/path/with/anchor#id=42'),\n Origin(url='https://some.long.host.name/stuff/'),\n Origin(url='deb://example.com'),\n Origin(url='http://some.long.host.name/stuff'),\n Origin(url='deb://some.long.host.name/path/to/resource'),\n Origin(url='svn://example.com/path/to/resource'),\n Origin(url='mock://example.com/'),\n Origin(url='git://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='deb://some.long.host.name'),\n Origin(url='https://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='svn://example.com/path/with/qargs?q=1&b'),\n Origin(url='git://example.com/stuff/'),\n Origin(url='deb://xn--n28h.tld/'),\n Origin(url='http://xn--n28h.tld/stuff/'),\n Origin(url='http://some.long.host.name/path/to/resource'),\n Origin(url='deb://example.com/path/to/resource'),\n Origin(url='mock://example.com/stuff/'),\n Origin(url='svn://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='deb://example.com/path/with/anchor#id=42'),\n Origin(url='svn://xn--n28h.tld/'),\n Origin(url='svn://example.com'),\n Origin(url='https://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='mock://xn--n28h.tld/path/to/resource'),\n Origin(url='git://some.long.host.name/'),\n Origin(url='mock://some.long.host.name/'),\n Origin(url='http://xn--n28h.tld'),\n Origin(url='svn://some.long.host.name/'),\n Origin(url='svn://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='git://some.long.host.name/path/to/resource'),\n Origin(url='https://example.com/path/with/anchor#id=42'),\n Origin(url='git://example.com/stuff'),\n Origin(url='https://xn--n28h.tld/'),\n Origin(url='svn://example.com/path/with/anchor#id=42'),\n Origin(url='mock://some.long.host.name/stuff'),\n Origin(url='http://example.com'),\n Origin(url='deb://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='deb://some.long.host.name/'),\n Origin(url='http://example.com/stuff'),\n Origin(url='svn://some.long.host.name/stuff'),\n Origin(url='svn://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='http://xn--n28h.tld/stuff'),\n Origin(url='git://xn--n28h.tld/'),\n Origin(url='git://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='http://some.long.host.name/stuff/'),\n Origin(url='https://xn--n28h.tld'),\n Origin(url='git://some.long.host.name/stuff/'),\n Origin(url='http://xn--n28h.tld/'),\n Origin(url='svn://xn--n28h.tld/stuff'),\n Origin(url='http://example.com/path/to/resource'),\n Origin(url='deb://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='svn://some.long.host.name/path/to/resource'),\n Origin(url='deb://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='svn://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='svn://xn--n28h.tld/stuff/'),\n Origin(url='mock://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='mock://some.long.host.name/path/to/resource'),\n Origin(url='deb://xn--n28h.tld/stuff'),\n Origin(url='https://some.long.host.name/path/to/resource'),\n Origin(url='mock://example.com'),\n Origin(url='mock://example.com/path/to/resource'),\n Origin(url='git://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='https://example.com/stuff/'),\n Origin(url='https://example.com/path/to/resource'),\n Origin(url='mock://xn--n28h.tld/stuff'),\n Origin(url='deb://example.com/stuff'),\n Origin(url='https://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='mock://xn--n28h.tld/path/with/anchor#id=42'),\n Origin(url='http://example.com/stuff/'),\n Origin(url='git://some.long.host.name/stuff'),\n Origin(url='http://xn--n28h.tld/path/with/qargs?q=1&b'),\n Origin(url='svn://example.com/stuff/'),\n Origin(url='git://example.com'),\n Origin(url='https://xn--n28h.tld/path/to/resource'),\n Origin(url='mock://xn--n28h.tld'),\n Origin(url='deb://some.long.host.name/stuff/'),\n Origin(url='mock://some.long.host.name'),\n Origin(url='https://example.com'),\n Origin(url='svn://example.com/'),\n Origin(url='deb://example.com/path/with/qargs?q=1&b'),\n Origin(url='https://example.com/stuff'),\n Origin(url='git://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='git://example.com/path/with/anchor#id=42'),\n Origin(url='git://xn--n28h.tld/stuff/'),\n Origin(url='mock://example.com/path/with/qargs?q=1&b'),\n Origin(url='https://some.long.host.name/stuff'),\n Origin(url='https://example.com/path/with/qargs?q=1&b'),\n Origin(url='git://xn--n28h.tld'),\n Origin(url='git://xn--n28h.tld/stuff'),\n Origin(url='http://some.long.host.name/path/with/qargs?q=1&b'),\n Origin(url='http://xn--n28h.tld/path/to/resource'),\n Origin(url='http://some.long.host.name/'),\n Origin(url='mock://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='http://some.long.host.name/path/with/anchor#id=42'),\n Origin(url='git://example.com/path/to/resource'),\n Origin(url='https://xn--n28h.tld/stuff')]
E         At index 0 diff: Origin(url='deb://example.com') != Origin(url='http://xn--n28h.tld/path/with/anchor#id=42')
E         Full diff:
E           [
E         +  Origin(url='deb://example.com'),
E         +  Origin(url='deb://example.com/'),
E         +  Origin(url='deb://example.com/path/to/resource'),
E         +  Origin(url='deb://example.com/path/with/anchor#id=42'),
E         +  Origin(url='deb://example.com/path/with/qargs?q=1&b'),
E         +  Origin(url='deb://example.com/stuff'),
E         +  Origin(url='deb://some.long.host.name'),
E         +  Origin(url='deb://some.long.host.name/'),
E         +  Origin(url='deb://some.long.host.name/path/to/resource'),
E         +  Origin(url='deb://some.long.host.name/path/with/anchor#id=42'),
E         +  Origin(url='deb://some.long.host.name/path/with/qargs?q=1&b'),
E         +  Origin(url='deb://some.long.host.name/stuff/'),
E         +  Origin(url='deb://xn--n28h.tld/'),
E         +  Origin(url='deb://xn--n28h.tld/path/to/resource'),
E         +  Origin(url='deb://xn--n28h.tld/path/with/qargs?q=1&b'),
E         +  Origin(url='deb://xn--n28h.tld/stuff'),
E         +  Origin(url='git://example.com'),
E         +  Origin(url='git://example.com/path/to/resource'),
E         +  Origin(url='git://example.com/path/with/anchor#id=42'),
E         +  Origin(url='git://example.com/stuff'),
E         +  Origin(url='git://example.com/stuff/'),
E         +  Origin(url='git://some.long.host.name/'),
E         +  Origin(url='git://some.long.host.name/path/to/resource'),
E         +  Origin(url='git://some.long.host.name/path/with/anchor#id=42'),
E         +  Origin(url='git://some.long.host.name/path/with/qargs?q=1&b'),
E         +  Origin(url='git://some.long.host.name/stuff'),
E         +  Origin(url='git://some.long.host.name/stuff/'),
E         +  Origin(url='git://xn--n28h.tld'),
E         +  Origin(url='git://xn--n28h.tld/'),
E         +  Origin(url='git://xn--n28h.tld/path/with/anchor#id=42'),
E         +  Origin(url='git://xn--n28h.tld/path/with/qargs?q=1&b'),
E         +  Origin(url='git://xn--n28h.tld/stuff'),
E         +  Origin(url='git://xn--n28h.tld/stuff/'),
E         +  Origin(url='http://example.com'),
E         +  Origin(url='http://example.com/path/to/resource'),
E         +  Origin(url='http://example.com/path/with/anchor#id=42'),
E         +  Origin(url='http://example.com/path/with/qargs?q=1&b'),
E         +  Origin(url='http://example.com/stuff'),
E         +  Origin(url='http://example.com/stuff/'),
E         +  Origin(url='http://some.long.host.name/'),
E         +  Origin(url='http://some.long.host.name/path/to/resource'),
E         +  Origin(url='http://some.long.host.name/path/with/anchor#id=42'),
E         +  Origin(url='http://some.long.host.name/path/with/qargs?q=1&b'),
E         +  Origin(url='http://some.long.host.name/stuff'),
E         +  Origin(url='http://some.long.host.name/stuff/'),
E         +  Origin(url='http://xn--n28h.tld'),
E         +  Origin(url='http://xn--n28h.tld/'),
E         +  Origin(url='http://xn--n28h.tld/path/to/resource'),
E            Origin(url='http://xn--n28h.tld/path/with/anchor#id=42'),
E         +  Origin(url='http://xn--n28h.tld/path/with/qargs?q=1&b'),
E         +  Origin(url='http://xn--n28h.tld/stuff'),
E         +  Origin(url='http://xn--n28h.tld/stuff/'),
E         +  Origin(url='https://example.com'),
E         +  Origin(url='https://example.com/path/to/resource'),
E         +  Origin(url='https://example.com/path/with/anchor#id=42'),
E         -  Origin(url='http://example.com/path/with/qargs?q=1&b'),
E         +  Origin(url='https://example.com/path/with/qargs?q=1&b'),
E         ?                  +
E         +  Origin(url='https://example.com/stuff'),
E         +  Origin(url='https://example.com/stuff/'),
E         +  Origin(url='https://some.long.host.name/path/to/resource'),
E            Origin(url='https://some.long.host.name/path/with/anchor#id=42'),
E         +  Origin(url='https://some.long.host.name/path/with/qargs?q=1&b'),
E         +  Origin(url='https://some.long.host.name/stuff'),
E         -  Origin(url='deb://xn--n28h.tld/path/to/resource'),
E         -  Origin(url='deb://example.com/'),
E         -  Origin(url='http://example.com/path/with/anchor#id=42'),
E            Origin(url='https://some.long.host.name/stuff/'),
E         +  Origin(url='https://xn--n28h.tld'),
E         +  Origin(url='https://xn--n28h.tld/'),
E         +  Origin(url='https://xn--n28h.tld/path/to/resource'),
E         +  Origin(url='https://xn--n28h.tld/path/with/anchor#id=42'),
E         +  Origin(url='https://xn--n28h.tld/path/with/qargs?q=1&b'),
E         +  Origin(url='https://xn--n28h.tld/stuff'),
E         -  Origin(url='deb://example.com'),
E         ?              ^^^
E         +  Origin(url='mock://example.com'),
E         ?              ^^^^
E         +  Origin(url='mock://example.com/'),
E         +  Origin(url='mock://example.com/path/to/resource'),
E         +  Origin(url='mock://example.com/path/with/qargs?q=1&b'),
E         +  Origin(url='mock://example.com/stuff/'),
E         +  Origin(url='mock://some.long.host.name'),
E         -  Origin(url='http://some.long.host.name/stuff'),
E         ?              ^^^^                       -----
E         +  Origin(url='mock://some.long.host.name/'),
E         ?              ^^^^
E         -  Origin(url='deb://some.long.host.name/path/to/resource'),
E         ?              ^^^
E         +  Origin(url='mock://some.long.host.name/path/to/resource'),
E         ?              ^^^^
E         +  Origin(url='mock://some.long.host.name/path/with/anchor#id=42'),
E         +  Origin(url='mock://some.long.host.name/path/with/qargs?q=1&b'),
E         +  Origin(url='mock://some.long.host.name/stuff'),
E         +  Origin(url='mock://xn--n28h.tld'),
E         +  Origin(url='mock://xn--n28h.tld/path/to/resource'),
E         +  Origin(url='mock://xn--n28h.tld/path/with/anchor#id=42'),
E         +  Origin(url='mock://xn--n28h.tld/stuff'),
E         +  Origin(url='svn://example.com'),
E         +  Origin(url='svn://example.com/'),
E            Origin(url='svn://example.com/path/to/resource'),
E         -  Origin(url='mock://example.com/'),
E         -  Origin(url='git://xn--n28h.tld/path/with/anchor#id=42'),
E         ?              ^^^    ^^^^^^^ ^^^
E         +  Origin(url='svn://example.com/path/with/anchor#id=42'),
E         ?              ^^^   + ^^^^^ ^^^
E         -  Origin(url='deb://some.long.host.name'),
E         -  Origin(url='https://xn--n28h.tld/path/with/qargs?q=1&b'),
E            Origin(url='svn://example.com/path/with/qargs?q=1&b'),
E         -  Origin(url='git://example.com/stuff/'),
E         ?              ^^^
E         +  Origin(url='svn://example.com/stuff/'),
E         ?              ^^^
E         +  Origin(url='svn://some.long.host.name/'),
E         -  Origin(url='deb://xn--n28h.tld/'),
E         -  Origin(url='http://xn--n28h.tld/stuff/'),
E         -  Origin(url='http://some.long.host.name/path/to/resource'),
E         ?              ^^^^
E         +  Origin(url='svn://some.long.host.name/path/to/resource'),
E         ?              ^^^
E         +  Origin(url='svn://some.long.host.name/path/with/anchor#id=42'),
E         -  Origin(url='deb://example.com/path/to/resource'),
E         -  Origin(url='mock://example.com/stuff/'),
E            Origin(url='svn://some.long.host.name/path/with/qargs?q=1&b'),
E         -  Origin(url='deb://example.com/path/with/anchor#id=42'),
E         +  Origin(url='svn://some.long.host.name/stuff'),
E            Origin(url='svn://xn--n28h.tld/'),
E         -  Origin(url='svn://example.com'),
E         -  Origin(url='https://some.long.host.name/path/with/qargs?q=1&b'),
E         -  Origin(url='mock://xn--n28h.tld/path/to/resource'),
E         -  Origin(url='git://some.long.host.name/'),
E         -  Origin(url='mock://some.long.host.name/'),
E         -  Origin(url='http://xn--n28h.tld'),
E         -  Origin(url='svn://some.long.host.name/'),
E         -  Origin(url='svn://some.long.host.name/path/with/anchor#id=42'),
E         -  Origin(url='git://some.long.host.name/path/to/resource'),
E         -  Origin(url='https://example.com/path/with/anchor#id=42'),
E         -  Origin(url='git://example.com/stuff'),
E         -  Origin(url='https://xn--n28h.tld/'),
E         -  Origin(url='svn://example.com/path/with/anchor#id=42'),
E         -  Origin(url='mock://some.long.host.name/stuff'),
E         -  Origin(url='http://example.com'),
E         -  Origin(url='deb://some.long.host.name/path/with/qargs?q=1&b'),
E         -  Origin(url='deb://some.long.host.name/'),
E         -  Origin(url='http://example.com/stuff'),
E         -  Origin(url='svn://some.long.host.name/stuff'),
E            Origin(url='svn://xn--n28h.tld/path/with/anchor#id=42'),
E         -  Origin(url='http://xn--n28h.tld/stuff'),
E         -  Origin(url='git://xn--n28h.tld/'),
E         -  Origin(url='git://xn--n28h.tld/path/with/qargs?q=1&b'),
E         ?              ^^^
E         +  Origin(url='svn://xn--n28h.tld/path/with/qargs?q=1&b'),
E         ?              ^^^
E         -  Origin(url='http://some.long.host.name/stuff/'),
E         -  Origin(url='https://xn--n28h.tld'),
E         -  Origin(url='git://some.long.host.name/stuff/'),
E         -  Origin(url='http://xn--n28h.tld/'),
E            Origin(url='svn://xn--n28h.tld/stuff'),
E         -  Origin(url='http://example.com/path/to/resource'),
E         -  Origin(url='deb://xn--n28h.tld/path/with/qargs?q=1&b'),
E         -  Origin(url='svn://some.long.host.name/path/to/resource'),
E         -  Origin(url='deb://some.long.host.name/path/with/anchor#id=42'),
E         -  Origin(url='svn://xn--n28h.tld/path/with/qargs?q=1&b'),
E            Origin(url='svn://xn--n28h.tld/stuff/'),
E         -  Origin(url='mock://some.long.host.name/path/with/qargs?q=1&b'),
E         -  Origin(url='mock://some.long.host.name/path/to/resource'),
E         -  Origin(url='deb://xn--n28h.tld/stuff'),
E         -  Origin(url='https://some.long.host.name/path/to/resource'),
E         -  Origin(url='mock://example.com'),
E         -  Origin(url='mock://example.com/path/to/resource'),
E         -  Origin(url='git://some.long.host.name/path/with/anchor#id=42'),
E         -  Origin(url='https://example.com/stuff/'),
E         -  Origin(url='https://example.com/path/to/resource'),
E         -  Origin(url='mock://xn--n28h.tld/stuff'),
E         -  Origin(url='deb://example.com/stuff'),
E         -  Origin(url='https://xn--n28h.tld/path/with/anchor#id=42'),
E         -  Origin(url='mock://xn--n28h.tld/path/with/anchor#id=42'),
E         -  Origin(url='http://example.com/stuff/'),
E         -  Origin(url='git://some.long.host.name/stuff'),
E         -  Origin(url='http://xn--n28h.tld/path/with/qargs?q=1&b'),
E         -  Origin(url='svn://example.com/stuff/'),
E         -  Origin(url='git://example.com'),
E         -  Origin(url='https://xn--n28h.tld/path/to/resource'),
E         -  Origin(url='mock://xn--n28h.tld'),
E         -  Origin(url='deb://some.long.host.name/stuff/'),
E         -  Origin(url='mock://some.long.host.name'),
E         -  Origin(url='https://example.com'),
E         -  Origin(url='svn://example.com/'),
E         -  Origin(url='deb://example.com/path/with/qargs?q=1&b'),
E         -  Origin(url='https://example.com/stuff'),
E         -  Origin(url='git://some.long.host.name/path/with/qargs?q=1&b'),
E         -  Origin(url='git://example.com/path/with/anchor#id=42'),
E         -  Origin(url='git://xn--n28h.tld/stuff/'),
E         -  Origin(url='mock://example.com/path/with/qargs?q=1&b'),
E         -  Origin(url='https://some.long.host.name/stuff'),
E         -  Origin(url='https://example.com/path/with/qargs?q=1&b'),
E         -  Origin(url='git://xn--n28h.tld'),
E         -  Origin(url='git://xn--n28h.tld/stuff'),
E         -  Origin(url='http://some.long.host.name/path/with/qargs?q=1&b'),
E         -  Origin(url='http://xn--n28h.tld/path/to/resource'),
E         -  Origin(url='http://some.long.host.name/'),
E         -  Origin(url='mock://some.long.host.name/path/with/anchor#id=42'),
E         -  Origin(url='http://some.long.host.name/path/with/anchor#id=42'),
E         -  Origin(url='git://example.com/path/to/resource'),
E         -  Origin(url='https://xn--n28h.tld/stuff'),
E           ]

swh/storage/tests/test_storage.py:3992: AssertionError
=========================================================================================================== short test summary info ===========================================================================================================
FAILED swh/storage/tests/test_in_memory.py::TestStorageGeneratedData::test_origin_list[1] - AssertionError: assert [Origin(url='deb://example.com'),\n Origin(url='deb://example.com/'),\n Origin(url='deb://example.com/path/to/resource'),...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================================================== 1 failed, 159 deselected in 0.79s ======================================================================================================

Adapt test_origin_list so the issue is more apparent

Build has FAILED

Patch application report for D3683 (id=12970)

Could not rebase; Attempt merge onto 92f1183de0...

Updating 92f1183d..20e3e15d
Fast-forward
 swh/storage/algos/origin.py            | 32 ++++---------
 swh/storage/cassandra/storage.py       | 12 ++++-
 swh/storage/in_memory.py               | 45 ++++++++++--------
 swh/storage/interface.py               | 30 +++++-------
 swh/storage/storage.py                 | 14 +++++-
 swh/storage/tests/algos/test_origin.py | 84 +++++-----------------------------
 swh/storage/tests/test_cassandra.py    |  8 ----
 swh/storage/tests/test_storage.py      | 65 ++++++++++----------------
 8 files changed, 104 insertions(+), 186 deletions(-)

Changes applied before test

commit 20e3e15d72a2508c3535dc21b2c94470e49a8ab9
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 11:15:56 2020 +0200

    storage.in_memory: Fix origin_list implementation
    
    This kept on creating unrelated errors on other modules.
    
    Related to T645

commit 77757a0dccb429bd84154df11ad58c8442d75b79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

commit 87c5ba223508b212ed6939496a6380a771b05e43
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 09:07:12 2020 +0200

    storage*: Do not allow unknown visit status in origin_visit*_get_latest
    
    That makes some storage (pg-storage) fail without that filtering.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/650/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/650/console

Harbormaster failed remote builds in B14208: Diff 12970!Aug 1 2020, 2:54 PM

then that's not a bug. The order of results of origin_list is not guaranteed to be the order of insertion. Providing this property is equivalent to having numeric origin ids.

then that's not a bug. The order of results of origin_list is not guaranteed to be the order of insertion.

yeah, you are right. I'm slowly realizing this.
I gather the cassandra storage agrees with you and thus fails
(I did not check but believe that's the source of the current failure... yeah it is).

Although now, I don't really know what i can check in the scheduler cli D3682.
Only that the number of origins is correct then, i guess.

Providing this property is equivalent to having numeric origin ids.

yeah, i probably made a shortcut there ;)

ardumont abandoned this revision.Aug 1 2020, 6:19 PM

Revision Contents
Changeset List

Path

Size

Coverage (All)

Coverage (Touched)

swh/

storage/

in_memory.py

21 lines

97%

Diff 12969

View Options

storage.in_memory: Fix origin_list implementationAbandonedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D3683 (id=12966)

Changes applied before test

Patch application report for D3683 (id=12967)

Changes applied before test

Patch application report for D3683 (id=12969)

Changes applied before test

Patch application report for D3683 (id=12970)

Changes applied before test

Revision ContentsChangeset List

Diff 12969

swh/storage/in_memory.py

storage.in_memory: Fix origin_list implementation
AbandonedPublic
Actions

Revision Contents
Changeset List