Page MenuHomeSoftware Heritage
Differential D958

storage: Add origin_get_range method to list all archived origins
ClosedPublic
Actions

Authored by anlambert on Jan 16 2019, 11:01 AM.

Details

Reviewers
olasd
vlorentz
Group Reviewers
Reviewers
Commits
rDSTOC76d7838c01e5: storage: Add origin_get_range method to list all archived origins
R65:76d7838c01e5: storage: Add origin_get_range method to list all archived origins
rDSTO76d7838c01e5: storage: Add origin_get_range method to list all archived origins
Summary

Add a new method to the storage interface in order to list all archived
origins in a paginated way.

Related T1399

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Jan 16 2019, 11:01 AM
anlambert added a reviewer: Reviewers.Jan 16 2019, 11:02 AM
swh-public-ci added a subscriber: swh-public-ci.Jan 16 2019, 11:05 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/156/ for more details.

Harbormaster completed remote builds in B3555: Diff 3049.Jan 16 2019, 11:05 AM
anlambert added a child revision: D959: Web API: Add endpoint to list all archived software origins.Jan 16 2019, 12:04 PM
vlorentz accepted this revision as: vlorentz.Jan 16 2019, 6:13 PM
vlorentz added a subscriber: vlorentz.

You should rename origins_get to origin_get_range, for consistance with content_get_range.

And it could use more testing of edge cases for origin_from.

Other than that, lgtm

This revision is now accepted and ready to land.Jan 16 2019, 6:13 PM
anlambert added a comment.Jan 17 2019, 10:40 AM

Ack, will update accordingly before landing that diff.

anlambert updated this revision to Diff 3069.Jan 17 2019, 11:44 AM

Update:

  • rename new method to origin_get_range
  • improve unit test
anlambert retitled this revision from storage: Add origins_get method to list all archived origins to storage: Add origin_get_range method to list all archived origins.Jan 17 2019, 11:44 AM
swh-public-ci added a comment.Jan 17 2019, 11:47 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/159/ for more details.

Harbormaster completed remote builds in B3578: Diff 3069.Jan 17 2019, 11:47 AM
anlambert updated this revision to Diff 3072.Jan 17 2019, 11:51 AM

Update: simplify test in in_memory.storage.origin_get_range

swh-public-ci added a comment.Jan 17 2019, 11:54 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/160/ for more details.

Harbormaster completed remote builds in B3581: Diff 3072.Jan 17 2019, 11:54 AM
Closed by commit rDSTO76d7838c01e5: storage: Add origin_get_range method to list all archived origins (authored by anlambert). · Explain WhyJan 17 2019, 12:04 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
storage/
api/
4 lines
6 lines
20 lines
22 lines
23 lines
tests/
34 lines
39 lines

Diff 3076

View Options

swh/storage/api/client.py

# Copyright (C) 2019-2021 The Software Heritage developers # Copyright (C) 2019-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from itertools import permutations
from hypothesis import given, settings, strategies from hypothesis import given, settings, strategies
import pytest import pytest
from swh.core.api.classes import stream_results from swh.core.api.classes import stream_results
NOW = datetime.now(tz=timezone.utc).isoformat()
NOW_MINUS_5_HOURS = (datetime.now(tz=timezone.utc) - timedelta(hours=5)).isoformat()
NOW_PLUS_5_HOURS = (datetime.now(tz=timezone.utc) + timedelta(hours=5)).isoformat()
VISIT_STATUSES = [
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_1",
"last_eventful_visit_date": NOW,
},
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_1",
"last_eventful_visit_date": NOW_MINUS_5_HOURS,
},
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_2",
"last_eventful_visit_date": NOW_PLUS_5_HOURS,
},
]
class CommonSearchTest: class CommonSearchTest:
def test_origin_url_unique_word_prefix(self): def test_origin_url_unique_word_prefix(self):
origin_foobar_baz = {"url": "http://foobar.baz"} origin_foobar_baz = {"url": "http://foobar.baz"}
origin_barbaz_qux = {"url": "http://barbaz.qux"} origin_barbaz_qux = {"url": "http://barbaz.qux"}
origin_qux_quux = {"url": "http://qux.quux"} origin_qux_quux = {"url": "http://qux.quux"}
origins = [origin_foobar_baz, origin_barbaz_qux, origin_qux_quux] origins = [origin_foobar_baz, origin_barbaz_qux, origin_qux_quux]
▲ Show 20 Lines • Show All 231 Lines • ▼ Show 20 Lines def test_origin_last_visit_date_update_search(self):
_check_min_last_visit_date(now_minus_5_hours) # Works for < _check_min_last_visit_date(now_minus_5_hours) # Works for <
with pytest.raises(AssertionError): with pytest.raises(AssertionError):
_check_min_last_visit_date(now_plus_5_hours) # Fails for > _check_min_last_visit_date(now_plus_5_hours) # Fails for >
_update_last_visit_date(now_plus_5_hours) _update_last_visit_date(now_plus_5_hours)
_check_min_last_visit_date(now_plus_5_hours) # Works for = _check_min_last_visit_date(now_plus_5_hours) # Works for =
_check_min_last_visit_date(now) # Works for < _check_min_last_visit_date(now) # Works for <
vsellierUnsubmitted
Not Done
Inline Actions

A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order gives the expected result
Please have a look at the equivalent on the scheduler:
https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/tests/test_journal_client.py$418

vsellier: A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order…
def test_journal_client_origin_visit_status_permutation(self):
for visit_statuses in permutations(VISIT_STATUSES, len(VISIT_STATUSES)):
self.search.origin_update(visit_statuses)
self.search.flush()
origin_url = "http://foobar.baz"
actual_page = self.search.origin_search(
url_pattern=origin_url, min_last_eventful_visit_date=NOW_PLUS_5_HOURS,
)
assert actual_page.next_page_token is None
results = [r["url"] for r in actual_page.results]
expected_results = [origin_url]
assert sorted(results) == sorted(expected_results)
def test_origin_last_eventful_visit_date_update_search(self):
origin_url = "http://foobar.baz"
self.search.origin_update([{"url": origin_url}])
self.search.flush()
def _update_last_eventful_visit_date(snapshot_id, last_eventful_visit_date):
self.search.origin_update(
[
{
"url": origin_url,
"snapshot_id": snapshot_id,
"last_eventful_visit_date": last_eventful_visit_date,
}
]
vsellierUnsubmitted
Not Done
Inline Actions

Is there a reason to use permutation instead of a parameterized test?
With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

@vlorentz WDYT ?

vsellier: Is there a reason to use permutation instead of a parameterized test? With this implementation…
vlorentzUnsubmitted
Not Done
Inline Actions

I agree

vlorentz: I agree
KShivenduAuthorUnsubmitted
Done
Inline Actions

I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work-with-unittest-class-based-tests which says.

unittest.TestCase methods cannot directly receive fixture function arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.

Is this stackoverflow answer wrong ?

KShivendu: I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work…
KShivenduAuthorUnsubmitted
Done
Inline Actions

With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

Ahh, I see. My bad.

KShivendu: >> With this implementation the index is not reset between 2 iterations so after the first loop…
vlorentzUnsubmitted
Not Done
Inline Actions

This is not a unittest.TestCase-based test; it's just a regular class.

vlorentz: This is not a `unittest.TestCase`-based test; it's just a regular class.
)
self.search.flush()
def _check_min_last_eventful_visit_date(min_last_eventful_visit_date):
actual_page = self.search.origin_search(
url_pattern=origin_url,
min_last_eventful_visit_date=min_last_eventful_visit_date,
)
assert actual_page.next_page_token is None
results = [r["url"] for r in actual_page.results]
expected_results = [origin_url]
assert sorted(results) == sorted(expected_results)
now = datetime.now(tz=timezone.utc).isoformat()
now_minus_5_hours = (
datetime.now(tz=timezone.utc) - timedelta(hours=5)
).isoformat()
now_plus_5_hours = (
datetime.now(tz=timezone.utc) + timedelta(hours=5)
).isoformat()
snapshot_1 = "SNAPSHOT_1"
snapshot_2 = "SNAPSHOT_2"
_update_last_eventful_visit_date(snapshot_1, now)
_check_min_last_eventful_visit_date(now) # Works for =
_check_min_last_eventful_visit_date(now_minus_5_hours) # Works for <
with pytest.raises(AssertionError):
_check_min_last_eventful_visit_date(now_plus_5_hours) # Fails for >
_update_last_eventful_visit_date(
snapshot_1, now_plus_5_hours
) # Revisit(not eventful) same origin
_check_min_last_eventful_visit_date(
now
) # Should remain the same because recent visit wasn't eventful
with pytest.raises(AssertionError):
_check_min_last_eventful_visit_date(now_plus_5_hours)
_update_last_eventful_visit_date(
snapshot_2, now_plus_5_hours
) # Revisit(eventful) same origin
_check_min_last_eventful_visit_date(now_plus_5_hours) # Works for =
_check_min_last_eventful_visit_date(now) # Works for <
def test_origin_update_with_no_visit_types(self): def test_origin_update_with_no_visit_types(self):
""" """
Update an origin with visit types first then with no visit types, Update an origin with visit types first then with no visit types,
check origin can still be searched with visit types afterwards. check origin can still be searched with visit types afterwards.
""" """
origin_url = "http://foobar.baz" origin_url = "http://foobar.baz"
self.search.origin_update([{"url": origin_url, "visit_types": ["git"]}]) self.search.origin_update([{"url": origin_url, "visit_types": ["git"]}])
self.search.flush() self.search.flush()
▲ Show 20 Lines • Show All 427 Lines • Show Last 20 Lines
View Options

swh/storage/api/server.py

# Copyright (C) 2019-2021 The Software Heritage developers # Copyright (C) 2019-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from itertools import permutations
from hypothesis import given, settings, strategies from hypothesis import given, settings, strategies
import pytest import pytest
from swh.core.api.classes import stream_results from swh.core.api.classes import stream_results
NOW = datetime.now(tz=timezone.utc).isoformat()
NOW_MINUS_5_HOURS = (datetime.now(tz=timezone.utc) - timedelta(hours=5)).isoformat()
NOW_PLUS_5_HOURS = (datetime.now(tz=timezone.utc) + timedelta(hours=5)).isoformat()
VISIT_STATUSES = [
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_1",
"last_eventful_visit_date": NOW,
},
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_1",
"last_eventful_visit_date": NOW_MINUS_5_HOURS,
},
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_2",
"last_eventful_visit_date": NOW_PLUS_5_HOURS,
},
]
class CommonSearchTest: class CommonSearchTest:
def test_origin_url_unique_word_prefix(self): def test_origin_url_unique_word_prefix(self):
origin_foobar_baz = {"url": "http://foobar.baz"} origin_foobar_baz = {"url": "http://foobar.baz"}
origin_barbaz_qux = {"url": "http://barbaz.qux"} origin_barbaz_qux = {"url": "http://barbaz.qux"}
origin_qux_quux = {"url": "http://qux.quux"} origin_qux_quux = {"url": "http://qux.quux"}
origins = [origin_foobar_baz, origin_barbaz_qux, origin_qux_quux] origins = [origin_foobar_baz, origin_barbaz_qux, origin_qux_quux]
▲ Show 20 Lines • Show All 231 Lines • ▼ Show 20 Lines def test_origin_last_visit_date_update_search(self):
_check_min_last_visit_date(now_minus_5_hours) # Works for < _check_min_last_visit_date(now_minus_5_hours) # Works for <
with pytest.raises(AssertionError): with pytest.raises(AssertionError):
_check_min_last_visit_date(now_plus_5_hours) # Fails for > _check_min_last_visit_date(now_plus_5_hours) # Fails for >
_update_last_visit_date(now_plus_5_hours) _update_last_visit_date(now_plus_5_hours)
_check_min_last_visit_date(now_plus_5_hours) # Works for = _check_min_last_visit_date(now_plus_5_hours) # Works for =
_check_min_last_visit_date(now) # Works for < _check_min_last_visit_date(now) # Works for <
vsellierUnsubmitted
Not Done
Inline Actions

A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order gives the expected result
Please have a look at the equivalent on the scheduler:
https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/tests/test_journal_client.py$418

vsellier: A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order…
def test_journal_client_origin_visit_status_permutation(self):
for visit_statuses in permutations(VISIT_STATUSES, len(VISIT_STATUSES)):
self.search.origin_update(visit_statuses)
self.search.flush()
origin_url = "http://foobar.baz"
actual_page = self.search.origin_search(
url_pattern=origin_url, min_last_eventful_visit_date=NOW_PLUS_5_HOURS,
)
assert actual_page.next_page_token is None
results = [r["url"] for r in actual_page.results]
expected_results = [origin_url]
assert sorted(results) == sorted(expected_results)
def test_origin_last_eventful_visit_date_update_search(self):
origin_url = "http://foobar.baz"
self.search.origin_update([{"url": origin_url}])
self.search.flush()
def _update_last_eventful_visit_date(snapshot_id, last_eventful_visit_date):
self.search.origin_update(
[
{
"url": origin_url,
"snapshot_id": snapshot_id,
"last_eventful_visit_date": last_eventful_visit_date,
}
]
vsellierUnsubmitted
Not Done
Inline Actions

Is there a reason to use permutation instead of a parameterized test?
With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

@vlorentz WDYT ?

vsellier: Is there a reason to use permutation instead of a parameterized test? With this implementation…
vlorentzUnsubmitted
Not Done
Inline Actions

I agree

vlorentz: I agree
KShivenduAuthorUnsubmitted
Done
Inline Actions

I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work-with-unittest-class-based-tests which says.

unittest.TestCase methods cannot directly receive fixture function arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.

Is this stackoverflow answer wrong ?

KShivendu: I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work…
KShivenduAuthorUnsubmitted
Done
Inline Actions

With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

Ahh, I see. My bad.

KShivendu: >> With this implementation the index is not reset between 2 iterations so after the first loop…
vlorentzUnsubmitted
Not Done
Inline Actions

This is not a unittest.TestCase-based test; it's just a regular class.

vlorentz: This is not a `unittest.TestCase`-based test; it's just a regular class.
)
self.search.flush()
def _check_min_last_eventful_visit_date(min_last_eventful_visit_date):
actual_page = self.search.origin_search(
url_pattern=origin_url,
min_last_eventful_visit_date=min_last_eventful_visit_date,
)
assert actual_page.next_page_token is None
results = [r["url"] for r in actual_page.results]
expected_results = [origin_url]
assert sorted(results) == sorted(expected_results)
now = datetime.now(tz=timezone.utc).isoformat()
now_minus_5_hours = (
datetime.now(tz=timezone.utc) - timedelta(hours=5)
).isoformat()
now_plus_5_hours = (
datetime.now(tz=timezone.utc) + timedelta(hours=5)
).isoformat()
snapshot_1 = "SNAPSHOT_1"
snapshot_2 = "SNAPSHOT_2"
_update_last_eventful_visit_date(snapshot_1, now)
_check_min_last_eventful_visit_date(now) # Works for =
_check_min_last_eventful_visit_date(now_minus_5_hours) # Works for <
with pytest.raises(AssertionError):
_check_min_last_eventful_visit_date(now_plus_5_hours) # Fails for >
_update_last_eventful_visit_date(
snapshot_1, now_plus_5_hours
) # Revisit(not eventful) same origin
_check_min_last_eventful_visit_date(
now
) # Should remain the same because recent visit wasn't eventful
with pytest.raises(AssertionError):
_check_min_last_eventful_visit_date(now_plus_5_hours)
_update_last_eventful_visit_date(
snapshot_2, now_plus_5_hours
) # Revisit(eventful) same origin
_check_min_last_eventful_visit_date(now_plus_5_hours) # Works for =
_check_min_last_eventful_visit_date(now) # Works for <
def test_origin_update_with_no_visit_types(self): def test_origin_update_with_no_visit_types(self):
""" """
Update an origin with visit types first then with no visit types, Update an origin with visit types first then with no visit types,
check origin can still be searched with visit types afterwards. check origin can still be searched with visit types afterwards.
""" """
origin_url = "http://foobar.baz" origin_url = "http://foobar.baz"
self.search.origin_update([{"url": origin_url, "visit_types": ["git"]}]) self.search.origin_update([{"url": origin_url, "visit_types": ["git"]}])
self.search.flush() self.search.flush()
▲ Show 20 Lines • Show All 427 Lines • Show Last 20 Lines
View Options

swh/storage/db.py

# Copyright (C) 2019-2021 The Software Heritage developers # Copyright (C) 2019-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from itertools import permutations
from hypothesis import given, settings, strategies from hypothesis import given, settings, strategies
import pytest import pytest
from swh.core.api.classes import stream_results from swh.core.api.classes import stream_results
NOW = datetime.now(tz=timezone.utc).isoformat()
NOW_MINUS_5_HOURS = (datetime.now(tz=timezone.utc) - timedelta(hours=5)).isoformat()
NOW_PLUS_5_HOURS = (datetime.now(tz=timezone.utc) + timedelta(hours=5)).isoformat()
VISIT_STATUSES = [
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_1",
"last_eventful_visit_date": NOW,
},
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_1",
"last_eventful_visit_date": NOW_MINUS_5_HOURS,
},
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_2",
"last_eventful_visit_date": NOW_PLUS_5_HOURS,
},
]
class CommonSearchTest: class CommonSearchTest:
def test_origin_url_unique_word_prefix(self): def test_origin_url_unique_word_prefix(self):
origin_foobar_baz = {"url": "http://foobar.baz"} origin_foobar_baz = {"url": "http://foobar.baz"}
origin_barbaz_qux = {"url": "http://barbaz.qux"} origin_barbaz_qux = {"url": "http://barbaz.qux"}
origin_qux_quux = {"url": "http://qux.quux"} origin_qux_quux = {"url": "http://qux.quux"}
origins = [origin_foobar_baz, origin_barbaz_qux, origin_qux_quux] origins = [origin_foobar_baz, origin_barbaz_qux, origin_qux_quux]
▲ Show 20 Lines • Show All 231 Lines • ▼ Show 20 Lines def test_origin_last_visit_date_update_search(self):
_check_min_last_visit_date(now_minus_5_hours) # Works for < _check_min_last_visit_date(now_minus_5_hours) # Works for <
with pytest.raises(AssertionError): with pytest.raises(AssertionError):
_check_min_last_visit_date(now_plus_5_hours) # Fails for > _check_min_last_visit_date(now_plus_5_hours) # Fails for >
_update_last_visit_date(now_plus_5_hours) _update_last_visit_date(now_plus_5_hours)
_check_min_last_visit_date(now_plus_5_hours) # Works for = _check_min_last_visit_date(now_plus_5_hours) # Works for =
_check_min_last_visit_date(now) # Works for < _check_min_last_visit_date(now) # Works for <
vsellierUnsubmitted
Not Done
Inline Actions

A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order gives the expected result
Please have a look at the equivalent on the scheduler:
https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/tests/test_journal_client.py$418

vsellier: A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order…
def test_journal_client_origin_visit_status_permutation(self):
for visit_statuses in permutations(VISIT_STATUSES, len(VISIT_STATUSES)):
self.search.origin_update(visit_statuses)
self.search.flush()
origin_url = "http://foobar.baz"
actual_page = self.search.origin_search(
url_pattern=origin_url, min_last_eventful_visit_date=NOW_PLUS_5_HOURS,
)
assert actual_page.next_page_token is None
results = [r["url"] for r in actual_page.results]
expected_results = [origin_url]
assert sorted(results) == sorted(expected_results)
def test_origin_last_eventful_visit_date_update_search(self):
origin_url = "http://foobar.baz"
self.search.origin_update([{"url": origin_url}])
self.search.flush()
def _update_last_eventful_visit_date(snapshot_id, last_eventful_visit_date):
self.search.origin_update(
[
{
"url": origin_url,
"snapshot_id": snapshot_id,
"last_eventful_visit_date": last_eventful_visit_date,
}
]
vsellierUnsubmitted
Not Done
Inline Actions

Is there a reason to use permutation instead of a parameterized test?
With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

@vlorentz WDYT ?

vsellier: Is there a reason to use permutation instead of a parameterized test? With this implementation…
vlorentzUnsubmitted
Not Done
Inline Actions

I agree

vlorentz: I agree
KShivenduAuthorUnsubmitted
Done
Inline Actions

I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work-with-unittest-class-based-tests which says.

unittest.TestCase methods cannot directly receive fixture function arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.

Is this stackoverflow answer wrong ?

KShivendu: I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work…
KShivenduAuthorUnsubmitted
Done
Inline Actions

With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

Ahh, I see. My bad.

KShivendu: >> With this implementation the index is not reset between 2 iterations so after the first loop…
vlorentzUnsubmitted
Not Done
Inline Actions

This is not a unittest.TestCase-based test; it's just a regular class.

vlorentz: This is not a `unittest.TestCase`-based test; it's just a regular class.
)
self.search.flush()
def _check_min_last_eventful_visit_date(min_last_eventful_visit_date):
actual_page = self.search.origin_search(
url_pattern=origin_url,
min_last_eventful_visit_date=min_last_eventful_visit_date,
)
assert actual_page.next_page_token is None
results = [r["url"] for r in actual_page.results]
expected_results = [origin_url]
assert sorted(results) == sorted(expected_results)
now = datetime.now(tz=timezone.utc).isoformat()
now_minus_5_hours = (
datetime.now(tz=timezone.utc) - timedelta(hours=5)
).isoformat()
now_plus_5_hours = (
datetime.now(tz=timezone.utc) + timedelta(hours=5)
).isoformat()
snapshot_1 = "SNAPSHOT_1"
snapshot_2 = "SNAPSHOT_2"
_update_last_eventful_visit_date(snapshot_1, now)
_check_min_last_eventful_visit_date(now) # Works for =
_check_min_last_eventful_visit_date(now_minus_5_hours) # Works for <
with pytest.raises(AssertionError):
_check_min_last_eventful_visit_date(now_plus_5_hours) # Fails for >
_update_last_eventful_visit_date(
snapshot_1, now_plus_5_hours
) # Revisit(not eventful) same origin
_check_min_last_eventful_visit_date(
now
) # Should remain the same because recent visit wasn't eventful
with pytest.raises(AssertionError):
_check_min_last_eventful_visit_date(now_plus_5_hours)
_update_last_eventful_visit_date(
snapshot_2, now_plus_5_hours
) # Revisit(eventful) same origin
_check_min_last_eventful_visit_date(now_plus_5_hours) # Works for =
_check_min_last_eventful_visit_date(now) # Works for <
def test_origin_update_with_no_visit_types(self): def test_origin_update_with_no_visit_types(self):
""" """
Update an origin with visit types first then with no visit types, Update an origin with visit types first then with no visit types,
check origin can still be searched with visit types afterwards. check origin can still be searched with visit types afterwards.
""" """
origin_url = "http://foobar.baz" origin_url = "http://foobar.baz"
self.search.origin_update([{"url": origin_url, "visit_types": ["git"]}]) self.search.origin_update([{"url": origin_url, "visit_types": ["git"]}])
self.search.flush() self.search.flush()
▲ Show 20 Lines • Show All 427 Lines • Show Last 20 Lines
View Options

swh/storage/in_memory.py

# Copyright (C) 2019-2021 The Software Heritage developers # Copyright (C) 2019-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from itertools import permutations
from hypothesis import given, settings, strategies from hypothesis import given, settings, strategies
import pytest import pytest
from swh.core.api.classes import stream_results from swh.core.api.classes import stream_results
NOW = datetime.now(tz=timezone.utc).isoformat()
NOW_MINUS_5_HOURS = (datetime.now(tz=timezone.utc) - timedelta(hours=5)).isoformat()
NOW_PLUS_5_HOURS = (datetime.now(tz=timezone.utc) + timedelta(hours=5)).isoformat()
VISIT_STATUSES = [
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_1",
"last_eventful_visit_date": NOW,
},
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_1",
"last_eventful_visit_date": NOW_MINUS_5_HOURS,
},
{
"url": "http://foobar.baz",
"snapshot_id": "SNAPSHOT_2",
"last_eventful_visit_date": NOW_PLUS_5_HOURS,
},
]
class CommonSearchTest: class CommonSearchTest:
def test_origin_url_unique_word_prefix(self): def test_origin_url_unique_word_prefix(self):
origin_foobar_baz = {"url": "http://foobar.baz"} origin_foobar_baz = {"url": "http://foobar.baz"}
origin_barbaz_qux = {"url": "http://barbaz.qux"} origin_barbaz_qux = {"url": "http://barbaz.qux"}
origin_qux_quux = {"url": "http://qux.quux"} origin_qux_quux = {"url": "http://qux.quux"}
origins = [origin_foobar_baz, origin_barbaz_qux, origin_qux_quux] origins = [origin_foobar_baz, origin_barbaz_qux, origin_qux_quux]
▲ Show 20 Lines • Show All 231 Lines • ▼ Show 20 Lines def test_origin_last_visit_date_update_search(self):
_check_min_last_visit_date(now_minus_5_hours) # Works for < _check_min_last_visit_date(now_minus_5_hours) # Works for <
with pytest.raises(AssertionError): with pytest.raises(AssertionError):
_check_min_last_visit_date(now_plus_5_hours) # Fails for > _check_min_last_visit_date(now_plus_5_hours) # Fails for >
_update_last_visit_date(now_plus_5_hours) _update_last_visit_date(now_plus_5_hours)
_check_min_last_visit_date(now_plus_5_hours) # Works for = _check_min_last_visit_date(now_plus_5_hours) # Works for =
_check_min_last_visit_date(now) # Works for < _check_min_last_visit_date(now) # Works for <
vsellierUnsubmitted
Not Done
Inline Actions

A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order gives the expected result
Please have a look at the equivalent on the scheduler:
https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/tests/test_journal_client.py$418

vsellier: A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order…
def test_journal_client_origin_visit_status_permutation(self):
for visit_statuses in permutations(VISIT_STATUSES, len(VISIT_STATUSES)):
self.search.origin_update(visit_statuses)
self.search.flush()
origin_url = "http://foobar.baz"
actual_page = self.search.origin_search(
url_pattern=origin_url, min_last_eventful_visit_date=NOW_PLUS_5_HOURS,
)
assert actual_page.next_page_token is None
results = [r["url"] for r in actual_page.results]
expected_results = [origin_url]
assert sorted(results) == sorted(expected_results)
def test_origin_last_eventful_visit_date_update_search(self):
origin_url = "http://foobar.baz"
self.search.origin_update([{"url": origin_url}])
self.search.flush()
def _update_last_eventful_visit_date(snapshot_id, last_eventful_visit_date):
self.search.origin_update(
[
{
"url": origin_url,
"snapshot_id": snapshot_id,
"last_eventful_visit_date": last_eventful_visit_date,
}
]
vsellierUnsubmitted
Not Done
Inline Actions

Is there a reason to use permutation instead of a parameterized test?
With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

@vlorentz WDYT ?

vsellier: Is there a reason to use permutation instead of a parameterized test? With this implementation…
vlorentzUnsubmitted
Not Done
Inline Actions

I agree

vlorentz: I agree
KShivenduAuthorUnsubmitted
Done
Inline Actions

I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work-with-unittest-class-based-tests which says.

unittest.TestCase methods cannot directly receive fixture function arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.

Is this stackoverflow answer wrong ?

KShivendu: I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work…
KShivenduAuthorUnsubmitted
Done
Inline Actions

With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

Ahh, I see. My bad.

KShivendu: >> With this implementation the index is not reset between 2 iterations so after the first loop…
vlorentzUnsubmitted
Not Done
Inline Actions

This is not a unittest.TestCase-based test; it's just a regular class.

vlorentz: This is not a `unittest.TestCase`-based test; it's just a regular class.
)
self.search.flush()
def _check_min_last_eventful_visit_date(min_last_eventful_visit_date):
actual_page = self.search.origin_search(
url_pattern=origin_url,
min_last_eventful_visit_date=min_last_eventful_visit_date,
)
assert actual_page.next_page_token is None
results = [r["url"] for r in actual_page.results]
expected_results = [origin_url]
assert sorted(results) == sorted(expected_results)
now = datetime.now(tz=timezone.utc).isoformat()
now_minus_5_hours = (
datetime.now(tz=timezone.utc) - timedelta(hours=5)
).isoformat()
now_plus_5_hours = (
datetime.now(tz=timezone.utc) + timedelta(hours=5)
).isoformat()
snapshot_1 = "SNAPSHOT_1"
snapshot_2 = "SNAPSHOT_2"
_update_last_eventful_visit_date(snapshot_1, now)
_check_min_last_eventful_visit_date(now) # Works for =
_check_min_last_eventful_visit_date(now_minus_5_hours) # Works for <
with pytest.raises(AssertionError):
_check_min_last_eventful_visit_date(now_plus_5_hours) # Fails for >
_update_last_eventful_visit_date(
snapshot_1, now_plus_5_hours
) # Revisit(not eventful) same origin
_check_min_last_eventful_visit_date(
now
) # Should remain the same because recent visit wasn't eventful
with pytest.raises(AssertionError):
_check_min_last_eventful_visit_date(now_plus_5_hours)
_update_last_eventful_visit_date(
snapshot_2, now_plus_5_hours
) # Revisit(eventful) same origin
_check_min_last_eventful_visit_date(now_plus_5_hours) # Works for =
_check_min_last_eventful_visit_date(now) # Works for <
def test_origin_update_with_no_visit_types(self): def test_origin_update_with_no_visit_types(self):
""" """
Update an origin with visit types first then with no visit types, Update an origin with visit types first then with no visit types,
check origin can still be searched with visit types afterwards. check origin can still be searched with visit types afterwards.
""" """
origin_url = "http://foobar.baz" origin_url = "http://foobar.baz"
self.search.origin_update([{"url": origin_url, "visit_types": ["git"]}]) self.search.origin_update([{"url": origin_url, "visit_types": ["git"]}])
self.search.flush() self.search.flush()
▲ Show 20 Lines • Show All 427 Lines • Show Last 20 Lines
View Options

swh/storage/storage.py

Loading...
View Options

swh/storage/tests/generate_data_test.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal