Page MenuHomeSoftware Heritage
Differential D2289

add /last sub-endpoint to only return destination in walks
ClosedPublic
Actions

Authored by zack on Nov 16 2019, 3:14 PM.

Details

Reviewers
seirl
Group Reviewers
Reviewers
Maniphest Tasks
T2084: swh-graph: add /last endpoint variants to the REST API
Commits
rDGRPH97e84765c276: REST API doc: document /last for all applicable methods
rDGRPH51d6b602c3e8: add /last sub-endpoint to only return destination in walks
Summary

works for both random and non-random walks

Closes T2084

Test Plan

tox

Diff Detail

Repository
rDGRPH Compressed graph representation
Branch
feature/random-walk-last
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 9083
Build 13286: tox-on-jenkinsJenkins
Build 13285: arc lint + arc unit

Event Timeline

zack created this revision.Nov 16 2019, 3:14 PM
swh-public-ci added a comment.Nov 16 2019, 3:16 PM

Build is green
See https://jenkins.softwareheritage.org/job/DGRPH/job/tox/101/ for more details.

Harbormaster completed remote builds in B9060: Diff 7872.Nov 16 2019, 3:16 PM
zack updated this revision to Diff 7875.Nov 16 2019, 7:31 PM
  • REST API doc: document /last for all applicable methods
Harbormaster failed remote builds in B9064: Diff 7875!Nov 16 2019, 7:32 PM
swh-public-ci added a comment.Nov 16 2019, 7:32 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/103/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/103/console

zack updated this revision to Diff 7876.Nov 16 2019, 8:02 PM
  • add /last sub-endpoint to only return destination in walks
  • REST API doc: document /last for all applicable methods
swh-public-ci added a comment.Nov 16 2019, 8:04 PM

Build is green
See https://jenkins.softwareheritage.org/job/DGRPH/job/tox/104/ for more details.

Harbormaster completed remote builds in B9065: Diff 7876.Nov 16 2019, 8:04 PM
seirl requested changes to this revision.Nov 18 2019, 1:36 PM
seirl added inline comments.
swh/graph/tests/test_api_client.py
108

I don't think it's useful to do copy() if you're assigning it to kwargs anyway, the old value will just get garbage collected. Either don't copy() or use a different variable name maybe?

131

ditto

This revision now requires changes to proceed.Nov 18 2019, 1:36 PM
zack updated this revision to Diff 7891.Nov 18 2019, 1:49 PM
zack marked 2 inline comments as done.
  • add /last sub-endpoint to only return destination in walks
  • REST API doc: document /last for all applicable methods
swh/graph/tests/test_api_client.py
108

lol, yeah, I meant to use a different variable name here :-)

swh-public-ci added a comment.Nov 18 2019, 1:51 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/107/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/107/console

Harbormaster failed remote builds in B9083: Diff 7891!Nov 18 2019, 1:51 PM
seirl accepted this revision.Nov 18 2019, 1:53 PM
This revision is now accepted and ready to land.Nov 18 2019, 1:53 PM
zack updated this revision to Diff 7892.Nov 18 2019, 1:53 PM
  • add /last sub-endpoint to only return destination in walks
  • REST API doc: document /last for all applicable methods
swh-public-ci added a comment.Nov 18 2019, 1:55 PM

Build is green
See https://jenkins.softwareheritage.org/job/DGRPH/job/tox/108/ for more details.

Harbormaster completed remote builds in B9085: Diff 7892.Nov 18 2019, 1:55 PM
Closed by commit rDGRPH51d6b602c3e8: add /last sub-endpoint to only return destination in walks (authored by zack). · Explain WhyNov 18 2019, 1:56 PM
This revision was automatically updated to reflect the committed changes.
zack added a commit: rDGRPH51d6b602c3e8: add /last sub-endpoint to only return destination in walks.
zack added a commit: rDGRPH97e84765c276: REST API doc: document /last for all applicable methods.

Revision Contents
Changeset List

PathSize
docs/
16 lines
swh/
graph/
12 lines
server/
13 lines
tests/
36 lines

Diff 7891

View Options

docs/api.rst

Show First 20 Lines • Show All 103 Lines • ▼ Show 20 Linesdef test_retrieve_sources(swh_config, requests_mock_datadir):
j = parse_sources(retrieve_sources(sources_url)) j = parse_sources(retrieve_sources(sources_url))
assert "sources" in j.keys() assert "sources" in j.keys()
assert len(j["sources"]) == 2 assert len(j["sources"]) == 2
def test_nixguix_url_not_found(swh_config, requests_mock_datadir): def test_nixguix_url_not_found(swh_config, requests_mock_datadir):
"""When failing to read from the url, the visit is marked as not_found. """When failing to read from the url, the visit is marked as not_found.
Here the sources url does not exist, so requests_mock_datadir returns a 404.
Resulting in a NotFound raised within the package loader's main loop.
This results in the task with status failed and a visit_status with status
"not_found".
""" """
unknown_url = "https://non-existing-url" unknown_url = "https://non-existing-url/"
loader = NixGuixLoader(unknown_url) loader = NixGuixLoader(unknown_url)
# during the retrieval step # during the retrieval step
load_status = loader.load() load_status = loader.load()
assert load_status == {"status": "failed"} assert load_status == {"status": "failed"}
assert_last_visit_matches( assert_last_visit_matches(
loader.storage, unknown_url, status="not_found", type="nixguix", snapshot=None loader.storage, unknown_url, status="not_found", type="nixguix", snapshot=None
) )
assert len(requests_mock_datadir.request_history) == 1
assert requests_mock_datadir.request_history[0].url == unknown_url
vlorentzUnsubmitted
Not Done
Inline Actions

why this conditional?

vlorentz: why this conditional?
vlorentzUnsubmitted
Not Done
Inline Actions

(disregard this, I wrote it in the "full" comment)

vlorentz: (disregard this, I wrote it in the "full" comment)
def test_nixguix_url_with_decoding_error(swh_config, requests_mock_datadir): def test_nixguix_url_with_decoding_error(swh_config, requests_mock_datadir):
"""Other errors during communication with the url, the visit is marked as failed""" """Other errors during communication with the url, the visit is marked as failed
requests_mock_datadir will intercept the requests to sources_url. Since the file
exists, returns a 200 with the requested content of the query. As file.txt is no
json, fails do decode and raises a JSONDecodeError. In effect failing the visit.
"""
sources_url = "https://example.com/file.txt" sources_url = "https://example.com/file.txt"
loader = NixGuixLoader(sources_url) loader = NixGuixLoader(sources_url)
load_status = loader.load() load_status = loader.load()
assert load_status == {"status": "failed"} assert load_status == {"status": "failed"}
assert_last_visit_matches( assert_last_visit_matches(
loader.storage, sources_url, status="failed", type="nixguix", snapshot=None loader.storage, sources_url, status="failed", type="nixguix", snapshot=None
) )
assert len(requests_mock_datadir.request_history) == 1
assert requests_mock_datadir.request_history[0].url == sources_url
def test_clean_sources_invalid_schema(swh_config, requests_mock_datadir): def test_clean_sources_invalid_schema(swh_config, requests_mock_datadir):
sources = {} sources = {}
with pytest.raises(ValueError, match="sources structure invalid, missing: .*"): with pytest.raises(ValueError, match="sources structure invalid, missing: .*"):
clean_sources(sources) clean_sources(sources)
def test_clean_sources_invalid_version(swh_config, requests_mock_datadir): def test_clean_sources_invalid_version(swh_config, requests_mock_datadir):
▲ Show 20 Lines • Show All 547 Lines • Show Last 20 Lines
View Options

swh/graph/client.py

Show First 20 Lines • Show All 103 Lines • ▼ Show 20 Linesdef test_retrieve_sources(swh_config, requests_mock_datadir):
j = parse_sources(retrieve_sources(sources_url)) j = parse_sources(retrieve_sources(sources_url))
assert "sources" in j.keys() assert "sources" in j.keys()
assert len(j["sources"]) == 2 assert len(j["sources"]) == 2
def test_nixguix_url_not_found(swh_config, requests_mock_datadir): def test_nixguix_url_not_found(swh_config, requests_mock_datadir):
"""When failing to read from the url, the visit is marked as not_found. """When failing to read from the url, the visit is marked as not_found.
Here the sources url does not exist, so requests_mock_datadir returns a 404.
Resulting in a NotFound raised within the package loader's main loop.
This results in the task with status failed and a visit_status with status
"not_found".
""" """
unknown_url = "https://non-existing-url" unknown_url = "https://non-existing-url/"
loader = NixGuixLoader(unknown_url) loader = NixGuixLoader(unknown_url)
# during the retrieval step # during the retrieval step
load_status = loader.load() load_status = loader.load()
assert load_status == {"status": "failed"} assert load_status == {"status": "failed"}
assert_last_visit_matches( assert_last_visit_matches(
loader.storage, unknown_url, status="not_found", type="nixguix", snapshot=None loader.storage, unknown_url, status="not_found", type="nixguix", snapshot=None
) )
assert len(requests_mock_datadir.request_history) == 1
assert requests_mock_datadir.request_history[0].url == unknown_url
vlorentzUnsubmitted
Not Done
Inline Actions

why this conditional?

vlorentz: why this conditional?
vlorentzUnsubmitted
Not Done
Inline Actions

(disregard this, I wrote it in the "full" comment)

vlorentz: (disregard this, I wrote it in the "full" comment)
def test_nixguix_url_with_decoding_error(swh_config, requests_mock_datadir): def test_nixguix_url_with_decoding_error(swh_config, requests_mock_datadir):
"""Other errors during communication with the url, the visit is marked as failed""" """Other errors during communication with the url, the visit is marked as failed
requests_mock_datadir will intercept the requests to sources_url. Since the file
exists, returns a 200 with the requested content of the query. As file.txt is no
json, fails do decode and raises a JSONDecodeError. In effect failing the visit.
"""
sources_url = "https://example.com/file.txt" sources_url = "https://example.com/file.txt"
loader = NixGuixLoader(sources_url) loader = NixGuixLoader(sources_url)
load_status = loader.load() load_status = loader.load()
assert load_status == {"status": "failed"} assert load_status == {"status": "failed"}
assert_last_visit_matches( assert_last_visit_matches(
loader.storage, sources_url, status="failed", type="nixguix", snapshot=None loader.storage, sources_url, status="failed", type="nixguix", snapshot=None
) )
assert len(requests_mock_datadir.request_history) == 1
assert requests_mock_datadir.request_history[0].url == sources_url
def test_clean_sources_invalid_schema(swh_config, requests_mock_datadir): def test_clean_sources_invalid_schema(swh_config, requests_mock_datadir):
sources = {} sources = {}
with pytest.raises(ValueError, match="sources structure invalid, missing: .*"): with pytest.raises(ValueError, match="sources structure invalid, missing: .*"):
clean_sources(sources) clean_sources(sources)
def test_clean_sources_invalid_version(swh_config, requests_mock_datadir): def test_clean_sources_invalid_version(swh_config, requests_mock_datadir):
▲ Show 20 Lines • Show All 547 Lines • Show Last 20 Lines
View Options

swh/graph/server/app.py

Show First 20 Lines • Show All 103 Lines • ▼ Show 20 Linesdef test_retrieve_sources(swh_config, requests_mock_datadir):
j = parse_sources(retrieve_sources(sources_url)) j = parse_sources(retrieve_sources(sources_url))
assert "sources" in j.keys() assert "sources" in j.keys()
assert len(j["sources"]) == 2 assert len(j["sources"]) == 2
def test_nixguix_url_not_found(swh_config, requests_mock_datadir): def test_nixguix_url_not_found(swh_config, requests_mock_datadir):
"""When failing to read from the url, the visit is marked as not_found. """When failing to read from the url, the visit is marked as not_found.
Here the sources url does not exist, so requests_mock_datadir returns a 404.
Resulting in a NotFound raised within the package loader's main loop.
This results in the task with status failed and a visit_status with status
"not_found".
""" """
unknown_url = "https://non-existing-url" unknown_url = "https://non-existing-url/"
loader = NixGuixLoader(unknown_url) loader = NixGuixLoader(unknown_url)
# during the retrieval step # during the retrieval step
load_status = loader.load() load_status = loader.load()
assert load_status == {"status": "failed"} assert load_status == {"status": "failed"}
assert_last_visit_matches( assert_last_visit_matches(
loader.storage, unknown_url, status="not_found", type="nixguix", snapshot=None loader.storage, unknown_url, status="not_found", type="nixguix", snapshot=None
) )
assert len(requests_mock_datadir.request_history) == 1
assert requests_mock_datadir.request_history[0].url == unknown_url
vlorentzUnsubmitted
Not Done
Inline Actions

why this conditional?

vlorentz: why this conditional?
vlorentzUnsubmitted
Not Done
Inline Actions

(disregard this, I wrote it in the "full" comment)

vlorentz: (disregard this, I wrote it in the "full" comment)
def test_nixguix_url_with_decoding_error(swh_config, requests_mock_datadir): def test_nixguix_url_with_decoding_error(swh_config, requests_mock_datadir):
"""Other errors during communication with the url, the visit is marked as failed""" """Other errors during communication with the url, the visit is marked as failed
requests_mock_datadir will intercept the requests to sources_url. Since the file
exists, returns a 200 with the requested content of the query. As file.txt is no
json, fails do decode and raises a JSONDecodeError. In effect failing the visit.
"""
sources_url = "https://example.com/file.txt" sources_url = "https://example.com/file.txt"
loader = NixGuixLoader(sources_url) loader = NixGuixLoader(sources_url)
load_status = loader.load() load_status = loader.load()
assert load_status == {"status": "failed"} assert load_status == {"status": "failed"}
assert_last_visit_matches( assert_last_visit_matches(
loader.storage, sources_url, status="failed", type="nixguix", snapshot=None loader.storage, sources_url, status="failed", type="nixguix", snapshot=None
) )
assert len(requests_mock_datadir.request_history) == 1
assert requests_mock_datadir.request_history[0].url == sources_url
def test_clean_sources_invalid_schema(swh_config, requests_mock_datadir): def test_clean_sources_invalid_schema(swh_config, requests_mock_datadir):
sources = {} sources = {}
with pytest.raises(ValueError, match="sources structure invalid, missing: .*"): with pytest.raises(ValueError, match="sources structure invalid, missing: .*"):
clean_sources(sources) clean_sources(sources)
def test_clean_sources_invalid_version(swh_config, requests_mock_datadir): def test_clean_sources_invalid_version(swh_config, requests_mock_datadir):
▲ Show 20 Lines • Show All 547 Lines • Show Last 20 Lines
View Options

swh/graph/tests/test_api_client.py

Show First 20 Lines • Show All 103 Lines • ▼ Show 20 Linesdef test_retrieve_sources(swh_config, requests_mock_datadir):
j = parse_sources(retrieve_sources(sources_url)) j = parse_sources(retrieve_sources(sources_url))
assert "sources" in j.keys() assert "sources" in j.keys()
assert len(j["sources"]) == 2 assert len(j["sources"]) == 2
def test_nixguix_url_not_found(swh_config, requests_mock_datadir): def test_nixguix_url_not_found(swh_config, requests_mock_datadir):
"""When failing to read from the url, the visit is marked as not_found. """When failing to read from the url, the visit is marked as not_found.
Here the sources url does not exist, so requests_mock_datadir returns a 404.
Resulting in a NotFound raised within the package loader's main loop.
This results in the task with status failed and a visit_status with status
"not_found".
""" """
unknown_url = "https://non-existing-url" unknown_url = "https://non-existing-url/"
loader = NixGuixLoader(unknown_url) loader = NixGuixLoader(unknown_url)
# during the retrieval step # during the retrieval step
load_status = loader.load() load_status = loader.load()
assert load_status == {"status": "failed"} assert load_status == {"status": "failed"}
assert_last_visit_matches( assert_last_visit_matches(
loader.storage, unknown_url, status="not_found", type="nixguix", snapshot=None loader.storage, unknown_url, status="not_found", type="nixguix", snapshot=None
) )
assert len(requests_mock_datadir.request_history) == 1
assert requests_mock_datadir.request_history[0].url == unknown_url
vlorentzUnsubmitted
Not Done
Inline Actions

why this conditional?

vlorentz: why this conditional?
vlorentzUnsubmitted
Not Done
Inline Actions

(disregard this, I wrote it in the "full" comment)

vlorentz: (disregard this, I wrote it in the "full" comment)
def test_nixguix_url_with_decoding_error(swh_config, requests_mock_datadir): def test_nixguix_url_with_decoding_error(swh_config, requests_mock_datadir):
"""Other errors during communication with the url, the visit is marked as failed""" """Other errors during communication with the url, the visit is marked as failed
requests_mock_datadir will intercept the requests to sources_url. Since the file
exists, returns a 200 with the requested content of the query. As file.txt is no
json, fails do decode and raises a JSONDecodeError. In effect failing the visit.
"""
sources_url = "https://example.com/file.txt" sources_url = "https://example.com/file.txt"
loader = NixGuixLoader(sources_url) loader = NixGuixLoader(sources_url)
load_status = loader.load() load_status = loader.load()
assert load_status == {"status": "failed"} assert load_status == {"status": "failed"}
assert_last_visit_matches( assert_last_visit_matches(
loader.storage, sources_url, status="failed", type="nixguix", snapshot=None loader.storage, sources_url, status="failed", type="nixguix", snapshot=None
) )
assert len(requests_mock_datadir.request_history) == 1
assert requests_mock_datadir.request_history[0].url == sources_url
def test_clean_sources_invalid_schema(swh_config, requests_mock_datadir): def test_clean_sources_invalid_schema(swh_config, requests_mock_datadir):
sources = {} sources = {}
with pytest.raises(ValueError, match="sources structure invalid, missing: .*"): with pytest.raises(ValueError, match="sources structure invalid, missing: .*"):
clean_sources(sources) clean_sources(sources)
def test_clean_sources_invalid_version(swh_config, requests_mock_datadir): def test_clean_sources_invalid_version(swh_config, requests_mock_datadir):
▲ Show 20 Lines • Show All 547 Lines • Show Last 20 Lines
About · Developer information · Legal