Page MenuHomeSoftware Heritage
Differential D8049

Crates loader, use **kwargs on task and loader initialisation instead of named args
ClosedPublic
Actions

Authored by franckbret on Jun 29 2022, 2:58 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDLDBASE50bde53da53b: Crates loader, use **kwargs on task and loader initialisation instead of
Summary

This fix an issue discovered while testing loader on docker environment,
error was (load_crates() got an unexpected keyword argument 'lister_name')

Related T4104

Diff Detail

Event Timeline

franckbret created this revision.Jun 29 2022, 2:58 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 29 2022, 2:58 PM
swh-public-ci added a comment.Jun 29 2022, 3:01 PM

Build is green

Patch application report for D8049 (id=29039)

Rebasing onto d627c2598a...

Current branch diff-target is up to date.
Changes applied before test
commit 50bde53da53b7fce9616da3f208831948700ce8e
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Jun 29 14:49:27 2022 +0200

    Crates loader, use **kwargs on task and loader initialisation instead of
    named args
    
    This fix an issue discovered while testing loader on docker environment,
    error was (load_crates() got an unexpected keyword argument 'lister_name')
    
    Related T4104

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/809/ for more details.

Harbormaster completed remote builds in B30125: Diff 29039.Jun 29 2022, 3:01 PM
franckbret requested review of this revision.Jun 29 2022, 3:01 PM
franckbret mentioned this in T4104: Ingest crates.io (Rust).Jun 29 2022, 3:03 PM
ardumont accepted this revision.Jun 29 2022, 3:20 PM
This revision is now accepted and ready to land.Jun 29 2022, 3:20 PM
Closed by commit rDLDBASE50bde53da53b: Crates loader, use **kwargs on task and loader initialisation instead of (authored by franckbret). · Explain WhyJun 29 2022, 3:28 PM
This revision was automatically updated to reflect the committed changes.
franckbret added a commit: rDLDBASE50bde53da53b: Crates loader, use **kwargs on task and loader initialisation instead of.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
loader/
package/
crates/
5 lines100%
Loading...
4 lines100%
Loading...

Diff 29039

View Options

swh/loader/package/crates/loader.py

Show First 20 Lines • Show All 77 Lines • ▼ Show 20 Lines
def generate_archive_graph_api(swhid: SWHID) -> None: def generate_archive_graph_api(swhid: SWHID) -> None:
if swhid.object_type == REVISION: if swhid.object_type == REVISION:
# Empty history for all revisions (except REV_SMALL_HISTORY used in tests) # Empty history for all revisions (except REV_SMALL_HISTORY used in tests)
url = swhid_to_graph_url(swhid, GRAPH_API_REQUEST.HISTORY) url = swhid_to_graph_url(swhid, GRAPH_API_REQUEST.HISTORY)
MOCK_ARCHIVE[url] = "" MOCK_ARCHIVE[url] = ""
if str(swhid) == REV_SMALL_HISTORY: if str(swhid) == REV_SMALL_HISTORY:
# TODO: temporary fix, retrieve from the graph API once it is public # TODO: temporary fix, retrieve from the graph API once it is public
MOCK_ARCHIVE[ history = """
url
] = """
swh:1:rev:37426e42cf78a43779312d780eecb21a64006d99 swh:1:rev:0cf3c2ad935be699281ed20fb3d2f29554e6229b swh:1:rev:37426e42cf78a43779312d780eecb21a64006d99 swh:1:rev:0cf3c2ad935be699281ed20fb3d2f29554e6229b
swh:1:rev:0cf3c2ad935be699281ed20fb3d2f29554e6229b swh:1:rev:37180552769b316e7239d047008f187127e630e6 swh:1:rev:0cf3c2ad935be699281ed20fb3d2f29554e6229b swh:1:rev:37180552769b316e7239d047008f187127e630e6
swh:1:rev:37180552769b316e7239d047008f187127e630e6 swh:1:rev:dd2716f56c7cf55f2904fbbf4dfabaab1afbcd88 swh:1:rev:37180552769b316e7239d047008f187127e630e6 swh:1:rev:dd2716f56c7cf55f2904fbbf4dfabaab1afbcd88
swh:1:rev:dd2716f56c7cf55f2904fbbf4dfabaab1afbcd88 swh:1:rev:968ec145278d3d6562e4b5ec4006af97dc0da563 swh:1:rev:dd2716f56c7cf55f2904fbbf4dfabaab1afbcd88 swh:1:rev:968ec145278d3d6562e4b5ec4006af97dc0da563
swh:1:rev:968ec145278d3d6562e4b5ec4006af97dc0da563 swh:1:rev:34dc7053ebfd440648f49dc83d2538ab5e7ceda5 swh:1:rev:968ec145278d3d6562e4b5ec4006af97dc0da563 swh:1:rev:34dc7053ebfd440648f49dc83d2538ab5e7ceda5
swh:1:rev:34dc7053ebfd440648f49dc83d2538ab5e7ceda5 swh:1:rev:c56a729ff1d9467d612bf522614519ac7b97f798 swh:1:rev:34dc7053ebfd440648f49dc83d2538ab5e7ceda5 swh:1:rev:c56a729ff1d9467d612bf522614519ac7b97f798
swh:1:rev:c56a729ff1d9467d612bf522614519ac7b97f798 swh:1:rev:eb7807c4fe7a2c2ad3c074705fb70de5eae5abe3 swh:1:rev:c56a729ff1d9467d612bf522614519ac7b97f798 swh:1:rev:eb7807c4fe7a2c2ad3c074705fb70de5eae5abe3
swh:1:rev:eb7807c4fe7a2c2ad3c074705fb70de5eae5abe3 swh:1:rev:d601b357ecbb1fa33dc10c177bb557868be07deb swh:1:rev:eb7807c4fe7a2c2ad3c074705fb70de5eae5abe3 swh:1:rev:d601b357ecbb1fa33dc10c177bb557868be07deb
Show All 26 Lines
swh:1:rev:4a1f86ccd7e823f63d12208baef79b1e74479203 swh:1:rev:0016473117e4bc3c8959bf2fd49368844847d74c swh:1:rev:4a1f86ccd7e823f63d12208baef79b1e74479203 swh:1:rev:0016473117e4bc3c8959bf2fd49368844847d74c
swh:1:rev:0016473117e4bc3c8959bf2fd49368844847d74c swh:1:rev:935442babcf4f8ae52c1a13bb9ce07270a302886 swh:1:rev:0016473117e4bc3c8959bf2fd49368844847d74c swh:1:rev:935442babcf4f8ae52c1a13bb9ce07270a302886
swh:1:rev:935442babcf4f8ae52c1a13bb9ce07270a302886 swh:1:rev:1f3cff91f6762b0f47f41025b5e2c5ac942479ba swh:1:rev:935442babcf4f8ae52c1a13bb9ce07270a302886 swh:1:rev:1f3cff91f6762b0f47f41025b5e2c5ac942479ba
swh:1:rev:1f3cff91f6762b0f47f41025b5e2c5ac942479ba swh:1:rev:bc286c7f2ceb5c3d2e06ec72f78d28842f94ef65 swh:1:rev:1f3cff91f6762b0f47f41025b5e2c5ac942479ba swh:1:rev:bc286c7f2ceb5c3d2e06ec72f78d28842f94ef65
swh:1:rev:bc286c7f2ceb5c3d2e06ec72f78d28842f94ef65 swh:1:rev:f038f4d533f897a29f9422510d1b3f0caac97388 swh:1:rev:bc286c7f2ceb5c3d2e06ec72f78d28842f94ef65 swh:1:rev:f038f4d533f897a29f9422510d1b3f0caac97388
swh:1:rev:f038f4d533f897a29f9422510d1b3f0caac97388 swh:1:rev:d6b7c96c3eb29b9244ece0c046d3f372ff432d04 swh:1:rev:f038f4d533f897a29f9422510d1b3f0caac97388 swh:1:rev:d6b7c96c3eb29b9244ece0c046d3f372ff432d04
swh:1:rev:d6b7c96c3eb29b9244ece0c046d3f372ff432d04 swh:1:rev:c01efc669f09508b55eced32d3c88702578a7c3e swh:1:rev:d6b7c96c3eb29b9244ece0c046d3f372ff432d04 swh:1:rev:c01efc669f09508b55eced32d3c88702578a7c3e
""" # NoQA: E501 """ # NoQA: E501
MOCK_ARCHIVE[url] = history
hist_nodes = set(
map(
parse_swhid,
[edge.split(" ")[1] for edge in history.strip().split("\n")],
)
)
for swhid in hist_nodes:
generate_archive_web_api(swhid, recursive=False)
for entry in ALL_ENTRIES: for entry in ALL_ENTRIES:
swhid = parse_swhid(entry) swhid = parse_swhid(entry)
generate_archive_web_api(swhid, recursive=True) generate_archive_web_api(swhid, recursive=True)
generate_archive_graph_api(swhid) generate_archive_graph_api(swhid)
print("# GENERATED FILE, DO NOT EDIT.") print("# GENERATED FILE, DO NOT EDIT.")
print("# Run './gen-api-data.py > api_data.py' instead.") print("# Run './gen-api-data.py > api_data.py' instead.")
print("# flake8: noqa") print("# flake8: noqa")
print("from typing import Any, Dict") print("from typing import Any, Dict")
print("") print("")
print(f"API_URL = '{API_URL_test}'\n") print(f"API_URL = '{API_URL_test}'\n")
print(f"MOCK_ARCHIVE: Dict[str, Any] = {MOCK_ARCHIVE}") print(f"MOCK_ARCHIVE: Dict[str, Any] = {MOCK_ARCHIVE}")
View Options

swh/loader/package/crates/tasks.py

Show First 20 Lines • Show All 77 Lines • ▼ Show 20 Lines
def generate_archive_graph_api(swhid: SWHID) -> None: def generate_archive_graph_api(swhid: SWHID) -> None:
if swhid.object_type == REVISION: if swhid.object_type == REVISION:
# Empty history for all revisions (except REV_SMALL_HISTORY used in tests) # Empty history for all revisions (except REV_SMALL_HISTORY used in tests)
url = swhid_to_graph_url(swhid, GRAPH_API_REQUEST.HISTORY) url = swhid_to_graph_url(swhid, GRAPH_API_REQUEST.HISTORY)
MOCK_ARCHIVE[url] = "" MOCK_ARCHIVE[url] = ""
if str(swhid) == REV_SMALL_HISTORY: if str(swhid) == REV_SMALL_HISTORY:
# TODO: temporary fix, retrieve from the graph API once it is public # TODO: temporary fix, retrieve from the graph API once it is public
MOCK_ARCHIVE[ history = """
url
] = """
swh:1:rev:37426e42cf78a43779312d780eecb21a64006d99 swh:1:rev:0cf3c2ad935be699281ed20fb3d2f29554e6229b swh:1:rev:37426e42cf78a43779312d780eecb21a64006d99 swh:1:rev:0cf3c2ad935be699281ed20fb3d2f29554e6229b
swh:1:rev:0cf3c2ad935be699281ed20fb3d2f29554e6229b swh:1:rev:37180552769b316e7239d047008f187127e630e6 swh:1:rev:0cf3c2ad935be699281ed20fb3d2f29554e6229b swh:1:rev:37180552769b316e7239d047008f187127e630e6
swh:1:rev:37180552769b316e7239d047008f187127e630e6 swh:1:rev:dd2716f56c7cf55f2904fbbf4dfabaab1afbcd88 swh:1:rev:37180552769b316e7239d047008f187127e630e6 swh:1:rev:dd2716f56c7cf55f2904fbbf4dfabaab1afbcd88
swh:1:rev:dd2716f56c7cf55f2904fbbf4dfabaab1afbcd88 swh:1:rev:968ec145278d3d6562e4b5ec4006af97dc0da563 swh:1:rev:dd2716f56c7cf55f2904fbbf4dfabaab1afbcd88 swh:1:rev:968ec145278d3d6562e4b5ec4006af97dc0da563
swh:1:rev:968ec145278d3d6562e4b5ec4006af97dc0da563 swh:1:rev:34dc7053ebfd440648f49dc83d2538ab5e7ceda5 swh:1:rev:968ec145278d3d6562e4b5ec4006af97dc0da563 swh:1:rev:34dc7053ebfd440648f49dc83d2538ab5e7ceda5
swh:1:rev:34dc7053ebfd440648f49dc83d2538ab5e7ceda5 swh:1:rev:c56a729ff1d9467d612bf522614519ac7b97f798 swh:1:rev:34dc7053ebfd440648f49dc83d2538ab5e7ceda5 swh:1:rev:c56a729ff1d9467d612bf522614519ac7b97f798
swh:1:rev:c56a729ff1d9467d612bf522614519ac7b97f798 swh:1:rev:eb7807c4fe7a2c2ad3c074705fb70de5eae5abe3 swh:1:rev:c56a729ff1d9467d612bf522614519ac7b97f798 swh:1:rev:eb7807c4fe7a2c2ad3c074705fb70de5eae5abe3
swh:1:rev:eb7807c4fe7a2c2ad3c074705fb70de5eae5abe3 swh:1:rev:d601b357ecbb1fa33dc10c177bb557868be07deb swh:1:rev:eb7807c4fe7a2c2ad3c074705fb70de5eae5abe3 swh:1:rev:d601b357ecbb1fa33dc10c177bb557868be07deb
Show All 26 Lines
swh:1:rev:4a1f86ccd7e823f63d12208baef79b1e74479203 swh:1:rev:0016473117e4bc3c8959bf2fd49368844847d74c swh:1:rev:4a1f86ccd7e823f63d12208baef79b1e74479203 swh:1:rev:0016473117e4bc3c8959bf2fd49368844847d74c
swh:1:rev:0016473117e4bc3c8959bf2fd49368844847d74c swh:1:rev:935442babcf4f8ae52c1a13bb9ce07270a302886 swh:1:rev:0016473117e4bc3c8959bf2fd49368844847d74c swh:1:rev:935442babcf4f8ae52c1a13bb9ce07270a302886
swh:1:rev:935442babcf4f8ae52c1a13bb9ce07270a302886 swh:1:rev:1f3cff91f6762b0f47f41025b5e2c5ac942479ba swh:1:rev:935442babcf4f8ae52c1a13bb9ce07270a302886 swh:1:rev:1f3cff91f6762b0f47f41025b5e2c5ac942479ba
swh:1:rev:1f3cff91f6762b0f47f41025b5e2c5ac942479ba swh:1:rev:bc286c7f2ceb5c3d2e06ec72f78d28842f94ef65 swh:1:rev:1f3cff91f6762b0f47f41025b5e2c5ac942479ba swh:1:rev:bc286c7f2ceb5c3d2e06ec72f78d28842f94ef65
swh:1:rev:bc286c7f2ceb5c3d2e06ec72f78d28842f94ef65 swh:1:rev:f038f4d533f897a29f9422510d1b3f0caac97388 swh:1:rev:bc286c7f2ceb5c3d2e06ec72f78d28842f94ef65 swh:1:rev:f038f4d533f897a29f9422510d1b3f0caac97388
swh:1:rev:f038f4d533f897a29f9422510d1b3f0caac97388 swh:1:rev:d6b7c96c3eb29b9244ece0c046d3f372ff432d04 swh:1:rev:f038f4d533f897a29f9422510d1b3f0caac97388 swh:1:rev:d6b7c96c3eb29b9244ece0c046d3f372ff432d04
swh:1:rev:d6b7c96c3eb29b9244ece0c046d3f372ff432d04 swh:1:rev:c01efc669f09508b55eced32d3c88702578a7c3e swh:1:rev:d6b7c96c3eb29b9244ece0c046d3f372ff432d04 swh:1:rev:c01efc669f09508b55eced32d3c88702578a7c3e
""" # NoQA: E501 """ # NoQA: E501
MOCK_ARCHIVE[url] = history
hist_nodes = set(
map(
parse_swhid,
[edge.split(" ")[1] for edge in history.strip().split("\n")],
)
)
for swhid in hist_nodes:
generate_archive_web_api(swhid, recursive=False)
for entry in ALL_ENTRIES: for entry in ALL_ENTRIES:
swhid = parse_swhid(entry) swhid = parse_swhid(entry)
generate_archive_web_api(swhid, recursive=True) generate_archive_web_api(swhid, recursive=True)
generate_archive_graph_api(swhid) generate_archive_graph_api(swhid)
print("# GENERATED FILE, DO NOT EDIT.") print("# GENERATED FILE, DO NOT EDIT.")
print("# Run './gen-api-data.py > api_data.py' instead.") print("# Run './gen-api-data.py > api_data.py' instead.")
print("# flake8: noqa") print("# flake8: noqa")
print("from typing import Any, Dict") print("from typing import Any, Dict")
print("") print("")
print(f"API_URL = '{API_URL_test}'\n") print(f"API_URL = '{API_URL_test}'\n")
print(f"MOCK_ARCHIVE: Dict[str, Any] = {MOCK_ARCHIVE}") print(f"MOCK_ARCHIVE: Dict[str, Any] = {MOCK_ARCHIVE}")
About · Developer information · Legal