Page MenuHomeSoftware Heritage

[wip] package.loader: introduce load_target and resolve_target_from
Changes PlannedPublic

Authored by lewo on Mar 19 2020, 10:51 AM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

DO NOT REVIEW IT YET!

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
functional-file
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 11217
Build 16938: tox-on-jenkinsJenkins
Build 16937: arc lint + arc unit

Event Timeline

lewo edited the summary of this revision. (Show Details)
  • WIP
lewo added a subscriber: ardumont.

@ardumont I don't understand why the storage doesn't contains 2 contents (a tarball + a file). The test test_loader_one_visit is failing because there is only 1 content in the storage.

swh/loader/package/functional/loader.py
143

beware to_model can return a Content or a SkippedContent.
That will not be the same endpoint for the storage:

  • Content: self.storage.content_add
  • SkippedContent: self.storage.skipped_content_add

@ardumont I don't understand why the storage doesn't contains 2 contents (a tarball + a file). The test test_loader_one_visit is failing because there is only 1 content in the storage.

It happens to be the same content...
the 2 files you have in the revision.tar.gz and the second file targetted by url in your sample are the same.
It's then deduplicated in the storage...

Change the content of the targetted file
and you should be good to go ;)

This needs a rebase as there were quite some work on the loader core ;)

Cheers,

swh/loader/package/functional/loader.py
156

Discussing this case in [1], in the end, we could simply add a SnapshotBranch which targets a TargetType Content.
No need to create a Release object.

[1] https://forge.softwareheritage.org/D3745?id=13193#inline-25905