@samplet wow! that's pretty cool! Thank you ;)

I'm currently using the following regex to filter the exposed urls .tar.gz$|.zip$|tar.bz2$|.tbz$|.tar.xz$|.tgz$|.tar$ but I'm pretty sure it could be improved.

@zack

We need a name for this origin type, one of the hardest problem in CS :-)

Where is it used? Is it a new attribute?
We actually had to choose a name for the visit type, and with a lot of inspiration, we choose nixguix :-/

@zimoun

Do you mean filter the unsupported urls for the field "urls" in the "type": "url"?
Or do you mean only export "type": "url" and remove all the other types from 'sources.json', for instance "git"?

The nixguix loader is working well since 2 weeks on the nixpkgs sources.json file!
So, we can now consider the sources.json file format as stable and you could make the required changes on your sources.json file. A new SHW origin should then be added.

Fix ardumont comment and rebase

I'm wondering if we should not catch the exception on a missing metadatas instead. It is much more simple and I think we could have more branches like the evaluation one in the future. If we add git sources, the git loader will be in charge of loading the source and the nixguix loader will still add a pending branch to the snapshot.

What's the status of this patch series? Would be great to deploy it. :-)

Rebased

Rebased

Fix flake8

Addressed @ardumont comments. Thanks for your review!

Thanks @zack @ardumont. I addressed your comments.

@olasd I addressed all of your comments. Thanks for your review.

Thanks!

Remove the second loader instanciation in test_loader_incremental.

@olasd Another drawback is I would need to make one API call per release in order to list all integrity attributes of a snapshot. Even with a cache on the client side, the first run would require ~15000 calls to the SWH API. Do you have a rate limiting on these API endpoints?

Reduce commit diff :/

No longer use the integrity as branch name but the url instead.

Thanks for your review. I'm applying your suggestion which looks nice.

• nixguix: rename the test file

I don't know if nix_guix would be better than nixguix for the module name!

UPdate cmomit message to only mention the test

Rebased

What about this diff? I think it's still valuable since it catch more precisely errors and add a test case;)
WDYT?

Fix a typo

I added a test for the EOFError.

I actually succeeded with a bigger truncated file actually: 70KB

In D2859#68753, @vlorentz wrote:

Did you try truncating a .tar file to reproduce the issue?

@ardumont I don't understand why the storage doesn't contains 2 contents (a tarball + a file). The test test_loader_one_visit is failing because there is only 1 content in the storage.

• WIP

Rebased on master

• functional: create a branch named evaluation pointing to the evaluation commit

• package.loader: add extra_branches method
• functional: create a branch named evaluation pointing to the evaluation commit

@olasd Thanks for your comments. I addressed all of them.

• Add the functional loader

I'm not so sure about refactoring the base package loader to have two url and origin arguments that are the same most of the time but not all the time; In concrete terms, this means that we should refactor all loaders to use self.origin in the places where they currently use self.url.

For instance, we have the same issue with the pypi loader, where we generate an api url (which we're downloading) from the url of the origin (which is just a user-facing link).

All in all, I think we need to think about doing a refactoring to do this consistently, but I don't think this diff is the right place to do this refactoring; For now, we can just have a hardcoded map from origin url to index url.

• package.loader: add extra_branches method
• functional: create a branch named evaluation pointing to the evaluation commit

@ardumont I squashed almost all commits and added a task test.

• Add the functional loader

I think I need to implement a test for the task also. WDYT?

• package.loader: add extra_branches method
• functional: create a branch named evaluation pointing to the evaluation commit

• package.loader: ignore non tarball source
• package.loader: add origin argument
• Add the functional loader
• cli: add the functional loader in the cli tests
• functional: add test for non json sources files
• functional: improve test_loader_two_visits
• functional: add test_loader_incremental
• functional: move retrieve_sources out of the FunctionalLoader class
• functional: test the origin_visit
• functional: add test_uncompress_failure
• functional: minor fixes in tests

• package.loader: add extra_branches method
• functional: create a branch named evaluation pointing to the evaluation commit

• package.loader: add origin argument
• Add the functional loader
• cli: add the functional loader in the cli tests
• functional: add test for non json sources files
• functional: improve test_loader_two_visits
• functional: add test_loader_incremental
• functional: move retrieve_sources out of the FunctionalLoader class
• functional: test the origin_visit
• functional: add test_uncompress_failure
• functional: minor fixes in tests

So here, i'm missing:

• loading scenario which test the incremental nature of the visit (that'd test resolve_revision_from)

• functional: improve test_loader_two_visits
• functional: add test_loader_incremental
• functional: move retrieve_sources out of the FunctionalLoader class
• functional: test the origin_visit
• functional: add test_uncompress_failure

@civodul @zimoun I'm wondering if you generate a sources.json file for any commit of your guix repository or only for those that have been evaluated and pushed to your binary cache by your CI?

I actually don't know how should I manage errors. It seems exception are generally just emitted, without a dedicated log message.
How do you catch errors on production?

• functional: add test for non json sources files

Try to base on functional-loader branch

• functional: create a branch named evaluation pointing to the evaluation commit

• functional: create a branch named evaluation pointing to the evaluation commit

• package.loader: ignore non tarball source
• package.loader: add origin argument
• Add the functional loader
• cli: add the functional loader in the cli tests
• package.loader: add hook_branches method
• functional: create a branch named evaluation pointing to the evaluation commit