Tue, Nov 27
Oct 22 2018
Oct 18 2018
Ok, so reworked the group_by_exception snippet to have a more sensible output:
Oct 17 2018
In any case, for now, like i said in , we will first schedule back
those 1409 origins in error.
Oct 16 2018
Here is the pypi report about the loading errors.
Oct 5 2018
kibana dashboard will help in that matters (P311 because it's noisy).
Sep 21 2018
Sep 20 2018
Now, it's scheduled. Just need to wait for the swh-scheduler-runner.service to finish its loop on task_types.
swhscheduler@saatchi:~$ python3 -m swh.scheduler.cli task list-pending -t swh-lister-pypi Found 1 tasks
Schedule the lister-pypi:
Sep 19 2018
Sep 6 2018
Sep 4 2018
Aug 27 2018
Then again, i'll check the pypi api's documentation. Hopefully, it's explained somewhere ;)
Aug 23 2018
So, having one branch in the snapshot per distribution format (tar/zip/etc.) is a nice and clean way of handling this.
Aug 22 2018
The Debian loader doesn't create release objects. Our data model doesn't allow to attach arbitrary structured metadata to release objects (as Git doesn't either), so we've shortcut this level of indirection.
Aug 21 2018
There remains 3 actions to do for the current implementation to be complete:
Aug 2 2018
As far as I can tell from those examples, the metadata that PyPI gives you are the most recent ones, probably the ones extracted from the most recent version, so it would be incorrect to associate them to other releases.
For (1), I think what we currently do for Debian packages is as you said, i.e., snapshot -> release -> revision -> tarball root dir. Maybe you can check for comparison (or @olasd can chime in?). We should do the same here.
The basic loader will be the tarball loader, yes. In addition to that there are two aspects to be defined:
- the stack of objects to be added to the DAG
- the metadata to extract
Aug 1 2018
capable of extracting upstream metadata that are meaningful (and specific to) PyPI.
Jul 26 2018
As per comment , closing this as we will not implement mirroring.
the out-of-dated-ness is most likely due to the fact that, since I opened this task 2 years ago...
Jul 25 2018
better LWN link to the actual article covering this: https://lwn.net/Articles/751458/
Jul 24 2018
They have multiple apis:
- basic json one  which permits to request information on a per project basis (no listing)  (~> foresee the use of this one for the loader)
- xmlrpc deprecated one  (this one lists ~> that would be for the lister use)
- html page (listing all packages)
- rss feed (update events)