I used ngrok to forward webhook requests to my local machine.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Oct 25 2022
I started looking how to implement that task.
Looks good to me, I guess you need to update the loader too.
LGTM, one thought posted as inline comment.
Rebase
Oct 24 2022
Rebase and address @vlorentz comments
Oct 21 2022
Looks good to me , just noticed some remaining typos: one in code and another in commit message (s/sctructure/structure).
Looks good to me, thanks !
Oct 20 2022
Add missing test
@Alphare, fyi I improved the rubygems lister in that commit in order to gather all artifacts related to a gem and send these info to the loader as extra arguments.
Below is an example of the lister output for a gem:
ListedOrigin( url="https://rubygems.org/gems/haar_joke", visit_type="rubygems", last_update=iso8601.parse_date("2016-11-05T00:00:00+00:00"), extra_loader_arguments={ "artifacts": [ { "url": "https://rubygems.org/downloads/haar_joke-0.0.2.gem", "length": 8704, "version": "0.0.2", "filename": "haar_joke-0.0.2.gem", "checksums": { "sha256": "85a8cf5f41890e9605265eeebfe9e99aa0350a01a3c799f9f55a0615a31a2f5f" }, }, { "url": "https://rubygems.org/downloads/haar_joke-0.0.1.gem", "length": 8704, "version": "0.0.1", "filename": "haar_joke-0.0.1.gem", "checksums": { "sha256": "a2ee7052fb8ffcfc4ec0fdb77fae9a36e473f859af196a36870a0f386b5ab55e" }, }, ], "rubygem_metadata": [ { "date": "2016-11-05T00:00:00+00:00", "authors": "Gemma Gotch", "version": "0.0.2", "extrinsic_metadata_url": "https://rubygems.org/api/v2/rubygems/haar_joke/versions/0.0.2.json", }, { "date": "2016-07-23T00:00:00+00:00", "authors": "Gemma Gotch", "version": "0.0.1", "extrinsic_metadata_url": "https://rubygems.org/api/v2/rubygems/haar_joke/versions/0.0.1.json", }, ], }, }
It enables to improve the scheduling of loading tasks for Ruby gems (by providing last_update value to ListedOrigin)
and it will save you of couple of calls to RubyGems Web API in the loader to fetch the list of versions for a gem.
So loader implementation must be adapted to use these new arguments.
Oct 19 2022
Apart the md5 decoding issue, loader looks pretty stable when testing it in docker.
Looks good to me, just intrinsic metadata parsing code to improve for errors handling and one test file to update after recent changes in swh-loader-core, see inline comments.
Jenkins jobs for swh packages have been updated to only fetch the master branch of repositories, closing this.
Rebase and improve commit message
Rebase
Rebase
Oct 18 2022
Looks good to me.