- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Jan 8 2023
Oct 19 2022
Nov 18 2021
Oct 15 2021
Sourceforge origins are progressively activated back and getting ingested along the way [1].
Oct 14 2021
Lister got adapted to actually activate the sourceforge origins.
That lister got deployed.
Oct 11 2021
Sep 28 2021
I've triggered back run for mercurial and git origins (it's done).
So it should now have kept up with the eventual lags.
Those are regularly crawled.
Sep 23 2021
Heads up on this task, i'm actually waiting for the bitbucket ingestion (which is going faster now) to finish.
To re-use our worker17 to make one last run on all the mercurial origins.
Aug 6 2021
Aug 3 2021
Jul 30 2021
Jul 29 2021
The 'hg' ingestion started now that the latest loader mercurial got deployed.
Jul 22 2021
The 'git' ingestion caught up [1] so now let's make the svn origins finish [2]. In
effect, making the loader run as before. Activating back the svn queue consumption where
it remains few origins to consume.
Jul 20 2021
So, to improve the current situation, the svn ingestion got put in stand-by to let the
git ingestion progress as well.
So a bit of status report.
Jul 19 2021
Jul 9 2021
Jul 7 2021
Jul 5 2021
Jul 1 2021
Jun 28 2021
Jun 24 2021
Jun 22 2021
Jun 17 2021
Jun 14 2021
Jun 11 2021
Note: when this is (reasonably) done, we should document the addition of SourceForge to the archive coverage page at archive.s.o and also to the archive changelog.
Note: when this is (reasonably) done, we should document the addition of SourceForge to the archive coverage page at archive.s.o and also to the archive changelog.
It's deployed and the ingestion is ongoing.
Monitoring of the ingestion will be moved to a dedicated task [1]
Closing this now.
Jun 8 2021
Still running, both svn and git svn origins are ingested regularly.
Jun 4 2021
Current status, counting only git and svn origins, 26.8% [1] got done in ~24h
(somewhat... [2]).
Heads up:
- Concurrency bumped to 6 (for the loader).
- Migration of the mercurial origins dataset from https to http (scheduler prod/staging in progress) [1]
- Incremental lister deployed
Jun 3 2021
This started in worker17 with the content of the diff ^:
A small issue was found, @Alphare fixed it ^.
Notification to the sourceforge people about the ingestion starting soon got sent.
Jun 2 2021
Dedicated worker17 node got provisionned to make the first run on the sourceforge
origins (svn and git for now). Remains some code to actually schedule the origins we are
interested. And some plumbing to actually consume those messages with respect to the
concurrency defined in the description.
Jun 1 2021
(Last 2 comments was meant for T3350...)
New listed origins are enabled=f as expected so they they won't be selected just yet for ingestion.
Status:
- Updated the lister sourceforge code so sourceforge origins (as disabled) can occur
- Packaged and deployed the change
- Added the task to the scheduler [1] so the listing occurs [2] [3]
May 31 2021
Sounds simpler.
Another idea would be to add the SourceForge origins with enabled=false so they're not picked up by the scheduler, until we've done the first pass on them. This avoids needing to change the scheduler at all.
Currently the next gen scheduler does not allow to limit the number of tasks per forge.
So a plan forward would be to allow the listing but prevent the origins from getting
scheduled for ingestion from the actual scheduler cogs running. Then, trigger the
ingestion "manually" [1] with dedicated worker(s) which would consume specifically from
sourceforge and respecting the limits set in the description.
May 28 2021
roh, you know what i meant forge, not claim the task, resolve it... (anyway, closing)
I guess it's all fine now.
Remains to deploy this in production at some point.
Everything went fine as well:
May 27 2021
Installed and triggered a run for the incremental task on staging:
It went through \o/:
May 26 2021
New lister deployed:
ii python3-swh.lister 1.3.2-1~swh1~bpo10+1 all Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)
(Hopefully) Addressed in D5785