Page MenuHomeSoftware Heritage
Feed Advanced Search

Jan 8 2023

gitlab-migration changed the status of T735: SourceForge lister from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 9:57 PM · Origin-SourceForge

Oct 19 2022

gitlab-migration changed the status of T3470: lister-sourceforge: Activate sourceforge origins when listed, a subtask of T3374: Ingest sourceforge repositories (origins of type git, svn, hg), from Resolved to Migrated.
Oct 19 2022, 6:03 PM · System administration, Archive coverage, Origin-SourceForge
gitlab-migration changed the status of T3470: lister-sourceforge: Activate sourceforge origins when listed from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:03 PM · System administration, Archive coverage, Origin-SourceForge
gitlab-migration changed the status of T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:02 PM · System administration, Archive coverage, Origin-SourceForge
gitlab-migration changed the status of T3374: Ingest sourceforge repositories (origins of type git, svn, hg), a subtask of T735: SourceForge lister, from Resolved to Migrated.
Oct 19 2022, 6:02 PM · Origin-SourceForge
gitlab-migration changed the status of T3350: Deploy sourceforge lister in production, a subtask of T735: SourceForge lister, from Resolved to Migrated.
Oct 19 2022, 6:02 PM · Origin-SourceForge
gitlab-migration changed the status of T3350: Deploy sourceforge lister in production from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:02 PM · Archive coverage, System administration, Origin-SourceForge
gitlab-migration changed the status of T3310: Deploy sourceforge lister on staging from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:02 PM · System administration, Origin-SourceForge
gitlab-migration changed the status of T3310: Deploy sourceforge lister on staging, a subtask of T735: SourceForge lister, from Resolved to Migrated.
Oct 19 2022, 6:02 PM · Origin-SourceForge

Nov 18 2021

ardumont moved T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from code-review/await-feedback/pause to done on the System administration board.
Nov 18 2021, 3:17 PM · System administration, Archive coverage, Origin-SourceForge

Oct 15 2021

ardumont closed T735: SourceForge lister as Resolved.
Oct 15 2021, 9:44 AM · Origin-SourceForge
ardumont closed T3470: lister-sourceforge: Activate sourceforge origins when listed as Resolved.
Oct 15 2021, 9:44 AM · System administration, Archive coverage, Origin-SourceForge
ardumont closed T3470: lister-sourceforge: Activate sourceforge origins when listed, a subtask of T3374: Ingest sourceforge repositories (origins of type git, svn, hg), as Resolved.
Oct 15 2021, 9:44 AM · System administration, Archive coverage, Origin-SourceForge
ardumont moved T3470: lister-sourceforge: Activate sourceforge origins when listed from code-review/await-feedback/pause to deployed/landed/monitoring on the System administration board.
Oct 15 2021, 9:43 AM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3470: lister-sourceforge: Activate sourceforge origins when listed.

Sourceforge origins are progressively activated back and getting ingested along the way [1].

Oct 15 2021, 9:43 AM · System administration, Archive coverage, Origin-SourceForge

Oct 14 2021

ardumont moved T3470: lister-sourceforge: Activate sourceforge origins when listed from in-progress to code-review/await-feedback/pause on the System administration board.
Oct 14 2021, 6:09 PM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3470: lister-sourceforge: Activate sourceforge origins when listed.

Lister got adapted to actually activate the sourceforge origins.
That lister got deployed.

Oct 14 2021, 6:09 PM · System administration, Archive coverage, Origin-SourceForge
ardumont changed the status of T3470: lister-sourceforge: Activate sourceforge origins when listed from Open to Work in Progress.
Oct 14 2021, 3:00 PM · System administration, Archive coverage, Origin-SourceForge
ardumont changed the status of T3470: lister-sourceforge: Activate sourceforge origins when listed, a subtask of T3374: Ingest sourceforge repositories (origins of type git, svn, hg), from Open to Work in Progress.
Oct 14 2021, 3:00 PM · System administration, Archive coverage, Origin-SourceForge

Oct 11 2021

ardumont added a revision to T3470: lister-sourceforge: Activate sourceforge origins when listed: D6450: Let sourceforge origins be enabled by default.
Oct 11 2021, 1:06 PM · System administration, Archive coverage, Origin-SourceForge

Sep 28 2021

ardumont closed T3374: Ingest sourceforge repositories (origins of type git, svn, hg), a subtask of T735: SourceForge lister, as Resolved.
Sep 28 2021, 5:55 PM · Origin-SourceForge
ardumont closed T3374: Ingest sourceforge repositories (origins of type git, svn, hg) as Resolved.

I've triggered back run for mercurial and git origins (it's done).
So it should now have kept up with the eventual lags.
Those are regularly crawled.

Sep 28 2021, 5:55 PM · System administration, Archive coverage, Origin-SourceForge
ardumont moved T3470: lister-sourceforge: Activate sourceforge origins when listed from Backlog to Weekly backlog on the System administration board.
Sep 28 2021, 11:47 AM · System administration, Archive coverage, Origin-SourceForge

Sep 23 2021

ardumont added a comment to T3374: Ingest sourceforge repositories (origins of type git, svn, hg).

Heads up on this task, i'm actually waiting for the bitbucket ingestion (which is going faster now) to finish.
To re-use our worker17 to make one last run on all the mercurial origins.

Sep 23 2021, 9:19 AM · System administration, Archive coverage, Origin-SourceForge

Aug 6 2021

ardumont triaged T3470: lister-sourceforge: Activate sourceforge origins when listed as High priority.
Aug 6 2021, 4:09 PM · System administration, Archive coverage, Origin-SourceForge

Aug 3 2021

ardumont added a revision to T3374: Ingest sourceforge repositories (origins of type git, svn, hg): D6051: changelog: Reference first completion of sourceforge hg origins.
Aug 3 2021, 11:43 AM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Aug 3 2021, 10:17 AM · System administration, Archive coverage, Origin-SourceForge

Jul 30 2021

ardumont moved T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from in-progress to code-review/await-feedback/pause on the System administration board.
Jul 30 2021, 11:22 AM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 30 2021, 10:25 AM · System administration, Archive coverage, Origin-SourceForge

Jul 29 2021

ardumont moved T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from Backlog to in-progress on the System administration board.
Jul 29 2021, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont added a project to T3374: Ingest sourceforge repositories (origins of type git, svn, hg): System administration.
Jul 29 2021, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont renamed T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from Ingest sourceforge repositories to Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 29 2021, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3374: Ingest sourceforge repositories (origins of type git, svn, hg).

The 'hg' ingestion started now that the latest loader mercurial got deployed.

Jul 29 2021, 5:43 PM · System administration, Archive coverage, Origin-SourceForge
ardumont moved T3310: Deploy sourceforge lister on staging from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration, Origin-SourceForge
ardumont moved T3350: Deploy sourceforge lister in production from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · Archive coverage, System administration, Origin-SourceForge

Jul 22 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 22 2021, 10:15 AM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3374: Ingest sourceforge repositories (origins of type git, svn, hg).

The 'git' ingestion caught up [1] so now let's make the svn origins finish [2]. In
effect, making the loader run as before. Activating back the svn queue consumption where
it remains few origins to consume.

Jul 22 2021, 9:43 AM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 22 2021, 9:40 AM · System administration, Archive coverage, Origin-SourceForge

Jul 20 2021

ardumont added a comment to T3374: Ingest sourceforge repositories (origins of type git, svn, hg).

So, to improve the current situation, the svn ingestion got put in stand-by to let the
git ingestion progress as well.

Jul 20 2021, 10:09 AM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 20 2021, 9:56 AM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3374: Ingest sourceforge repositories (origins of type git, svn, hg).

So a bit of status report.

Jul 20 2021, 9:53 AM · System administration, Archive coverage, Origin-SourceForge

Jul 19 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 19 2021, 9:42 AM · System administration, Archive coverage, Origin-SourceForge

Jul 9 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 9 2021, 4:27 PM · System administration, Archive coverage, Origin-SourceForge

Jul 7 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 7 2021, 11:13 AM · System administration, Archive coverage, Origin-SourceForge

Jul 5 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 5 2021, 11:18 AM · System administration, Archive coverage, Origin-SourceForge

Jul 1 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 1 2021, 10:05 AM · System administration, Archive coverage, Origin-SourceForge
ardumont added a revision to T3374: Ingest sourceforge repositories (origins of type git, svn, hg): D5952: changelog: Reference first completion of sourceforge git/svn origins.
Jul 1 2021, 9:26 AM · System administration, Archive coverage, Origin-SourceForge

Jun 28 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 28 2021, 12:04 PM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 28 2021, 12:03 PM · System administration, Archive coverage, Origin-SourceForge

Jun 24 2021

ardumont changed the status of T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from Open to Work in Progress.
Jun 24 2021, 4:48 PM · System administration, Archive coverage, Origin-SourceForge
ardumont changed the status of T3374: Ingest sourceforge repositories (origins of type git, svn, hg), a subtask of T735: SourceForge lister, from Open to Work in Progress.
Jun 24 2021, 4:48 PM · Origin-SourceForge

Jun 22 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 22 2021, 11:54 AM · System administration, Archive coverage, Origin-SourceForge

Jun 17 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 17 2021, 10:28 AM · System administration, Archive coverage, Origin-SourceForge
ardumont claimed T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 17 2021, 10:24 AM · System administration, Archive coverage, Origin-SourceForge

Jun 14 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 14 2021, 11:29 AM · System administration, Archive coverage, Origin-SourceForge

Jun 11 2021

ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 11 2021, 12:39 PM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 11 2021, 12:37 PM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3374: Ingest sourceforge repositories (origins of type git, svn, hg).

Note: when this is (reasonably) done, we should document the addition of SourceForge to the archive coverage page at archive.s.o and also to the archive changelog.

Jun 11 2021, 12:31 PM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 11 2021, 12:31 PM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 11 2021, 12:29 PM · System administration, Archive coverage, Origin-SourceForge
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jun 11 2021, 12:28 PM · System administration, Archive coverage, Origin-SourceForge
zack added a project to T3374: Ingest sourceforge repositories (origins of type git, svn, hg): Archive coverage.

Note: when this is (reasonably) done, we should document the addition of SourceForge to the archive coverage page at archive.s.o and also to the archive changelog.

Jun 11 2021, 12:27 PM · System administration, Archive coverage, Origin-SourceForge
ardumont closed T3350: Deploy sourceforge lister in production, a subtask of T735: SourceForge lister, as Resolved.
Jun 11 2021, 12:24 PM · Origin-SourceForge
ardumont closed T3350: Deploy sourceforge lister in production as Resolved.
Jun 11 2021, 12:24 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a comment to T3350: Deploy sourceforge lister in production.

It's deployed and the ingestion is ongoing.
Monitoring of the ingestion will be moved to a dedicated task [1]
Closing this now.

Jun 11 2021, 12:23 PM · Archive coverage, System administration, Origin-SourceForge
ardumont triaged T3374: Ingest sourceforge repositories (origins of type git, svn, hg) as High priority.
Jun 11 2021, 12:22 PM · System administration, Archive coverage, Origin-SourceForge

Jun 8 2021

zack added a project to T3350: Deploy sourceforge lister in production: Archive coverage.
Jun 8 2021, 11:45 AM · Archive coverage, System administration, Origin-SourceForge
ardumont added a comment to T3350: Deploy sourceforge lister in production.

Still running, both svn and git svn origins are ingested regularly.

Jun 8 2021, 11:16 AM · Archive coverage, System administration, Origin-SourceForge

Jun 4 2021

ardumont added a comment to T3350: Deploy sourceforge lister in production.

Current status, counting only git and svn origins, 26.8% [1] got done in ~24h
(somewhat... [2]).

Jun 4 2021, 5:40 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a comment to T3350: Deploy sourceforge lister in production.

Heads up:

  • Concurrency bumped to 6 (for the loader).
  • Migration of the mercurial origins dataset from https to http (scheduler prod/staging in progress) [1]
  • Incremental lister deployed
Jun 4 2021, 4:56 PM · Archive coverage, System administration, Origin-SourceForge

Jun 3 2021

ardumont changed the status of T3350: Deploy sourceforge lister in production from Open to Work in Progress.
Jun 3 2021, 6:18 PM · Archive coverage, System administration, Origin-SourceForge
ardumont changed the status of T3350: Deploy sourceforge lister in production, a subtask of T735: SourceForge lister, from Open to Work in Progress.
Jun 3 2021, 6:18 PM · Origin-SourceForge
ardumont added a project to T3350: Deploy sourceforge lister in production: System administration.
Jun 3 2021, 6:18 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a comment to T3350: Deploy sourceforge lister in production.

This started in worker17 with the content of the diff ^:

Jun 3 2021, 6:17 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a revision to T3350: Deploy sourceforge lister in production: D5818: send-to-celery: Add more options to allow scheduling of edge case origins.
Jun 3 2021, 4:07 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a comment to T3350: Deploy sourceforge lister in production.

A small issue was found, @Alphare fixed it ^.
Notification to the sourceforge people about the ingestion starting soon got sent.

Jun 3 2021, 10:43 AM · Archive coverage, System administration, Origin-SourceForge
ardumont added a revision to T3350: Deploy sourceforge lister in production: D5817: sourceforge: set the protocol for origin urls.
Jun 3 2021, 10:11 AM · Archive coverage, System administration, Origin-SourceForge

Jun 2 2021

ardumont added a revision to T3350: Deploy sourceforge lister in production: D5814: Dedicate a loader_oneshot service for temporary use.
Jun 2 2021, 5:36 PM · Archive coverage, System administration, Origin-SourceForge
ardumont claimed T3350: Deploy sourceforge lister in production.
Jun 2 2021, 5:32 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a comment to T3350: Deploy sourceforge lister in production.

Dedicated worker17 node got provisionned to make the first run on the sourceforge
origins (svn and git for now). Remains some code to actually schedule the origins we are
interested. And some plumbing to actually consume those messages with respect to the
concurrency defined in the description.

Jun 2 2021, 9:55 AM · Archive coverage, System administration, Origin-SourceForge

Jun 1 2021

ardumont added a comment to T735: SourceForge lister.

(Last 2 comments was meant for T3350...)

Jun 1 2021, 10:26 AM · Origin-SourceForge
ardumont added a comment to T735: SourceForge lister.

New listed origins are enabled=f as expected so they they won't be selected just yet for ingestion.

Jun 1 2021, 10:15 AM · Origin-SourceForge
ardumont added a comment to T735: SourceForge lister.

Status:

  • Updated the lister sourceforge code so sourceforge origins (as disabled) can occur
  • Packaged and deployed the change
  • Added the task to the scheduler [1] so the listing occurs [2] [3]
Jun 1 2021, 10:07 AM · Origin-SourceForge

May 31 2021

ardumont updated the task description for T3350: Deploy sourceforge lister in production.
May 31 2021, 5:03 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a revision to T3350: Deploy sourceforge lister in production: D5803: Disable the sourceforge lister origins.
May 31 2021, 4:04 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a comment to T3350: Deploy sourceforge lister in production.

Sounds simpler.

May 31 2021, 3:55 PM · Archive coverage, System administration, Origin-SourceForge
olasd added a comment to T3350: Deploy sourceforge lister in production.

Another idea would be to add the SourceForge origins with enabled=false so they're not picked up by the scheduler, until we've done the first pass on them. This avoids needing to change the scheduler at all.

May 31 2021, 3:28 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a revision to T3350: Deploy sourceforge lister in production: D5802: backend: Allow lister exclusion from the grab_next_visit endpoint.
May 31 2021, 3:20 PM · Archive coverage, System administration, Origin-SourceForge
ardumont added a comment to T3350: Deploy sourceforge lister in production.

Currently the next gen scheduler does not allow to limit the number of tasks per forge.
So a plan forward would be to allow the listing but prevent the origins from getting
scheduled for ingestion from the actual scheduler cogs running. Then, trigger the
ingestion "manually" [1] with dedicated worker(s) which would consume specifically from
sourceforge and respecting the limits set in the description.

May 31 2021, 3:13 PM · Archive coverage, System administration, Origin-SourceForge
ardumont updated the task description for T3350: Deploy sourceforge lister in production.
May 31 2021, 10:56 AM · Archive coverage, System administration, Origin-SourceForge

May 28 2021

ardumont triaged T3350: Deploy sourceforge lister in production as High priority.
May 28 2021, 11:37 AM · Archive coverage, System administration, Origin-SourceForge
ardumont closed T3310: Deploy sourceforge lister on staging, a subtask of T735: SourceForge lister, as Resolved.
May 28 2021, 11:12 AM · Origin-SourceForge
ardumont closed T3310: Deploy sourceforge lister on staging as Resolved.

roh, you know what i meant forge, not claim the task, resolve it... (anyway, closing)

May 28 2021, 11:12 AM · System administration, Origin-SourceForge
ardumont claimed T3310: Deploy sourceforge lister on staging.

I guess it's all fine now.
Remains to deploy this in production at some point.

May 28 2021, 11:11 AM · System administration, Origin-SourceForge
ardumont added a comment to T3310: Deploy sourceforge lister on staging.

Everything went fine as well:

May 28 2021, 11:10 AM · System administration, Origin-SourceForge

May 27 2021

ardumont added a comment to T3310: Deploy sourceforge lister on staging.

Installed and triggered a run for the incremental task on staging:

May 27 2021, 5:11 PM · System administration, Origin-SourceForge
ardumont added a comment to T3310: Deploy sourceforge lister on staging.

It went through \o/:

May 27 2021, 5:02 PM · System administration, Origin-SourceForge

May 26 2021

ardumont added a comment to T3310: Deploy sourceforge lister on staging.

New lister deployed:

ii  python3-swh.lister 1.3.2-1~swh1~bpo10+1 all          Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)
May 26 2021, 12:58 PM · System administration, Origin-SourceForge
ardumont added a revision to T3310: Deploy sourceforge lister on staging: D5785: sourceforge: retry for all retryable exceptions.
May 26 2021, 12:16 PM · System administration, Origin-SourceForge
Alphare added a comment to T3310: Deploy sourceforge lister on staging.

(Hopefully) Addressed in D5785

May 26 2021, 12:15 PM · System administration, Origin-SourceForge