Page MenuHomeSoftware Heritage
Feed Advanced Search

Sep 8 2022

ardumont closed T4330: Deploy maven stack in production, a subtask of T1724: Maven Central repository support, as Resolved.
Sep 8 2022, 3:03 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont closed T4330: Deploy maven stack in production as Resolved.
Sep 8 2022, 3:03 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
vsellier added a comment to T4330: Deploy maven stack in production.

\o/ great

Sep 8 2022, 2:41 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont moved T4330: Deploy maven stack in production from in-progress to deployed/landed/monitoring on the System administration board.
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Checks:

  • task has been scheduled by the scheduler runner process [1]
  • listing is being consumed by one worker [2]
  • 'maven' listed origins is steadily growing [3]
  • New 'maven' listed origins are getting scheduled for ingestion [4]
  • maven loaders are ingesting those [5]
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:11 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a revision to T4330: Deploy maven stack in production: D8421: production: Update lister to consume maven listing.
Sep 8 2022, 2:11 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:06 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Schedule maven-central listing:

swhscheduler@saatchi:~$ curl -s https://repo1.maven.org/maven2/ | head -2
<!DOCTYPE html>
<html>
swhscheduler@saatchi:~$ curl -s https://maven-exporter.internal.softwareheritage.org/export-maven-central.fld | head -2
doc 0
  field 0
swhscheduler@saatchi:~$ curl -s http://saatchi.internal.softwareheritage.org:5008/
<html>
<head><title>Software Heritage scheduler RPC server</title></head>
<body>
<p>You have reached the
<a href="https://www.softwareheritage.org/">Software Heritage</a>
scheduler RPC server.<br />
See its
<a href="https://docs.softwareheritage.org/devel/swh-scheduler/">documentation
and API</a> for more information</p>
</body>
</html>swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   task add list-maven-full \
>     url=https://repo1.maven.org/maven2/ \
>     index_url=https://maven-exporter.internal.softwareheritage.org/export-maven-central.fld
Created 1 tasks
Sep 8 2022, 2:04 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 1:47 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Finally, export is done on maven central [1], the fld is computed [2]...
And it's also exposed, hence reachable from lister worker nodes.

Sep 8 2022, 1:47 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont changed the status of T4330: Deploy maven stack in production, a subtask of T1724: Maven Central repository support, from Open to Work in Progress.
Sep 8 2022, 11:18 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont changed the status of T4330: Deploy maven stack in production from Open to Work in Progress.
Sep 8 2022, 11:18 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a project to T4330: Deploy maven stack in production: System administration.
Sep 8 2022, 11:18 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:30 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:19 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:05 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 9:57 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Sep 7 2022

ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 7 2022, 6:27 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Sep 6 2022

anlambert added a project to T4423: Gogs pagination API breaks because of fatal repos: Gogs lister.
Sep 6 2022, 4:48 PM · Gogs lister, Origin-Gitea/Gogs, Lister, Archive coverage
anlambert added a project to T1721: Implementation of Gogs Lister: Gogs lister.
Sep 6 2022, 4:48 PM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister
anlambert added projects to T4124: Golang support: Golang lister, Golang loader.
Sep 6 2022, 4:35 PM · Golang loader, Golang lister, Archive coverage
olasd added a project to T4494: ingest hackage.haskell.org, the Haskell package repository : Archive coverage.
Sep 6 2022, 4:24 PM · Hackage loader, Hackage lister, Archive coverage
anlambert added a project to T4475: Ingest bower.io (Javascript package manager): Bower lister.
Sep 6 2022, 4:17 PM · Bower lister, Archive coverage
anlambert added projects to T4465: Ingest pub.dev (Dart, Flutter): PubDev lister, PubDev loader.
Sep 6 2022, 3:56 PM · PubDev loader, PubDev lister, Archive coverage
anlambert added projects to T4466: Ingest Aur repository (Arch User Repository): AUR lister, AUR loader.
Sep 6 2022, 3:12 PM · AUR loader, AUR lister, Archive coverage
anlambert added projects to T4233: Ingest Arch Linux: Arch Lister, Arch loader.
Sep 6 2022, 3:12 PM · Arch loader, Arch Lister, Archive coverage
ardumont added a revision to T4330: Deploy maven stack in production: D8399: production: Provision new maven-exporter node.
Sep 6 2022, 2:56 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a revision to T4330: Deploy maven stack in production: D8397: Deploy maven-exporter production node.
Sep 6 2022, 11:50 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T3691: Implement CVS loader.
Sep 6 2022, 11:27 AM · CVS loader, Archive coverage

Sep 5 2022

franckbret updated the task description for T4466: Ingest Aur repository (Arch User Repository).
Sep 5 2022, 9:50 AM · AUR loader, AUR lister, Archive coverage
franckbret updated the task description for T4466: Ingest Aur repository (Arch User Repository).
Sep 5 2022, 9:50 AM · AUR loader, AUR lister, Archive coverage

Sep 2 2022

bchauvet added a parent task for T2833: cpan.loader - archive Perl modules from CPAN: Unknown Object (Maniphest Task).
Sep 2 2022, 11:00 AM · CPAN lister, Archive coverage
bchauvet added a parent task for T1777: Rubygems Lister: Unknown Object (Maniphest Task).
Sep 2 2022, 10:59 AM · RubyGems lister, Archive coverage
bchauvet added a parent task for T1002: ingest Hackage, the Haskell package repository (meta task): Unknown Object (Maniphest Task).
Sep 2 2022, 10:59 AM · Hackage loader, Hackage lister, Archive coverage
bchauvet added a parent task for T1776: packagist (PHP) Lister: Unknown Object (Maniphest Task).
Sep 2 2022, 10:52 AM · Lister, Archive coverage
bchauvet added a parent task for T4475: Ingest bower.io (Javascript package manager): Unknown Object (Maniphest Task).
Sep 2 2022, 10:50 AM · Bower lister, Archive coverage
bchauvet added a parent task for T4124: Golang support: Unknown Object (Maniphest Task).
Sep 2 2022, 10:48 AM · Golang loader, Golang lister, Archive coverage
bchauvet added a parent task for T4465: Ingest pub.dev (Dart, Flutter): Unknown Object (Maniphest Task).
Sep 2 2022, 10:45 AM · PubDev loader, PubDev lister, Archive coverage

Sep 1 2022

KShivendu added a subtask for T1721: Implementation of Gogs Lister: T4478: Deploy Gogs lister to staging.
Sep 1 2022, 10:03 AM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister

Aug 31 2022

anlambert added a revision to T4465: Ingest pub.dev (Dart, Flutter): D8355: pubdev: Modify loader after origin URL modification in related lister.
Aug 31 2022, 11:38 AM · PubDev loader, PubDev lister, Archive coverage
anlambert added a revision to T4465: Ingest pub.dev (Dart, Flutter): D8354: pubdev: Modify origin URL and retrieve package last update.
Aug 31 2022, 11:38 AM · PubDev loader, PubDev lister, Archive coverage

Aug 30 2022

franckbret added a comment to T4466: Ingest Aur repository (Arch User Repository).

I've made a complete run on docker

Aug 30 2022, 1:50 PM · AUR loader, AUR lister, Archive coverage
ardumont triaged T4475: Ingest bower.io (Javascript package manager) as Normal priority.
Aug 30 2022, 11:10 AM · Bower lister, Archive coverage
franckbret added a comment to T4233: Ingest Arch Linux.

Arch Linux Lister Docker Report

The lister takes a lot of time and fail on max retries when scraping repository directory (It has run fine a few weeks ago.). Not sure at this point, but I suspect that's a random problem related to network / http server. WIll run it multiple time to see if it failed on the same resource.

By the way I guess that the we need to define a strategy for those exceptions.

Aug 30 2022, 10:28 AM · Arch loader, Arch Lister, Archive coverage
franckbret updated the task description for T4233: Ingest Arch Linux.
Aug 30 2022, 10:14 AM · Arch loader, Arch Lister, Archive coverage
franckbret updated the task description for T4233: Ingest Arch Linux.
Aug 30 2022, 10:06 AM · Arch loader, Arch Lister, Archive coverage
franckbret updated the task description for T4465: Ingest pub.dev (Dart, Flutter).
Aug 30 2022, 8:14 AM · PubDev loader, PubDev lister, Archive coverage
franckbret added a comment to T4465: Ingest pub.dev (Dart, Flutter).

Loader runs on Docker report

Loader runs fine on docker.
I've first launched 100 and then 1900 loader tasks, it complete in less than an hour.

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful';
 count 
-------
  1450

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed';
 count 
-------
   550
Aug 30 2022, 8:14 AM · PubDev loader, PubDev lister, Archive coverage

Aug 29 2022

franckbret updated the task description for T4465: Ingest pub.dev (Dart, Flutter).
Aug 29 2022, 12:19 PM · PubDev loader, PubDev lister, Archive coverage

Aug 26 2022

vlorentz triaged T4466: Ingest Aur repository (Arch User Repository) as Normal priority.
Aug 26 2022, 5:36 PM · AUR loader, AUR lister, Archive coverage
vlorentz triaged T4465: Ingest pub.dev (Dart, Flutter) as Normal priority.
Aug 26 2022, 5:36 PM · PubDev loader, PubDev lister, Archive coverage
franckbret updated the task description for T4233: Ingest Arch Linux.
Aug 26 2022, 5:21 PM · Arch loader, Arch Lister, Archive coverage
franckbret updated the task description for T4233: Ingest Arch Linux.
Aug 26 2022, 5:20 PM · Arch loader, Arch Lister, Archive coverage
franckbret added a comment to T4233: Ingest Arch Linux.

Arch Linux Lister Docker Report

Aug 26 2022, 5:11 PM · Arch loader, Arch Lister, Archive coverage
franckbret added a comment to T4466: Ingest Aur repository (Arch User Repository).

Aur Loader runs in Docker report

Aug 26 2022, 4:57 PM · AUR loader, AUR lister, Archive coverage
franckbret updated the task description for T4466: Ingest Aur repository (Arch User Repository).
Aug 26 2022, 4:34 PM · AUR loader, AUR lister, Archive coverage
franckbret added a comment to T4466: Ingest Aur repository (Arch User Repository).

AUR Lister runs in Docker report

Aug 26 2022, 4:33 PM · AUR loader, AUR lister, Archive coverage
franckbret created T4466: Ingest Aur repository (Arch User Repository).
Aug 26 2022, 4:30 PM · AUR loader, AUR lister, Archive coverage
franckbret updated the task description for T4465: Ingest pub.dev (Dart, Flutter).
Aug 26 2022, 4:10 PM · PubDev loader, PubDev lister, Archive coverage
franckbret updated the task description for T4465: Ingest pub.dev (Dart, Flutter).
Aug 26 2022, 4:06 PM · PubDev loader, PubDev lister, Archive coverage
franckbret updated the task description for T4465: Ingest pub.dev (Dart, Flutter).
Aug 26 2022, 4:05 PM · PubDev loader, PubDev lister, Archive coverage
franckbret updated the task description for T4465: Ingest pub.dev (Dart, Flutter).
Aug 26 2022, 4:04 PM · PubDev loader, PubDev lister, Archive coverage
franckbret added a comment to T4465: Ingest pub.dev (Dart, Flutter).

Loader runs on Docker report

Aug 26 2022, 4:03 PM · PubDev loader, PubDev lister, Archive coverage
franckbret added a comment to T4465: Ingest pub.dev (Dart, Flutter).

Lister runs on Docker report

Aug 26 2022, 3:35 PM · PubDev loader, PubDev lister, Archive coverage
franckbret created T4465: Ingest pub.dev (Dart, Flutter).
Aug 26 2022, 3:31 PM · PubDev loader, PubDev lister, Archive coverage

Aug 22 2022

vlorentz added a project to T2645: Add listing tasks for gitea instances: Origin-Gitea/Gogs.
Aug 22 2022, 10:46 AM · Origin-Gitea/Gogs, Archive coverage, Lister
vlorentz added a project to T4423: Gogs pagination API breaks because of fatal repos: Origin-Gitea/Gogs.
Aug 22 2022, 10:45 AM · Gogs lister, Origin-Gitea/Gogs, Lister, Archive coverage
vlorentz added a project to T1721: Implementation of Gogs Lister: Origin-Gitea/Gogs.
Aug 22 2022, 10:45 AM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister

Aug 20 2022

KShivendu updated the task description for T4448: Implementation of Fedora Lister.
Aug 20 2022, 10:17 AM · RPM lister, Archive coverage
KShivendu updated the task description for T4448: Implementation of Fedora Lister.
Aug 20 2022, 10:11 AM · RPM lister, Archive coverage
KShivendu triaged T4448: Implementation of Fedora Lister as Normal priority.
Aug 20 2022, 9:58 AM · RPM lister, Archive coverage

Aug 18 2022

bchauvet edited projects for T4233: Ingest Arch Linux, added: Archive coverage; removed Unknown Object (Project).
Aug 18 2022, 2:17 PM · Arch loader, Arch Lister, Archive coverage

Aug 17 2022

KShivendu updated the task description for T1721: Implementation of Gogs Lister.
Aug 17 2022, 11:38 AM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister

Aug 11 2022

vlorentz closed T4421: Prioritize archival from gitlab.com as Invalid.
Aug 11 2022, 1:57 PM · Archive coverage, Origin-GitLab
vlorentz triaged T4411: Archive https://rsync.libreboot.org/ as Normal priority.
Aug 11 2022, 1:56 PM · Archive coverage

Aug 9 2022

KShivendu updated the task description for T1721: Implementation of Gogs Lister.
Aug 9 2022, 8:16 AM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister
KShivendu added a revision to T1721: Implementation of Gogs Lister: D8218: feat: Make the Gogs lister incremental.
Aug 9 2022, 7:45 AM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister

Aug 8 2022

zack added a comment to T4346: Create SourceHut Lister.

This has now been discussed on the sourcehut mailing list and I took part in the conversation.

Aug 8 2022, 11:14 AM · Archive coverage, Lister

Aug 5 2022

ardumont updated the task description for T4104: Ingest crates.io (Rust).
Aug 5 2022, 1:45 PM · Crates loader, Crates lister, Archive coverage
vlorentz added a comment to T4421: Prioritize archival from gitlab.com.

We discussed internally what to do with inactive repositories.
We reached a decision to move unused repos to object storage.
Once implemented, they will still be accessible but take a bit longer to access after a long period of inactivity.

Aug 5 2022, 8:41 AM · Archive coverage, Origin-GitLab

Aug 4 2022

olasd added a comment to T4421: Prioritize archival from gitlab.com.

Looks like there's many more repos that should be visitable but aren't:

Aug 4 2022, 4:51 PM · Archive coverage, Origin-GitLab
ardumont added a comment to T1721: Implementation of Gogs Lister.

worth opening a dedicated forge issue

Done. T4423

open an upstream issue

Done. https://github.com/gogs/gogs/issues/7124

Aug 4 2022, 4:02 PM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister
ardumont raised the priority of T1721: Implementation of Gogs Lister from Low to Normal.
Aug 4 2022, 3:59 PM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister
ardumont raised the priority of T4423: Gogs pagination API breaks because of fatal repos from Low to Normal.
Aug 4 2022, 3:58 PM · Gogs lister, Origin-Gitea/Gogs, Lister, Archive coverage
olasd added a comment to T4421: Prioritize archival from gitlab.com.

updated query running:

Aug 4 2022, 2:29 PM · Archive coverage, Origin-GitLab
olasd added a comment to T4421: Prioritize archival from gitlab.com.

As usual, I'm uneasy with the (general) idea of manually handling some repositories to resorb one bit of lag. This will only increase lag in another area that we will want to cover next. Rinse, repeat.

Aug 4 2022, 2:18 PM · Archive coverage, Origin-GitLab
vlorentz added a comment to T4421: Prioritize archival from gitlab.com.

answer: 4755

Aug 4 2022, 2:05 PM · Archive coverage, Origin-GitLab
vlorentz updated the task description for T4421: Prioritize archival from gitlab.com.
Aug 4 2022, 10:38 AM · Archive coverage, Origin-GitLab
vlorentz added a comment to T4421: Prioritize archival from gitlab.com.

I am currently running a query to find how many origins are over one year overdue for a visit:

Aug 4 2022, 10:29 AM · Archive coverage, Origin-GitLab
vlorentz triaged T4421: Prioritize archival from gitlab.com as Unbreak Now! priority.
Aug 4 2022, 10:28 AM · Archive coverage, Origin-GitLab

Jul 28 2022

ardumont added a project to T4411: Archive https://rsync.libreboot.org/: Archive coverage.
Jul 28 2022, 5:49 PM · Archive coverage

Jul 19 2022

vlorentz placed T1718: Implement a NuGet(.NET) lister up for grabs.
Jul 19 2022, 11:24 PM · Archive coverage

Jul 13 2022

vlorentz added a project to T4346: Create SourceHut Lister: Archive coverage.
Jul 13 2022, 10:52 AM · Archive coverage, Lister

Jun 29 2022

franckbret added a comment to T4104: Ingest crates.io (Rust).

Will work on the incremental lister, and then document (not already done).

Jun 29 2022, 3:43 PM · Crates loader, Crates lister, Archive coverage
ardumont updated the task description for T4104: Ingest crates.io (Rust).
Jun 29 2022, 3:19 PM · Crates loader, Crates lister, Archive coverage
ardumont added a comment to T4104: Ingest crates.io (Rust).

What the next steps here?

Jun 29 2022, 3:19 PM · Crates loader, Crates lister, Archive coverage
ardumont updated the task description for T4104: Ingest crates.io (Rust).
Jun 29 2022, 3:15 PM · Crates loader, Crates lister, Archive coverage
ardumont updated the task description for T4104: Ingest crates.io (Rust).
Jun 29 2022, 3:15 PM · Crates loader, Crates lister, Archive coverage