Page MenuHomeSoftware Heritage

ListerFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Recent Activity

Sep 29 2021

ardumont closed T3358: Opam lister as Resolved.

It's been deployed for a while.

Sep 29 2021, 9:01 AM · Archive coverage, Lister

Sep 18 2021

zack added a project to T3358: Opam lister: Archive coverage.
Sep 18 2021, 8:29 AM · Archive coverage, Lister

Aug 25 2021

olasd added a comment to T3403: Use forge URL network location as default lister instance name.

I've duplicated the credentials for the relevant forges, and updated the following instance names:

Aug 25 2021, 2:38 PM · Scheduling utilities, Lister

Aug 3 2021

ardumont removed a revision from T3403: Use forge URL network location as default lister instance name: D6052: Install update-metrics as a service called daily.
Aug 3 2021, 2:32 PM · Scheduling utilities, Lister
ardumont added a revision to T3403: Use forge URL network location as default lister instance name: D6052: Install update-metrics as a service called daily.
Aug 3 2021, 2:25 PM · Scheduling utilities, Lister

Jul 29 2021

ardumont closed T3443: Deploy lister v1.5.0 as Resolved.
Jul 29 2021, 5:48 PM · System administration, Lister
ardumont moved T3443: Deploy lister v1.5.0 from in-progress to deployed/landed on the System administration board.
Jul 29 2021, 5:48 PM · System administration, Lister
ardumont added a comment to T3443: Deploy lister v1.5.0.

Still ongoing.

17:31:19 softwareheritage-scheduler@belvedere:5432=> select now(), visit_type, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where l.name='gitlab' and l.instance_name='gitlab' group by visit_type order by count(
*) desc;
+-------------------------------+------------+---------+
|              now              | visit_type |  count  |
+-------------------------------+------------+---------+
| 2021-07-29 15:46:55.979469+00 | git        | 1747880 |
+-------------------------------+------------+---------+
(1 row)
Jul 29 2021, 5:47 PM · System administration, Lister

Jul 28 2021

ardumont added a comment to T3443: Deploy lister v1.5.0.

It's still ongoing:

Jul 28 2021, 6:27 PM · System administration, Lister
ardumont claimed T3443: Deploy lister v1.5.0.
Jul 28 2021, 9:17 AM · System administration, Lister
ardumont added a comment to T3443: Deploy lister v1.5.0.

Listing ongoing and number of origins increasing regularly. From 200k yesterday [1]
(prior to the deployment of the lister) to ~800k [2] today. Progress.

Jul 28 2021, 9:17 AM · System administration, Lister

Jul 27 2021

ardumont changed the status of T3443: Deploy lister v1.5.0 from Open to Work in Progress.
Jul 27 2021, 6:24 PM · System administration, Lister
ardumont added a comment to T3443: Deploy lister v1.5.0.

Deployed v1.5.0 and run triggered and ongoing:

Jul 27 2021, 6:24 PM · System administration, Lister

Jul 24 2021

anlambert added a comment to T3442: Make GitLab lister more robust to HTTP errors .
swh-lister_1                     | [2021-07-24 08:58:25,127: INFO/ForkPoolWorker-1] Task swh.lister.gitlab.tasks.FullGitLabRelister[717c82b2-175a-492c-b701-22b4fd34e5e2] succeeded in 139476.781209187s: {'pages': 27470, 'origins': 2746838}
Jul 24 2021, 11:47 AM · Lister

Jul 23 2021

anlambert closed T3442: Make GitLab lister more robust to HTTP errors as Resolved.

Closing this as requested.

Jul 23 2021, 4:48 PM · Lister
ardumont added projects to T3443: Deploy lister v1.5.0: Lister, System administration.
Jul 23 2021, 4:43 PM · System administration, Lister
ardumont added a comment to T3442: Make GitLab lister more robust to HTTP errors .

Great, Thanks a lot.

Jul 23 2021, 4:40 PM · Lister
anlambert added a comment to T3442: Make GitLab lister more robust to HTTP errors .

For the record, my lister is still running, 1320500 gitlab.com origins listed so far.

Jul 23 2021, 2:39 PM · Lister
anlambert added a revision to T3442: Make GitLab lister more robust to HTTP errors : D6022: gitlab: Handle HTTP status code 500 when listing projects.
Jul 23 2021, 2:37 PM · Lister
anlambert added a revision to T3442: Make GitLab lister more robust to HTTP errors : D6020: gitlab: Adapt requests retry policy to consider HTTP 50x status codes.
Jul 23 2021, 1:54 PM · Lister
ardumont added a comment to T3442: Make GitLab lister more robust to HTTP errors .

As usual, very nice analysis \o/

Jul 23 2021, 12:25 PM · Lister
anlambert updated the task description for T3442: Make GitLab lister more robust to HTTP errors .
Jul 23 2021, 12:19 PM · Lister
anlambert triaged T3442: Make GitLab lister more robust to HTTP errors as Normal priority.
Jul 23 2021, 12:14 PM · Lister

Jul 20 2021

ardumont added a revision to T3358: Opam lister: D6010: opam: Directly use the --root flag instead of using an env variable.
Jul 20 2021, 4:56 PM · Archive coverage, Lister
ardumont renamed T3371: production: Deploy Tuleap lister from production: Deploy lister to production: Deploy Tuleap lister.
Jul 20 2021, 12:34 PM · System administration, Lister
ardumont renamed T3370: staging: Deploy Tuleap lister from staging: Deploy lister to staging: Deploy Tuleap lister.
Jul 20 2021, 12:34 PM · System administration, Lister

Jul 13 2021

anlambert added a revision to T3403: Use forge URL network location as default lister instance name: D5989: pattern: Use URL network location as instance name when not provided.
Jul 13 2021, 12:40 PM · Scheduling utilities, Lister
anlambert added a comment to T3403: Use forge URL network location as default lister instance name.

It seems the remaining lister instances to process are the phabricator ones that also need credentials.
This is what we currently have in the listers table in scheduler database.

Jul 13 2021, 11:47 AM · Scheduling utilities, Lister

Jul 9 2021

borisbaldassari added a comment to T3334: Tuleap lister.

Thanks @ardumont for the heads-up.
As a side note I still have on my to-do to add authentication and svn support to the tuleap lister (but that will happen *after* the maven lister work).

Jul 9 2021, 5:10 PM · Lister
olasd changed the status of T3403: Use forge URL network location as default lister instance name from Open to Work in Progress.

I've updated the listers with no credentials:

Jul 9 2021, 3:37 PM · Scheduling utilities, Lister
ardumont closed T3399: Improve PyPI lister to pull last update information when running incrementally as Resolved.

Deployed and running so closing.

Jul 9 2021, 2:52 PM · Origin-Pypi, Lister
ardumont added a comment to T3399: Improve PyPI lister to pull last update information when running incrementally.

Deployed in production as well and triggered a run:

Jul 9 2021, 2:51 PM · Origin-Pypi, Lister
ardumont added a comment to T3334: Tuleap lister.

It's not deployed but actually shipped in the lister v1.4 release onward.

Jul 9 2021, 2:22 PM · Lister
ardumont added a comment to T3358: Opam lister.

It's not deployed but actually shipped in the lister v1.4 release onward.

Jul 9 2021, 2:22 PM · Archive coverage, Lister
ardumont added a comment to T3399: Improve PyPI lister to pull last update information when running incrementally.

From a quick test, it looks like the "Project and release activity details" feed can go back multiple years without any issue, allowing us to backfill the data for all known origins, before adding the incremental behavior to the lister.

Jul 9 2021, 2:09 PM · Origin-Pypi, Lister
ardumont added a comment to T3399: Improve PyPI lister to pull last update information when running incrementally.

Deployed in staging and triggered a run:

Jul 9 2021, 2:03 PM · Origin-Pypi, Lister
ardumont added a revision to T3399: Improve PyPI lister to pull last update information when running incrementally: D5983: pypi.lister: Handle xml-rpc throttling properly.
Jul 9 2021, 10:50 AM · Origin-Pypi, Lister

Jul 7 2021

ardumont added a revision to T3399: Improve PyPI lister to pull last update information when running incrementally: D5977: Make PyPI lister incremental and complete in regards to last_update.
Jul 7 2021, 4:20 PM · Origin-Pypi, Lister
ardumont added a parent task for T3358: Opam lister: T3424: Opam support.
Jul 7 2021, 11:02 AM · Archive coverage, Lister

Jun 24 2021

ardumont updated the task description for T3406: Archive the mathrice gitlab forge.
Jun 24 2021, 12:13 PM · Origin-GitLab, Lister
ardumont renamed T3406: Archive the mathrice gitlab forge from Ingest the mathrice gitlab forge to Archive the mathrice gitlab forge.
Jun 24 2021, 10:30 AM · Origin-GitLab, Lister
ardumont renamed T3406: Archive the mathrice gitlab forge from List the mathrice gitlab instance to Ingest the mathrice gitlab forge.
Jun 24 2021, 10:29 AM · Origin-GitLab, Lister
ardumont triaged T3406: Archive the mathrice gitlab forge as Normal priority.
Jun 24 2021, 10:29 AM · Origin-GitLab, Lister

Jun 23 2021

anlambert triaged T3403: Use forge URL network location as default lister instance name as Normal priority.
Jun 23 2021, 11:47 AM · Scheduling utilities, Lister

Jun 21 2021

olasd closed T2454: Stop creating tasks directly in listers as Resolved.

This has all landed during the early 2021 sprint.

Jun 21 2021, 4:03 PM · Lister
olasd closed T2453: Use the scheduler "lister api" in listers as Resolved.

This has all been landed during the sprint at the beginning of 2021.

Jun 21 2021, 4:03 PM · Lister
olasd closed T2453: Use the scheduler "lister api" in listers, a subtask of T2454: Stop creating tasks directly in listers, as Resolved.
Jun 21 2021, 4:03 PM · Lister
olasd triaged T3399: Improve PyPI lister to pull last update information when running incrementally as Normal priority.
Jun 21 2021, 2:48 PM · Origin-Pypi, Lister

Jun 9 2021

olasd added a revision to T3358: Opam lister: D5838: Add opam to the common dockerfile, for swh.lister.
Jun 9 2021, 3:33 PM · Archive coverage, Lister
ardumont triaged T3371: production: Deploy Tuleap lister as Normal priority.
Jun 9 2021, 12:34 PM · System administration, Lister