Page MenuHomeSoftware Heritage

Add LIP6 gitlab instance to regular crawling list
Open, HighPublic

Description

The LIP6 laboratory formally asked for its GitLab instance to be archived.
Here is the link to the forge https://gitlab.lip6.fr/

Event Timeline

rdicosmo triaged this task as Normal priority.Mar 7 2021, 8:40 AM
rdicosmo created this task.
rdicosmo raised the priority of this task from Normal to High.Mar 8 2021, 5:43 PM

We would like to see this in prod as soon as reasonably possible.

rdicosmo renamed this task from Ad LIP6 gitlab instance to regular crawling list to Add LIP6 gitlab instance to regular crawling list.Mar 8 2021, 7:02 PM
rdicosmo added a subscriber: olasd.

Checking in staging first, with:

swhworker@worker0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-gitlab-incremental url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 19957696
  Next run: today (2021-03-15T10:36:57.115547+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

Everything went fine:

worker1.internal.staging.swh.network: Mar 15 10:37:01 worker1 python3[2277003]: [2021-03-15 10:37:01,800: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.IncrementalGitLabLister[86f12806-f321-4ea1-8438-83c6fd0c457b]
worker1.internal.staging.swh.network: Mar 15 10:37:06 worker1 python3[2277067]: [2021-03-15 10:37:06,017: INFO/ForkPoolWorker-4] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[86f12806-f321-4ea1-8438-83c6fd0c457b] succeeded in 4.2116
25785101205s: {'pages': 5, 'origins': 64}

Listing deployed in production:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-incremental url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 377656882
  Next run: today (2021-03-15T10:47:18.408016+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-full url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 377656889
  Next run: today (2021-03-15T10:47:24.752195+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

Note: I may have pushed the scheduling of both incremental and full listing task a bit too close.

It worked nonetheless (as in same result than staging):

worker09: Mar 15 10:47:21 worker09 python3[2184142]: [2021-03-15 10:47:21,376: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.IncrementalGitLabLister[85bc5001-ab0c-493c-9e6a-c28e7c216859]
worker09: Mar 15 10:47:22 worker09 python3[2855486]: [2021-03-15 10:47:22,550: INFO/ForkPoolWorker-6] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[85bc5001-ab0c-493c-9e6a-c28e7c216859] succeeded in 0.8740711617283523s: {'pages': 5, 'origins': 64}
worker09: Mar 15 10:47:34 worker09 python3[2184142]: [2021-03-15 10:47:34,733: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.FullGitLabRelister[e7760b44-a48e-4ef6-abd2-f090ec7f16ba]
worker09: Mar 15 10:47:35 worker09 python3[2855486]: [2021-03-15 10:47:35,320: INFO/ForkPoolWorker-6] Task swh.lister.gitlab.tasks.FullGitLabRelister[e7760b44-a48e-4ef6-abd2-f090ec7f16ba] succeeded in 0.5820938930846751s: {'pages': 5, 'origins': 64}