Add LIP6 gitlab instance to regular crawling list
Closed, MigratedEdits Locked
Actions

Description

The LIP6 laboratory formally asked for its GitLab instance to be archived.
Here is the link to the forge https://gitlab.lip6.fr/

Event Timeline

rdicosmo triaged this task as Normal priority.Mar 7 2021, 8:40 AM

rdicosmo created this task.

We would like to see this in prod as soon as reasonably possible.

rdicosmo renamed this task from Ad LIP6 gitlab instance to regular crawling list to Add LIP6 gitlab instance to regular crawling list.Mar 8 2021, 7:02 PM

rdicosmo added a subscriber: olasd.

Checking in staging first, with:

swhworker@worker0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-gitlab-incremental url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 19957696
  Next run: today (2021-03-15T10:36:57.115547+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

Everything went fine:

worker1.internal.staging.swh.network: Mar 15 10:37:01 worker1 python3[2277003]: [2021-03-15 10:37:01,800: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.IncrementalGitLabLister[86f12806-f321-4ea1-8438-83c6fd0c457b]
worker1.internal.staging.swh.network: Mar 15 10:37:06 worker1 python3[2277067]: [2021-03-15 10:37:06,017: INFO/ForkPoolWorker-4] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[86f12806-f321-4ea1-8438-83c6fd0c457b] succeeded in 4.2116
25785101205s: {'pages': 5, 'origins': 64}

Listing deployed in production:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-incremental url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 377656882
  Next run: today (2021-03-15T10:47:18.408016+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-full url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 377656889
  Next run: today (2021-03-15T10:47:24.752195+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

Note: I may have pushed the scheduling of both incremental and full listing task a bit too close.

It worked nonetheless (as in same result than staging):

worker09: Mar 15 10:47:21 worker09 python3[2184142]: [2021-03-15 10:47:21,376: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.IncrementalGitLabLister[85bc5001-ab0c-493c-9e6a-c28e7c216859]
worker09: Mar 15 10:47:22 worker09 python3[2855486]: [2021-03-15 10:47:22,550: INFO/ForkPoolWorker-6] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[85bc5001-ab0c-493c-9e6a-c28e7c216859] succeeded in 0.8740711617283523s: {'pages': 5, 'origins': 64}
worker09: Mar 15 10:47:34 worker09 python3[2184142]: [2021-03-15 10:47:34,733: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.FullGitLabRelister[e7760b44-a48e-4ef6-abd2-f090ec7f16ba]
worker09: Mar 15 10:47:35 worker09 python3[2855486]: [2021-03-15 10:47:35,320: INFO/ForkPoolWorker-6] Task swh.lister.gitlab.tasks.FullGitLabRelister[e7760b44-a48e-4ef6-abd2-f090ec7f16ba] succeeded in 0.5820938930846751s: {'pages': 5, 'origins': 64}

I believe this is done: https://archive.softwareheritage.org/browse/search/?q=gitlab.lip6.fr&visit_type=git&with_content=true&with_visit=true
@ardumont: can you confirm?

This task has been migrated to GitLab.

Add LIP6 gitlab instance to regular crawling listClosed, MigratedEdits LockedActions

Description

Event Timeline

Add LIP6 gitlab instance to regular crawling list
Closed, MigratedEdits Locked
Actions