Query: Advanced Search

	Include stories about projects I am a member of.

On a related note, I found this list of Community-Hosted GitLab Instances. Most of them have public access and could be added to the set of Gitlab instances listed by Software Heritage.

All instances listed in this task have been added, so I'm closing this. Other instances can be added in the future, submitting matching task for tracking reasons as needed.

The "standard" listing (output recurring tasks with no priority) ran:

add it to our crawler rotation

A first round has been done:

Do we have an admin contact there, to make sure that cloning all their repos at once will not kill their infra?

$ curl --head https://gitlab.ow2.org/api/v4/projects
HTTP/2 200
server: nginx
date: Tue, 27 Aug 2019 10:21:30 GMT
content-type: application/json
content-length: 19658
vary: Accept-Encoding
cache-control: no-cache
link: <https://gitlab.ow2.org/api/v4/projects?membership=false&order_by=created_at&owned=false&page=2&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="next", <https://gitlab.ow2.org/api/v4/projects?membership=false&order_by=created_at&owned=false&page=1&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="first", <https://gitlab.ow2.org/api/v4/projects?membership=false&order_by=created_at&owned=false&page=51&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="last"
vary: Origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-next-page: 2
x-page: 1
x-per-page: 20
x-prev-page:
x-request-id: a8aqxWitUT6
x-runtime: 1.180268
x-total: 1003
x-total-pages: 51
strict-transport-security: max-age=31536000
referrer-policy: strict-origin-when-cross-origin

I think it's related to T1863.

Heads up, listing status for all instances (included new ones added):

This has been fixed.
The code changed the expected input.
We adapted the content of the lister tasks.

softwareheritage-scheduler=> select t.id, t.arguments, tr.status, tr.ended from task_run tr inner join task t on tr.task=t.id where t.type='swh-lister-gitlab-incremental' limit 10;
    id     |                                            arguments                                             |   status   |             ended
-----------+--------------------------------------------------------------------------------------------------+------------+-------------------------------
 104296277 | {"args": [{"instance": "framagit", "api_baseurl": "https://framagit.org/api/v4"}], "kwargs": {}} | failed     | 2019-02-14 14:24:05.082527+00

I had added framagit and 0xacab on Friday, forgot to update the task.

gitlab.lister: Do not expect gitlab instances to have credentials
gitlab.lister: Break asap when problem exists during fetch info

This should be good now.

A priori, at current speed, there remains ~7.5 days till the end of the gitlab origins ingestion.

In T1137#21093, @zack wrote:

this is a more far fetching question, but I wonder about the long-term sustainability of having a different DB for each lister. Is that scalable? I wonder if it wouldn't be better to have a single DB for all listers. I don't want to hold onto the deployment of the gitlab lister just for this, so go ahead as you feel better. But maybe take the chance to discuss this with other sysadms. If appropriate, we can consolidate things later.

I'll quite even possible separate the tasks in 2 (deploy gitlab lister, start listing gitlab.com)...
I'm a big fan of separation of concerns ;)

i guess this is about deploying the gitlab.com lister, right?

great, thanks for working on this!

Some repositories @olasd mentioned to me that qualifies as gitlab repositories (in parenthesis, their current size in term of repositories):

https://0xacab.org/api/v4/projects/ (600)
https://framagit.org/api/v4/projects/ (8619)
https://salsa.debian.org/api/v4/projects/ (25155)
https://gitlab.com/api/v4/projects/ (567086)
https://gitlab.freedesktop.org/api/v4/projects/ (254)
https://gitlab.gnome.org/api/v4/projects/ (3247)
https://gitlab.inria.fr/api/v4/projects/ (837)
...

Hi Katrin, just replied to your message with a more detailed overview of what is needed, adding it here too for reference
https://sympa.inria.fr/sympa/arc/swh-devel/2018-03/msg00002.html

Awesome! While you replied here, I wrote https://sympa.inria.fr/sympa/arc/swh-devel/2018-03/msg00000.html which this task felt necessary to create first :-D

Advanced Search
Use Results
Edit Query
Hide Query

Jun 24 2021

Feb 12 2021

Jan 27 2021

Jan 26 2021

Dec 11 2020

Oct 13 2020

Jan 23 2020

Jan 22 2020

Sep 3 2019

Sep 2 2019

Aug 29 2019

Aug 27 2019

Jul 16 2019

Jul 10 2019

Jul 1 2019

Jun 30 2019

Jun 29 2019

Jun 12 2019

Mar 30 2019

Feb 15 2019

Feb 14 2019

Nov 22 2018

Oct 29 2018

Oct 8 2018

Oct 5 2018

Sep 4 2018

Aug 24 2018

Jul 25 2018

Jul 20 2018

Jul 19 2018

Jul 18 2018

Jul 17 2018

Jul 5 2018

Jun 25 2018

Jun 19 2018

Mar 9 2018

Advanced SearchUse ResultsEdit QueryHide Query