Page MenuHomeSoftware Heritage

Use keyset pagination in Gitlab lister
Closed, MigratedEdits Locked

Description

Gitlab lister currently lists projects hosted on an instance using offset pagination when querying Gitlab REST API.

However, due to performance issues, a maximum allowed offset has been set (50000 for gitlab.com for instance), resulting to a partial projects listing when reaching it.

Fortunately, a more efficient pagination technique has recently been implemented in Gitlab: keyset pagination.

We should modify the lister implementation to use it in order to be able to list all projects hosted on a gitlab instance again.

For the record, I have tested on the instances we are currently listing in production (T1139) and keyset pagination is supported for all of them.

Related Objects