Change Details

``` |----------------+----------------------------------------+-------------+----------------+-----------| | listing status | url | instance | # listed repos | # origins | |----------------+----------------------------------------+-------------+----------------+-----------| | done | https://gitlab.com/api/v4 | gitlab | 964329 | 965554 | | done | https://salsa.debian.org/api/v4/ | debian | 23757 | 9782 | | done | https://framagit.org/api/v4/ | framagit | 12134 | 12153 | | done | https://gitlab.inria.fr/api/v4/ | inria | 1606 | 1606 | | done | https://gitlab.gnome.org/api/v4/ | gnome | 3825 | 3904 | | done | https://gitlab.freedesktop.org/api/v4/ | freedesktop | 1949 | 2849 | | done | https://0xacab.org/api/v4/ | riseup | 883 | 884 | | done | https://gitlab.common-lisp.net/api/v4 | common-lisp | 551 | 731 | | done | https://gite.lirmm.fr/api/v4 | gite.lirmm | 292 | 293 | |----------------+----------------------------------------+-------------+----------------+-----------| ``` (- https://anonscm.debian.org/cgit/: down as it got migrated to a gitlab instance, there seems to be an archive at https://alioth-archive.debian.org/) - [x] Initialize db's data model with new table - [x] Schedule the ready cgit instance listing [1] with @nahimilega's support [1] ``` SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.savannah.gnu.org/cgit/ instance=gnu-savannah SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://gitweb.torproject.org/ url_prefix=https://git.torproject.org/ instance=tor SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.kernel.org/ instance=git-kernel SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://fedorapeople.org/cgit/ instance=fedora SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://cgit.kde.org/ url_prefix=https://anongit.kde.org/ instance=kde SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.alpinelinux.org/ instance=alpinelinux SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://hdiff.luite.com/cgit/ instance=hdiff.luite SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.baserock.org/cgit/ url_prefix=https://git.baserock.org/git/ instance=baserock SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://code.qt.io/cgit/ url_prefix=http://code.qt.io/ instance=qt.io SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.yoctoproject.org/clean/cgit.cgi/ url_prefix=https://git.yoctoproject.org/git/ instance=yoctoproject ```

``` |----------------+----------------------------------------+-------------+----------------+-----------| | listing status | url | instance | # listed repos | # origins | |----------------+----------------------------------------+-------------+----------------+-----------| | done | https://gitlab.com/api/v4 | gitlab | 964329 | 965554 | | done | https://salsa.debian.org/api/v4/ | debian | 23757 | 9782 | | done | https://framagit.org/api/v4/ | framagit | 12134 | 12153 | | done | https://gitlab.inria.fr/api/v4/ | inria | 1606 | 1606 | | done | https://gitlab.gnome.org/api/v4/ | gnome | 3825 | 3904 | | done | https://gitlab.freedesktop.org/api/v4/ | freedesktop | 1949 | 2849 | | done | https://0xacab.org/api/v4/ | riseup | 883 | 884 | | done | https://gitlab.common-lisp.net/api/v4 | common-lisp | 551 | 731 | | done | https://gite.lirmm.fr/api/v4 | gite.lirmm | 292 | 293 | |----------------+----------------------------------------+-------------+----------------+-----------| ``` (- https://anonscm.debian.org/cgit/: down as it got migrated to a gitlab instance, there seems to be an archive at https://alioth-archive.debian.org/) - [x] Initialize db's data model with new table - [x] Schedule the ready cgit instance listing [1] with @nahimilega's support [1] ``` SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.savannah.gnu.org/cgit/ instance=gnu-savannah SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://gitweb.torproject.org/ url_prefix=https://git.torproject.org/ instance=tor SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.kernel.org/ instance=git-kernel SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://fedorapeople.org/cgit/ instance=fedora SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://cgit.kde.org/ url_prefix=https://anongit.kde.org/ instance=kde SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.alpinelinux.org/ instance=alpinelinux SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://hdiff.luite.com/cgit/ instance=hdiff.luite SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.baserock.org/cgit/ url_prefix=https://git.baserock.org/git/ instance=baserock SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://code.qt.io/cgit/ url_prefix=http://code.qt.io/ instance=qt.io SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.yoctoproject.org/clean/cgit.cgi/ url_prefix=https://git.yoctoproject.org/git/ instance=yoctoproject ``` Note: - The number of origins might be a tad superior to the number of listed repos because the `count(*)` on origin tables to compute the number might be not as accurate. - That means that the ingestion is either in progress or done by the way.