Change Details

``` |----------------+----------------------------------------+-------------+----------------+-----------| | listing status | url | instance | # listed repos | # origins | |----------------+----------------------------------------+-------------+----------------+-----------| | done | https://gitlab.com/api/v4 | gitlab | 964329 | 965554 | | done | https://salsa.debian.org/api/v4/ | debian | 23757 | 9782 | | done | https://framagit.org/api/v4/ | framagit | 12134 | 12153 | | done | https://gitlab.inria.fr/api/v4/ | inria | 1606 | 1606 | | done | https://gitlab.gnome.org/api/v4/ | gnome | 3825 | 3904 | | done | https://gitlab.freedesktop.org/api/v4/ | freedesktop | 1949 | 2849 | | done | https://0xacab.org/api/v4/ | riseup | 883 | 884 | | done | https://gitlab.common-lisp.net/api/v4 | common-lisp | 551 | 731 | | done | https://gite.lirmm.fr/api/v4 | gite.lirmm | 292 | 293 | |----------------+----------------------------------------+-------------+----------------+-----------| ``` (- https://anonscm.debian.org/cgit/: down as it got migrated to a gitlab instance, there seems to be an archive at https://alioth-archive.debian.org/) - [x] Initialize db's data model with new table - [x] Schedule the ready cgit instance listing [1] with @nahimilega's support [1] ``` SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.savannah.gnu.org/cgit/ instance=gnu-savannah SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://gitweb.torproject.org/ url_prefix=https://git.torproject.org/ instance=tor SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.kernel.org/ instance=git-kernel SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://fedorapeople.org/cgit/ instance=fedora SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://cgit.kde.org/ url_prefix=https://anongit.kde.org/ instance=kde SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.alpinelinux.org/ instance=alpinelinux SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://hdiff.luite.com/cgit/ instance=hdiff.luite SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.baserock.org/cgit/ url_prefix=https://git.baserock.org/git/ instance=baserock SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://code.qt.io/cgit/ url_prefix=http://code.qt.io/ instance=qt.io SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.yoctoproject.org/clean/cgit.cgi/ url_prefix=https://git.yoctoproject.org/git/ instance=yoctoproject ``` Note: - The number of origins might be a tad superior to the number of listed repos because the `count(*)` on origin tables to compute the number might be not as accurate. - That means that the ingestion is either in progress or done by the way.

``` |--------+---------------------------------------------+-----------------------------------+----------------+----------------+----------------| | status | url | url-prefix | instance | # repos listed | # repos loaded | |--------+---------------------------------------------+-----------------------------------+----------------+----------------+----------------| | done | http://git.savannah.gnu.org/cgit/ | http://git.savannah.gnu.org/git/ | gnu-savannah | 1018 | 6 | | done | https://git.kernel.org/ | X | git-kernel | 1002 | 1002 | | done | https://gitweb.torproject.org/ | https://git.torproject.org/ | tor | 492 | 501 | | done | https://fedorapeople.org/cgit/ | X | fedora | 751 | 841 | | done | https://cgit.kde.org/ | https://anongit.kde.org/ | kde | 2434 | 2434 | | done | https://www.happyassassin.net/cgit/ | X | happyassassin | 3 | 3 | | done | https://git.openembedded.org/ | X | openembedded | 15 | 15 | | done | https://git.zx2c4.com/ | X | zx2c4 | 140 | 140 | | done | http://git.gnu.org.ua/cgit/ | http://git.gnu.org.ua/repo/ | git.gnu.org.ua | 50 | 50 | | done | https://git.alpinelinux.org/ | X | alpinelinux | 187 | 187 | | done | https://git.baserock.org/cgit/ | https://git.baserock.org/git/ | baserock | 1456 | 1546 | | done | https://code.qt.io/cgit/ | http://code.qt.io/ | qt.io | 257 | 257 | | done | http://git.yoctoproject.org/clean/cgit.cgi/ | https://git.yoctoproject.org/git/ | yoctoproject | 170 | 170 | | done | http://hdiff.luite.com/cgit/ | X | hdiff.luite | 13722 | 13722 | |--------+---------------------------------------------+-----------------------------------+----------------+----------------+----------------| ``` (- https://anonscm.debian.org/cgit/: down as it got migrated to a gitlab instance, there seems to be an archive at https://alioth-archive.debian.org/) - [x] Initialize db's data model with new table - [x] Schedule the ready cgit instance listing [1] with @nahimilega's support [1] ``` SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.savannah.gnu.org/cgit/ instance=gnu-savannah SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://gitweb.torproject.org/ url_prefix=https://git.torproject.org/ instance=tor SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.kernel.org/ instance=git-kernel SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://fedorapeople.org/cgit/ instance=fedora SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://cgit.kde.org/ url_prefix=https://anongit.kde.org/ instance=kde SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.alpinelinux.org/ instance=alpinelinux SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://hdiff.luite.com/cgit/ instance=hdiff.luite SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.baserock.org/cgit/ url_prefix=https://git.baserock.org/git/ instance=baserock SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://code.qt.io/cgit/ url_prefix=http://code.qt.io/ instance=qt.io SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.yoctoproject.org/clean/cgit.cgi/ url_prefix=https://git.yoctoproject.org/git/ instance=yoctoproject ``` Note: - The number of origins might be a tad superior to the number of listed repos because the `count(*)` on origin tables to compute the number might be not as accurate. - That means that the ingestion is either in progress or done by the way.

``` |--------+---------------------------------------+------+-----------------------------------+----------------+----------------+----------------| | listing status | url | instance | # listed repos | # originsstatus | url | url-prefix | instance | # repos listed | # repos loaded | |--------+---------------------------------------+------+-----------------------------------+----------------+----------------+----------------| | done | https://gitlab.com/api/v4 | gitlab | 964329 | 965554| http://git.savannah.gnu.org/cgit/ | http://git.savannah.gnu.org/git/ | gnu-savannah | 1018 | 6 | | done | https://salsa.debian.org/api/v4/ | debian | 23757 | 978| https://git.kernel.org/ | X | git-kernel | 1002 | 1002 | | done | https://framagit.org/api/v4/ | framagit | 12134 | 12153| https://gitweb.torproject.org/ | https://git.torproject.org/ | tor | 492 | 501 | | done | https://gitlab.inria.fr/api/v4/ | inria | 1606 | 1606| https://fedorapeople.org/cgit/ | X | fedora | 751 | 841 | | done | https://gitlab.gnome.org/api/v4/ | gnome | 3825 | 390| https://cgit.kde.org/ | https://anongit.kde.org/ | kde | 2434 | 2434 | | done | https://gitlab.freedesktop.org/api/v4/ | freedesktop | 1949 | 2849| https://www.happyassassin.net/cgit/ | X | happyassassin | 3 | 3 | | done | https://0xacab.org/api/v4/ | riseup | 883 | 884| https://git.openembedded.org/ | X | openembedded | 15 | 15 | | done | https://gitlab.common-lisp.net/api/v4 | common-lisp | 551 | 731| https://git.zx2c4.com/ | X | zx2c4 | 140 | 140 | | done | https://gite.lirmm.fr/api/v4 | gite.lirmm | 292 | 293| http://git.gnu.org.ua/cgit/ | http://git.gnu.org.ua/repo/ | git.gnu.org.ua | 50 | 50 | |----------------+----------------------------------------+-------------+----------------+----------- done | https://git.alpinelinux.org/ | X | alpinelinux | 187 | 187 | | done | https://git.baserock.org/cgit/ | https://git.baserock.org/git/ | baserock | 1456 | 1546 | | done | https://code.qt.io/cgit/ | http://code.qt.io/ | qt.io | 257 | 257 | | done | http://git.yoctoproject.org/clean/cgit.cgi/ | https://git.yoctoproject.org/git/ | yoctoproject | 170 | 170 | | done | http://hdiff.luite.com/cgit/ | X | hdiff.luite | 13722 | 13722 | |--------+---------------------------------------------+-----------------------------------+----------------+----------------+----------------| ``` (- https://anonscm.debian.org/cgit/: down as it got migrated to a gitlab instance, there seems to be an archive at https://alioth-archive.debian.org/) - [x] Initialize db's data model with new table - [x] Schedule the ready cgit instance listing [1] with @nahimilega's support [1] ``` SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.savannah.gnu.org/cgit/ instance=gnu-savannah SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://gitweb.torproject.org/ url_prefix=https://git.torproject.org/ instance=tor SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.kernel.org/ instance=git-kernel SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://fedorapeople.org/cgit/ instance=fedora SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://cgit.kde.org/ url_prefix=https://anongit.kde.org/ instance=kde SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://www.happyassassin.net/cgit/ instance=happyassassin SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.openembedded.org/ instance=openembedded SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.zx2c4.com/ instance=zx2c4 SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.alpinelinux.org/ instance=alpinelinux SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://hdiff.luite.com/cgit/ instance=hdiff.luite SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.gnu.org.ua/cgit/ url_prefix=http://git.gnu.org.ua/repo/ instance=git.gnu.org.ua SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://git.baserock.org/cgit/ url_prefix=https://git.baserock.org/git/ instance=baserock SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=https://code.qt.io/cgit/ url_prefix=http://code.qt.io/ instance=qt.io SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/; swh scheduler --url $SCHEDULER_API_URL task add list-cgit url=http://git.yoctoproject.org/clean/cgit.cgi/ url_prefix=https://git.yoctoproject.org/git/ instance=yoctoproject ``` Note: - The number of origins might be a tad superior to the number of listed repos because the `count(*)` on origin tables to compute the number might be not as accurate. - That means that the ingestion is either in progress or done by the way.