Page MenuHomeSoftware Heritage

ingest Tor git repositories
Closed, MigratedEdits Locked

Description

they're at https://gitweb.torproject.org/

they're actually cgit, not gitweb, so there's that...

Event Timeline

zack triaged this task as Normal priority.Jun 12 2019, 5:20 PM
zack created this task.
anarcat renamed this task from ingest Tor gitweb repositories to ingest Tor git repositories.Jun 12 2019, 9:30 PM
anarcat updated the task description. (Show Details)

so the clone URLs are not exactly the same as the "gitweb" (AKA cgit) repo, so this requires further hacking... i tried this:

./cgit-lister https://gitweb.torproject.org/ | sed 's#https://gitweb#https://git#' | while read URL ; do
    printf "queuing $URL... " && 
    curl -X POST -sSL "https://archive.softwareheritage.org/api/1/origin/save/git/url/$URL" | jq . ; 
done

Unfortunately, I ended up getting throttled:

queuing https://git.torproject.org/atlas.git/... {
  "detail": "Request was throttled. Expected available in 2256 seconds."
}

So I am not sure where to go next. Here's the full list:

 https://git.torproject.org/atlas.git/
https://git.torproject.org/builders/gitian-builder.git/
https://git.torproject.org/builders/tor-browser-bundle.git/
https://git.torproject.org/compass.git/
https://git.torproject.org/globe.git/
https://git.torproject.org/jtorctl.git/
https://git.torproject.org/pyonionoo.git/
https://git.torproject.org/stormy.git/
https://git.torproject.org/tlsdate.git/
https://git.torproject.org/tor-browser/sandboxed-tor-browser.git/
https://git.torproject.org/tor-cloud.git/
https://git.torproject.org/torbel.git/
https://git.torproject.org/torbrowser.git/
https://git.torproject.org/torperf.git/
https://git.torproject.org/torstatus.git/
https://git.torproject.org/translation-attic.git/
https://git.torproject.org/vidalia.git/
https://git.torproject.org/vidalia-plugins.git/
https://git.torproject.org/weather.git/
https://git.torproject.org/webstats.git/
https://git.torproject.org/extern/libevent.git/
https://git.torproject.org/extern/openssl.git/
https://git.torproject.org/extern/qt.git/
https://git.torproject.org/extern/zlib.git/
https://git.torproject.org/admin/dns/auto-dns.git/
https://git.torproject.org/admin/dns/dns-helpers.git/
https://git.torproject.org/admin/dns/domains.git/
https://git.torproject.org/admin/dns/mini-nag.git/
https://git.torproject.org/admin/letsencrypt-domains.git/
https://git.torproject.org/admin/tor-jenkins.git/
https://git.torproject.org/admin/tor-nagios.git/
https://git.torproject.org/admin/trac/TracAccountManager.git/
https://git.torproject.org/admin/trac/trac-email.git/
https://git.torproject.org/admin/tsa-misc.git/
https://git.torproject.org/admin/userdir-ldap.git/
https://git.torproject.org/admin/userdir-ldap-cgi.git/
https://git.torproject.org/project/bridges/bridgedb-admin.git/
https://git.torproject.org/project/help/infra.git/
https://git.torproject.org/project/help/wiki.git/
https://git.torproject.org/project/jenkins/jobs.git/
https://git.torproject.org/project/jenkins/tools.git/
https://git.torproject.org/project/pipeline.git/
https://git.torproject.org/project/tor-browser/fastlane.git/
https://git.torproject.org/project/web/community.git/
https://git.torproject.org/project/web/donate.git/
https://git.torproject.org/project/web/manual.git/
https://git.torproject.org/project/web/newsletter.git/
https://git.torproject.org/project/web/styleguide.git/
https://git.torproject.org/project/web/support.git/
https://git.torproject.org/project/web/template.git/
https://git.torproject.org/project/web/tpo.git/
https://git.torproject.org/project/web/webwml.git/
https://git.torproject.org/mirror/libevent.git/
https://git.torproject.org/orfoxos/b2g.git/
https://git.torproject.org/orfoxos/b2g-flash-tool.git/
https://git.torproject.org/orfoxos/b2g-manifest.git/
https://git.torproject.org/orfoxos/gaia.git/
https://git.torproject.org/orfoxos/gecko.git/
https://git.torproject.org/orfoxos/gonk-misc.git/
https://git.torproject.org/orfoxos/libevent.git/
https://git.torproject.org/orfoxos/openssl.git/
https://git.torproject.org/orfoxos/tor.git/
https://git.torproject.org/debian/flashproxy.git/
https://git.torproject.org/debian/goptlib.git/
https://git.torproject.org/debian/meek.git/
https://git.torproject.org/debian/obfsproxy.git/
https://git.torproject.org/debian/obfsproxy-legacy.git/
https://git.torproject.org/debian/oonibackend.git/
https://git.torproject.org/debian/ooniprobe.git/
https://git.torproject.org/debian/pt-websocket.git/
https://git.torproject.org/debian/pyptlib.git/
https://git.torproject.org/debian/tor.git/
https://git.torproject.org/debian/tor-pristine-upstream.git/
https://git.torproject.org/debian/torproject-keyring.git/
https://git.torproject.org/debian/txtorcon.git/
https://git.torproject.org/arma/tor.git/
https://git.torproject.org/arma/torspec.git/
https://git.torproject.org/user/aagbsn/bridgedb.git/
https://git.torproject.org/user/aagbsn/pytorctl.git/
https://git.torproject.org/user/aagbsn/torflow.git/
https://git.torproject.org/user/ahf/tor.git/
https://git.torproject.org/user/ahf/tor-browser.git/
https://git.torproject.org/user/andrea/tor.git/
https://git.torproject.org/user/arma/tor.git/
https://git.torproject.org/user/arma/torspec.git/
https://git.torproject.org/user/art/bridgedb.git/
https://git.torproject.org/user/art/pytorctl.git/
https://git.torproject.org/user/art/tor.git/
https://git.torproject.org/user/art/torstatus.git/
https://git.torproject.org/user/asn/flashproxy.git/
https://git.torproject.org/user/asn/hax.git/
https://git.torproject.org/user/asn/hexchat.git/
https://git.torproject.org/user/asn/obfsproxy.git/
https://git.torproject.org/user/asn/pyptlib.git/
https://git.torproject.org/user/asn/tor.git/
https://git.torproject.org/user/asn/torspec.git/
https://git.torproject.org/user/atagar/doctor.git/
https://git.torproject.org/user/atagar/erebus.git/
https://git.torproject.org/user/atagar/nyx.git/
https://git.torproject.org/user/atagar/stem.git/
https://git.torproject.org/user/atagar/tor.git/
https://git.torproject.org/user/atagar/tor-utils.git/
https://git.torproject.org/user/atagar/torspec.git/
https://git.torproject.org/user/atagar/web/tpo.git/
https://git.torproject.org/user/blanu/blocking-test.git/
https://git.torproject.org/user/boklm/fpcentral.git/
https://git.torproject.org/user/boklm/rbm.git/
https://git.torproject.org/user/boklm/tor-browser-build.git/
https://git.torproject.org/user/boklm/tor-browser-bundle.git/
https://git.torproject.org/user/boklm/tor-browser-bundle-testsuite.git/
https://git.torproject.org/user/brade/gitian-builder.git/
https://git.torproject.org/user/brade/meek.git/
https://git.torproject.org/user/brade/tor-browser.git/
https://git.torproject.org/user/brade/tor-browser-build.git/
https://git.torproject.org/user/brade/tor-browser-bundle.git/
https://git.torproject.org/user/brade/tor-browser-spec.git/
https://git.torproject.org/user/brade/tor-launcher.git/
https://git.torproject.org/user/brade/torbrowser.git/
https://git.torproject.org/user/brade/torbutton.git/
https://git.torproject.org/user/chiiph/thandy.git/
https://git.torproject.org/user/chiiph/thandy-bootstrap.git/
https://git.torproject.org/user/chiiph/tor.git/
https://git.torproject.org/user/chiiph/vidalia.git/
https://git.torproject.org/user/chiiph/vidalia-plugins.git/
https://git.torproject.org/user/chrisd/polipo.git/
https://git.torproject.org/user/cohosh/tor-browser-build.git/
https://git.torproject.org/user/colin/curriculum.git/
https://git.torproject.org/user/colin/https-everywhere.git/
https://git.torproject.org/user/colin/support-portal.git/
https://git.torproject.org/user/colin/tb-videos.git/
https://git.torproject.org/user/colin/user-manual.git/
https://git.torproject.org/user/colin/webwml.git/
https://git.torproject.org/user/cristobal/erebus.git/
https://git.torproject.org/user/dcf/flashproxy.git/
https://git.torproject.org/user/dcf/oss.git/
https://git.torproject.org/user/dcf/snowflake.git/
https://git.torproject.org/user/dcf/tor-browser-build.git/
https://git.torproject.org/user/dcf/tor-browser-bundle.git/
https://git.torproject.org/user/dererk/arm.git/
https://git.torproject.org/user/dererk/pytorctl.git/
https://git.torproject.org/user/dgoulet/bridgedb.git/
https://git.torproject.org/user/dgoulet/bridgedb-admin.git/
https://git.torproject.org/user/dgoulet/chutney.git/
https://git.torproject.org/user/dgoulet/doctor.git/
https://git.torproject.org/user/dgoulet/tor.git/
https://git.torproject.org/user/dgoulet/torsocks.git/
https://git.torproject.org/user/dgoulet/torspec.git/
https://git.torproject.org/user/erinn/build-scripts.git/
https://git.torproject.org/user/erinn/thandy.git/
https://git.torproject.org/user/erinn/tor.git/
https://git.torproject.org/user/erinn/tor-browser-bundle.git/
https://git.torproject.org/user/erinn/torbrowser.git/
https://git.torproject.org/user/erinn/trac-test.git/
https://git.torproject.org/user/erinn/vidalia.git/
https://git.torproject.org/user/gk/bridgedb.git/
https://git.torproject.org/user/gk/gitian-builder.git/
https://git.torproject.org/user/gk/misc.git/
https://git.torproject.org/user/gk/tor-android-service.git/
https://git.torproject.org/user/gk/tor-browser.git/
https://git.torproject.org/user/gk/tor-browser-build.git/
https://git.torproject.org/user/gk/tor-browser-bundle.git/
https://git.torproject.org/user/gk/tor-browser-bundle-testsuite.git/
https://git.torproject.org/user/gk/tor-browser-spec.git/
https://git.torproject.org/user/gk/tor-launcher.git/
https://git.torproject.org/user/gk/torbutton.git/
https://git.torproject.org/user/gk/webwml.git/
https://git.torproject.org/user/griffin/stormy.git/
https://git.torproject.org/user/griffin/webwml.git/
https://git.torproject.org/user/gsathya/orbot.git/
https://git.torproject.org/user/gsathya/pathsim.git/
https://git.torproject.org/user/gsathya/pyonionoo.git/
https://git.torproject.org/user/gsathya/stem.git/
https://git.torproject.org/user/hacim/tor.git/
https://git.torproject.org/user/harmony/user-manual.git/
https://git.torproject.org/user/hiro/gettor.git/
https://git.torproject.org/user/hiro/jenkins-jobs.git/
https://git.torproject.org/user/hiro/jenkins-tools.git/
https://git.torproject.org/user/hiro/support.git/
https://git.torproject.org/user/hiro/webwml.git/
https://git.torproject.org/user/hiviah/rpm-build-scripts.git/
https://git.torproject.org/user/hiviah/rpm-tor.git/
https://git.torproject.org/user/hoganrobert/tor.git/
https://git.torproject.org/user/hoganrobert/torsocks.git/
https://git.torproject.org/user/huseby/b2g.git/
https://git.torproject.org/user/huseby/b2g-flash-tool.git/
https://git.torproject.org/user/huseby/b2g-manifest.git/
https://git.torproject.org/user/huseby/device-flame.git/
https://git.torproject.org/user/huseby/device-hamachi.git/
https://git.torproject.org/user/huseby/device-tarako.git/
https://git.torproject.org/user/huseby/gaia.git/
https://git.torproject.org/user/huseby/gecko.git/
https://git.torproject.org/user/huseby/gonk-misc.git/
https://git.torproject.org/user/huseby/libevent.git/
https://git.torproject.org/user/huseby/openssl.git/
https://git.torproject.org/user/huseby/tor.git/
https://git.torproject.org/user/igt0/tor-browser.git/
https://git.torproject.org/user/ilv/gettor.git/
https://git.torproject.org/user/irl/collector.git/
https://git.torproject.org/user/irl/exonerator.git/
https://git.torproject.org/user/irl/metrics-base.git/
https://git.torproject.org/user/irl/metrics-bot.git/
https://git.torproject.org/user/irl/metrics-cloud.git/
https://git.torproject.org/user/irl/metrics-lib.git/
https://git.torproject.org/user/irl/metrics-tasks.git/
https://git.torproject.org/user/irl/metrics-web.git/
https://git.torproject.org/user/irl/onionoo.git/
https://git.torproject.org/user/irl/research-web.git/
https://git.torproject.org/user/irl/styleguide.git/
https://git.torproject.org/user/irl/tech-reports.git/
https://git.torproject.org/user/irl/tor-icons.git/
https://git.torproject.org/user/irl/torspec.git/
https://git.torproject.org/user/isis/bridgedb.git/
https://git.torproject.org/user/isis/farfetchd.git/
https://git.torproject.org/user/isis/leekspin.git/
https://git.torproject.org/user/isis/ooni-probe.git/
https://git.torproject.org/user/isis/tor.git/
https://git.torproject.org/user/isis/torspec.git/
https://git.torproject.org/user/iwakeh/collector.git/
https://git.torproject.org/user/iwakeh/exonerator.git/
https://git.torproject.org/user/iwakeh/metrics-base.git/
https://git.torproject.org/user/iwakeh/metrics-lib.git/
https://git.torproject.org/user/iwakeh/metrics-web.git/
https://git.torproject.org/user/iwakeh/onionoo.git/
https://git.torproject.org/user/iwakeh/tor.git/
https://git.torproject.org/user/jvoisin/mat.git/
https://git.torproject.org/user/kaner/bridgedb.git/
https://git.torproject.org/user/kaner/gettor.git/
https://git.torproject.org/user/kaner/weather.git/
https://git.torproject.org/user/karsten/collector.git/
https://git.torproject.org/user/karsten/exonerator.git/
https://git.torproject.org/user/karsten/metrics-db.git/
https://git.torproject.org/user/karsten/metrics-lib.git/
https://git.torproject.org/user/karsten/metrics-tasks.git/
https://git.torproject.org/user/karsten/metrics-web.git/
https://git.torproject.org/user/karsten/onionoo.git/
https://git.torproject.org/user/karsten/research-web.git/
https://git.torproject.org/user/karsten/tech-reports.git/
https://git.torproject.org/user/karsten/tor.git/
https://git.torproject.org/user/karsten/torspec.git/
https://git.torproject.org/user/kat/webwml.git/
https://git.torproject.org/user/krkhan/arm.git/
https://git.torproject.org/user/linus/chutney.git/
https://git.torproject.org/user/linus/tor.git/
https://git.torproject.org/user/linus/tor-browser-bundle.git/
https://git.torproject.org/user/linus/torflow.git/
https://git.torproject.org/user/linus/torsocks.git/
https://git.torproject.org/user/linus/torspec.git/
https://git.torproject.org/user/lunar/metrics-tasks.git/
https://git.torproject.org/user/lunar/tor-and-https.git/
https://git.torproject.org/user/marlowe/rpm-tor.git/
https://git.torproject.org/user/mcs/tor-launcher.git/
https://git.torproject.org/user/mcs/torbrowser.git/
https://git.torproject.org/user/mcs/torbutton.git/
https://git.torproject.org/user/mikeperry/gitian-builder.git/
https://git.torproject.org/user/mikeperry/https-everywhere.git/
https://git.torproject.org/user/mikeperry/pytorctl.git/
https://git.torproject.org/user/mikeperry/tor.git/
https://git.torproject.org/user/mikeperry/tor-browser.git/
https://git.torproject.org/user/mikeperry/tor-browser-bundle.git/
https://git.torproject.org/user/mikeperry/tor-launcher.git/
https://git.torproject.org/user/mikeperry/torbrowser.git/
https://git.torproject.org/user/mikeperry/torbutton.git/
https://git.torproject.org/user/mikeperry/torflow.git/
https://git.torproject.org/user/mikeperry/torperf.git/
https://git.torproject.org/user/mikeperry/torspec.git/
https://git.torproject.org/user/n8fr8/orbot.git/
https://git.torproject.org/user/nickm/bridgedb.git/
https://git.torproject.org/user/nickm/calltool.git/
https://git.torproject.org/user/nickm/chutney.git/
https://git.torproject.org/user/nickm/githax.git/
https://git.torproject.org/user/nickm/obfsproxy.git/
https://git.torproject.org/user/nickm/tor.git/
https://git.torproject.org/user/nickm/tor-roadmaps.git/
https://git.torproject.org/user/nickm/torguts.git/
https://git.torproject.org/user/nickm/torsocks.git/
https://git.torproject.org/user/nickm/torspec.git/
https://git.torproject.org/user/nickm/trunnel.git/
https://git.torproject.org/user/nima/gettor.git/
https://git.torproject.org/user/nima/website.git/
https://git.torproject.org/user/pastly/tor.git/
https://git.torproject.org/user/pastly/webwml.git/
https://git.torproject.org/user/pde/https-everywhere.git/
https://git.torproject.org/user/phw/bridgedb.git/
https://git.torproject.org/user/phw/bridgedb-admin.git/
https://git.torproject.org/user/phw/exitmap.git/
https://git.torproject.org/user/phw/leekspin.git/
https://git.torproject.org/user/phw/obfs4portscan.git/
https://git.torproject.org/user/phw/sybilhunter.git/
https://git.torproject.org/user/phw/zoossh.git/
https://git.torproject.org/user/richard/tor-browser.git/
https://git.torproject.org/user/richard/tor-browser-build.git/
https://git.torproject.org/user/richard/tor-launcher.git/
https://git.torproject.org/user/richard/torbutton.git/
https://git.torproject.org/user/rransom/curve25519-donna.git/
https://git.torproject.org/user/rransom/libeventbook.git/
https://git.torproject.org/user/rransom/metrics-tasks.git/
https://git.torproject.org/user/rransom/scheme48.git/
https://git.torproject.org/user/rransom/tor.git/
https://git.torproject.org/user/rransom/tor-manual.git/
https://git.torproject.org/user/rransom/tor-utils.git/
https://git.torproject.org/user/rransom/torbrowser.git/
https://git.torproject.org/user/rransom/torflow.git/
https://git.torproject.org/user/rransom/torspec.git/
https://git.torproject.org/user/runa/gettor.git/
https://git.torproject.org/user/runa/tech-reports.git/
https://git.torproject.org/user/runa/tor-glossary.git/
https://git.torproject.org/user/runa/tor-manual.git/
https://git.torproject.org/user/runa/vidalia.git/
https://git.torproject.org/user/schoen/https-everywhere.git/
https://git.torproject.org/user/sebastian/arm.git/
https://git.torproject.org/user/sebastian/atlas.git/
https://git.torproject.org/user/sebastian/chutney.git/
https://git.torproject.org/user/sebastian/gettor.git/
https://git.torproject.org/user/sebastian/githax.git/
https://git.torproject.org/user/sebastian/gitian-builder.git/
https://git.torproject.org/user/sebastian/jtorctl.git/
https://git.torproject.org/user/sebastian/libevent.git/
https://git.torproject.org/user/sebastian/libeventbook.git/
https://git.torproject.org/user/sebastian/metrics-db.git/
https://git.torproject.org/user/sebastian/metrics-lib.git/
https://git.torproject.org/user/sebastian/metrics-web.git/
https://git.torproject.org/user/sebastian/obfsproxy.git/
https://git.torproject.org/user/sebastian/onionoo.git/
https://git.torproject.org/user/sebastian/orbot.git/
https://git.torproject.org/user/sebastian/stem.git/
https://git.torproject.org/user/sebastian/thandy.git/
https://git.torproject.org/user/sebastian/tor.git/
https://git.torproject.org/user/sebastian/tor-browser-bundle.git/
https://git.torproject.org/user/sebastian/tor-manual.git/
https://git.torproject.org/user/sebastian/tor-rust-dependencies.git/
https://git.torproject.org/user/sebastian/tor-utils.git/
https://git.torproject.org/user/sebastian/torbel.git/
https://git.torproject.org/user/sebastian/torbrowser.git/
https://git.torproject.org/user/sebastian/torperf.git/
https://git.torproject.org/user/sebastian/torspec.git/
https://git.torproject.org/user/sebastian/translation-tools.git/
https://git.torproject.org/user/sebastian/trunnel.git/
https://git.torproject.org/user/sebastian/vidalia.git/
https://git.torproject.org/user/sebastian/webstats.git/
https://git.torproject.org/user/serene/tor-browser-bundle.git/
https://git.torproject.org/user/shondoit/thandy.git/
https://git.torproject.org/user/shondoit/tor.git/
https://git.torproject.org/user/shondoit/torbrowser.git/
https://git.torproject.org/user/shondoit/vidalia.git/
https://git.torproject.org/user/sisbell/tor-browser-build.git/
https://git.torproject.org/user/sjm217/libnatpmp-patches.git/
https://git.torproject.org/user/sjm217/metrics.git/
https://git.torproject.org/user/sjm217/miniupnpc-patches.git/
https://git.torproject.org/user/sjm217/pluggable-transport.git/
https://git.torproject.org/user/sjm217/tor.git/
https://git.torproject.org/user/sjm217/torbrowser.git/
https://git.torproject.org/user/sjm217/torspec.git/
https://git.torproject.org/user/special/tor.git/
https://git.torproject.org/user/special/torspec.git/
https://git.torproject.org/user/sukhbir/gettor.git/
https://git.torproject.org/user/sukhbir/tor-browser-build.git/
https://git.torproject.org/user/sukhbir/webwml.git/
https://git.torproject.org/user/sysrqb/bridgedb.git/
https://git.torproject.org/user/sysrqb/chutney.git/
https://git.torproject.org/user/sysrqb/fastlane.git/
https://git.torproject.org/user/sysrqb/tor.git/
https://git.torproject.org/user/sysrqb/tor-browser.git/
https://git.torproject.org/user/sysrqb/tor-browser-build.git/
https://git.torproject.org/user/sysrqb/tor-launcher.git/
https://git.torproject.org/user/sysrqb/torsocks.git/
https://git.torproject.org/user/sysrqb/torspec.git/
https://git.torproject.org/user/tom/depictor.git/
https://git.torproject.org/user/tomb/torperf.git/
https://git.torproject.org/user/vmon/stegotorus.git/
https://git.torproject.org/user/weasel/onionff.git/
https://git.torproject.org/user/weasel/tor.git/
https://git.torproject.org/user/yawning/liballium.git/
https://git.torproject.org/user/yawning/obfsclient.git/
https://git.torproject.org/user/yawning/obfsproxy.git/
https://git.torproject.org/user/yawning/pyptlib.git/
https://git.torproject.org/user/zwol/stegotorus.git/
https://git.torproject.org/user/zwol/tech-reports.git/
https://git.torproject.org/user/zwol/torspec.git/
https://git.torproject.org/user/arma/vidalia.git/
https://git.torproject.org/user/atagar/exitmap.git/
https://git.torproject.org/user/atagar/pytorctl.git/
https://git.torproject.org/user/atagar/weather.git/
https://git.torproject.org/user/ioerror/build-scripts.git/
https://git.torproject.org/user/ioerror/obfsproxy.git/
https://git.torproject.org/user/ioerror/tor.git/
https://git.torproject.org/user/ioerror/tor-cloud.git/
https://git.torproject.org/user/ioerror/tor-extra.git/
https://git.torproject.org/user/ioerror/torbrowser.git/
https://git.torproject.org/user/ioerror/torwall.git/
https://git.torproject.org/user/ioerror/ttdnsd.git/
https://git.torproject.org/user/irl/atlas.git/
https://git.torproject.org/user/irl/debian/ooniprobe.git/
https://git.torproject.org/user/karsten/compass.git/
https://git.torproject.org/user/karsten/doctor.git/
https://git.torproject.org/user/karsten/torperf.git/
https://git.torproject.org/user/karsten/weather.git/
https://git.torproject.org/user/linus/tor-extra.git/
https://git.torproject.org/user/phw/atlas.git/
https://git.torproject.org/user/phw/obfsproxy.git/
https://git.torproject.org/user/phw/scramblesuit.git/
https://git.torproject.org/user/phw/trnnr.git/"

@anarcat please hold off from using save code now for now. As we're planning to have a proper cgit lister, we can just add your instance to your rotation once that's done (unless this is super urgent, that is). That will have the additional advantage that we will automatically notice when new repos show up.

(but thanks a lot for the list and for adapting my hackish script :-))

btw, the list is ~400 repos for now

got it, holding off. i'll let you handle this from here on! keep in mind that tor might switch to gitlab in the future, so might have to redo that process eventually.

got it, holding off. i'll let you handle this from here on! keep in mind that tor might switch to gitlab in the future, so might have to redo that process eventually.

Right. Thanks for the heads up.
We do have the gitlab lister running in production though.
So that will be simple enough to handle when we'll need to ;)

Cheers,

By the way, listing tor's git repositories instance is within the scope of T1835.

ardumont changed the task status from Open to Work in Progress.Jun 28 2019, 8:13 PM

Same as gnu-savannah, instance listed.
Now waiting for scheduling to kick-in...

was this completed? i suspect this issue can be closed if the gitweb is being regularly crawled.

see also T2954

It was (and thanks for the ping).

It is marked as listed (and will soon be again):

   type    |                                                 arguments                                                  |           next_run            |         status
-----------+------------------------------------------------------------------------------------------------------------+-------------------------------+------------------------
 list-cgit | {"args": [], "kwargs": {"url": "https://gitweb.torproject.org/", "instance": "tor"}}                       | 2021-01-26 18:24:55.408174+00 | next_run_not_scheduled

And repositories can be found in the archive [1]

[1] https://archive.softwareheritage.org/browse/search/?q=torproject.org

ardumont claimed this task.