Page MenuHomeSoftware Heritage

ingest Tor git repositories
Open, NormalPublic


they're at

they're actually cgit, not gitweb, so there's that...

Event Timeline

zack created this task.Wed, Jun 12, 5:20 PM
zack triaged this task as Normal priority.
anarcat renamed this task from ingest Tor gitweb repositories to ingest Tor git repositories.
anarcat updated the task description. (Show Details)

so the clone URLs are not exactly the same as the "gitweb" (AKA cgit) repo, so this requires further hacking... i tried this:

./cgit-lister | sed 's#https://gitweb#https://git#' | while read URL ; do
    printf "queuing $URL... " && 
    curl -X POST -sSL "$URL" | jq . ; 

Unfortunately, I ended up getting throttled:

queuing {
  "detail": "Request was throttled. Expected available in 2256 seconds."

So I am not sure where to go next. Here's the full list:"
zack added a comment.EditedWed, Jun 12, 11:10 PM

@anarcat please hold off from using save code now for now. As we're planning to have a proper cgit lister, we can just add your instance to your rotation once that's done (unless this is super urgent, that is). That will have the additional advantage that we will automatically notice when new repos show up.

(but thanks a lot for the list and for adapting my hackish script :-))

zack added a comment.Wed, Jun 12, 11:11 PM

btw, the list is ~400 repos for now

got it, holding off. i'll let you handle this from here on! keep in mind that tor might switch to gitlab in the future, so might have to redo that process eventually.