Some thoughts about this (with contributions from @moranegg and @olasd):

• the API endpoints used by the github and bitbucket listers does not show extrinsic metadata, so that option is out
• sending a request for each repository would need ~2 to 3 years for a full pass over github. That's with our current infrastructure, so it's not a hard limit.

Also adding this example:
https://github.com/etalab/data-codes-sources-fr

In T833#31772, @vlorentz wrote:

• sending a request for each repository would need ~2 to 3 years for a full pass over github. That's with our current infrastructure, so it's not a hard limit.

Where is the bottleneck for this? API rate limit or what? We already use multiple tokens for listing github, can't we just do the same here and speed up (almost) arbitrarily a complete pass?

@zack Yes, rate limit. And I determined this based on our current listing rate (@olasd said 2 to 3 days to fully list GitHub with 500 repos per request; so an API call for each repo would take a total of 500 times 2 or 3 days).

But reading etalab's code gave me an idea: they send an API call per github organization/user (or more, if they have a lot of repos), and get multiple repo metadata at once; we could do that too.