As a first approximation, the URI scheme might be something like:
swh:VERSION:OBJECT_KIND:OBJECT_ID
with more specific instances like:
In a transaction with swhstorage user on softwareheritage db:
It looks like we've moved on onto this, and settled for what we have.
We can reconsider in the future if specific needs arise.
Done during the big postgres 9.5 upgrade window.
worker01 is in charge
Here is the complete list of URL that can be used to "git clone" (via HTTPS) all the repositories available from the Gitorious valhalla:
There are also duplicates that need to be removed.
There should only be one origin_visit.
So the 187 number here should be 1.
The gnu urls are of the form 'rsync://', that's the way to have hold on the origins to update.
swh-loader-tar should use the time of mirroring instead of the current time to create the occurrences
Hint: Use the TIMESTAMP file which holds the date of the last rsync.
We now have a beginning of implementation in rDSCH v0.0.4
This is now deployed on moma. Further automation via puppet still needs to be done.
The initial import of the snapshot.debian.org data has been done in december.
This is now deployed on moma. Further automation via puppet still needs to be done.
We could probably cheat by using the data from ghtorrent for the repositories that have already been listed, as this is a one-shot job and stale data is more interesting than no data. Worst case scenario: we base our clone on a repo that doesn't exist or doesn't contain the right data, and then we just fallback to regular cloning.
A new git updater, based on @ardumont's proof of concept, is now available in rDLDGIT.
An implementation of this is now available in rDSCH.
It currently breaks on *completely* empty messages, but the patch seems fairly simple.
We did some f2f thinking about this, concentrating on the "origin update" part of the mechanism. The shortcoming of our mechanism is that it's completely specific to updating our origins, and we can do something better...
This is now supported.
No need of making estimates anymore, as we know know.
After GitHub + Debian snapshot + GNU we have the following: