There's some work in progress by @zack in branch feature/schema-revamp of rDSTO, I'll pick that up to drive it to completion.

And also make sure the one visit date is the right one:

I added T336 as parent so that the save code now is deployed only when this one is fixed.

Related D409

That's been done for a while now.

Spot checks:

Heads up on this btw.

Last edit to write all data to "temporary" table prior to actually cleanup (it was only done for release and snapshot so far. It's also done for origin_visit and fetch_history now).

In T1158#21531, @ardumont wrote:

@zack or @olasd if you have some time to review P286 at one point in time, that would be awesome.

Local tests are fine.

@zack or @olasd if you have some time to review P286 at one point in time, that would be awesome.

I added T336 as parent so that the save code now is deployed only when this one is fixed.

Last edit [1] to update code comments.

P286 is still wip as some now unclear exception occurs.

P286 is still wip as some now unclear exception occurs.

To find those releases:

why no errors reported at all in logs (or logs for that matters..., removing all filters, this seems to stop around the 7th of march 2018)

Current status, the queue is empty.

Finally, rescheduled using swh-scheduler.
Heading towards T986.

As in https://forge.softwareheritage.org/T879#16396, a limit of 2Gib on dump size was used to separate origins.
The current lists are stored at:

As this is not a bug, I am closing that task.

This example comes from parmap, see https://github.com/rdicosmo/parmap/

Thanks for the report. I haven't looked into this specific, so it's indeed possible it's a bug, but in the general case this is potentially normal behavior.
Branches can point to either releases or revisions (or, in fact, anything at all).
In the Git case, which looks like your case comes from, if one simply does a "git tag", that would create a ref pointing to a revision; whereas if one does "git tag -a" (annotated tag), that would create a release object (pointing to a revision) and a ref pointing to the release object. So an author that switched from using "git tag" to use "git tag -a" would justify what you have seen.

Status:

• [DONE] backup
• [IN-PROGRESS] Clean up in progress

Thanks for the heads up.

FWIW the backup has now completed.

I agree with taking tags from both sides and discarding all lines that don't fit the pattern.

Latest version with fixes deployed.
Still remains actions on T964 and clean up on T976 before scheduling another run.

postgres@prado:~$ pg_dump --format tar --table revision_history --table revision softwareheritage | gzip -c - > /srv/remote-backups/postgres/T970/revision-revision-history.tar.gz

Running a full backup of the table for the handful of revisions concerned here is a bit overkill! (better be safe than sorry and all that, but still...)

In T976#18114, @ardumont wrote:

In any case, we need to make a backup dump prior to touching those tables!

Backup running on prado:

postgres@prado:~$ pg_dump --format tar --table revision_history --table revision softwareheritage | gzip -c - > /srv/remote-backups/postgres/T970/revision-revision-history.tar.gz

In any case, we need to make a backup dump prior to touching those tables!

1. We can 'simply' delete the revision of type 'hg' as no other mercurial revision exists as of today.

I've chosen 3. to comply with the doc's suggestion.
As usual nothing is set in stone.

Well, sure, if the .hgtags is corrupted...

Last relevant error found in crossing multiple streams... (noooooo).

I have a problem in the log fetcher (that's why it's said to be incomplete in the task description).

I have a problem in the log fetcher (that's why it's said to be incomplete in the task description).

"PatoolError('error extracting /srv/storage/space/m": 19,

Rescheduled!

I close this as another mirror exists that we already browsed multiple times.

So back at updating the index and the origins for those.

Remains to update the existing origins (in db) with the right urls but not today.

Listing of the googlecode dumps recomputed with the right origin urls (in the actual INDEX-hg, old one is referenced as INDEX-hg.deprecated).