Yes, we are hitting the same problem.

Basic checks on the archive is fine:

Out of 127k (127048) only ~125k (124899, query on swh db) are referenced.

FYI all loaded repositories point to an empty snapshot.

FYI all loaded repositories point to an empty snapshot.

FYI all loaded repositories point to an empty snapshot.

Some error speaks for themselves (OSError, error during extraction), some are not.
I'm currently digging into this and will open dedicated tasks when deemed necessary.

Out of 127k (127048) only ~125k (124899, query on swh db) are referenced.

yay

rDSNIP26ea29b2d2abf9c931ba5efcf0f49d4194254e79

This is now running on our swh-workers, scheduling running on saatchi:

It appears that in this case, the properties must be changed not to the symlink but to its source.

It's more empty repository case than a repository starting its commit range at 0...

This is in stand-by during the snapshot migration.

P202 checked and ok locally.
Now asked for review as it will remove data from the main db.

After discussion with the team, it has been decided to remove from the re-scheduling the svn dumps whose compressed size exceeds 2Gib.
This reflects the same decision took for git repositories.

Scheduled back from saatchi (as i needed the producer credentials to access the queue properties):

The old tool is id 7, the new one is 9:

Depends on T761

Bogus mimetype values are identified by the following queries:

softwareheritage=> select count(*) from content_mimetype where mimetype LIKE '[%' or mimetype like '' and indexer_configuration_id=7;
 count
-------
 50733
(1 row)

Status:

• Final listing of bogus values: /srv/storage/space/lists/indexer/mimetype/sha1-with-bogus-values.txt.gz (50733)
• Queue reached the sane point.
• workers stopped.

I am waiting for the queue to drop at 10000 as that will avoid rescheduling the already done 10000 (well except for the new bogus values :)

There might be other bogus values in the stats that I haven't noticed.

I don't see how i can easily check this though since we don't have the sha1 provenance yet.

From the top of my head, i would say that i forgot to clean up those bogus values after the initial runs around december 2016.
I don't see how i can easily check this though since we don't have the sha1 provenance yet.

The directories have been corrected, but the bogus file entries have not been deleted yet.

This is now done on uffizi.
Potentially remaining sub-tasks before closing this:

Updated SQL to also delete objects from tables that references them, e.g., the indexer ones.

The replication is now functional, indexes have been recreated, and the frontend now points to the new database.

thanks for the review, I've updated the SQL query accordingly

As a comment on 4, the object_id column is per-table, so you should avoid carrying it over to the skipped_content table.