I just simplified the regexp to allow qualifiers permutation: https://github.com/inveniosoftware/idutils/blob/cc09640ffb457bab3cfe8d0eeb4822dd521fd36d/idutils/__init__.py#L245-L249

Is there a way to improve the regex in https://github.com/inveniosoftware/idutils/pull/60 to allow qualifiers to come in any order instead of the canonical one?

PR submitted: https://github.com/inveniosoftware/idutils/pull/60

List of revisions with no parents (1259):

I've used swh-graph to lookup the 74 still missing contents, I've managed to find 67 of them, see cnt→ori mapping in (tracing them back to actual origins requires T2045):

151 contents have been restored with help from the provenance index, thanks to @grouss.

We should investigate why they are there.

This is now done, aside from a minor issue noted below:

softwareheritage-indexer=# select count(*) from revision_intrinsic_metadata where metadata = '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}'::jsonb;
 count 
-------
     0
(1 row)

The replication process from prado to somerset is now complete, and the archive frontend has been switched over to this database.

The replicated cluster is now clear to be taken down for a rebuild.

@vlorentz: lather, rinse, repeat.

softwareheritage-indexer=# DELETE FROM revision_intrinsic_metadata
WHERE metadata = '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}'::jsonb ;
ERROR:  deadlock detected
DETAIL:  Process 23900 waits for ShareLock on transaction 212164862; blocked by process 20175.
Process 20175 waits for ShareLock on transaction 212164381; blocked by process 23900.
HINT:  See server log for query details.
CONTEXT:  while deleting tuple (772424,55) in relation "revision_intrinsic_metadata"
Time: 33048,828 ms (00:33,049)

(just happened, after indexers have been restarted including D1218)

In T1549#29103, @vlorentz wrote:

D1218

Once this is landed and deployed, ordering your DELETEs by revision_metadata.id will acquire locks in the same order as the idx_storage, solving the deadlock issue.

Once this is landed and deployed, ordering your DELETEs by revision_metadata.id will acquire locks in the same order as the idx_storage, solving the deadlock issue.

The update completed, but a first attempt at the second DELETE failed with a deadlock (?!):

softwareheritage-indexer=# DELETE FROM revision_metadata
WHERE translated_metadata = '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}'::jsonb ;                                                                                            ERROR:  deadlock detected
DETAIL:  Process 10966 waits for ShareLock on transaction 197265813; blocked by process 11754.
Process 11754 waits for ShareLock on transaction 197264487; blocked by process 10966.
HINT:  See server log for query details.
CONTEXT:  while deleting tuple (1380733,15) in relation "revision_metadata"
Time: 170864,091 ms (02:50,864)

The following fix for the above (suggested by @vlorentz ) is now running:

update revision_metadata
set translated_metadata = origin_intrinsic_metadata.metadata
from origin_intrinsic_metadata
where revision_metadata.id=origin_intrinsic_metadata.from_revision and revision_metadata.translated_metadata='{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}' and origin_intrinsic_metadata.metadata != '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}';

As discussed on IRC, even after cleaning up origin_intrinsic_metadata, the DELETE on revision_metadata fails with:

softwareheritage-indexer=# DELETE FROM revision_metadata
softwareheritage-indexer-# WHERE translated_metadata = '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}'::json
b ;

I've started the first of following queries on somerset (in a screen of my user):

DELETE FROM origin_intrinsic_metadata
WHERE metadata = '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}'::jsonb ;

To know that we went over the origin and did not find anything, without additional tables.

I don't understand your comment. What are the remaining arguments for using NULL instead of just deleting rows?

and the NULL option

Yes :-)
so, do we agree that the right fix for this task is just to get rid of empty-ish rows? or are there other arguments that we haven't considered yet?

Sounds like a solution for T1528 :)

In T1549#28882, @vlorentz wrote:

What is the provenance map?

What is the provenance map?

My tentative proposal is to delete all table entries for which no metadata has been found.
The invariant will be: if an origin/revision has metadata, there will be an entry in the table(s); if not, the origin/revision will not appear.

After some more stewing and discussion with @zack, we'll be going for the "upgrade to pg 11 and restart replication from scratch" route;

After reading some mailing list posts discussing the error message, and discussion with @ftigeot:

Logs on primary:

2019-02-19 14:27:44 UTC [15973]: [1-1] user=postgres,db=softwareheritage LOG:  starting logical decoding for slot "pgl_softwareheritage_prado_somerset"
2019-02-19 14:27:44 UTC [15973]: [2-1] user=postgres,db=softwareheritage DETAIL:  streaming transactions committing after 18607/9189A578, reading WAL from 18607/9189A578
2019-02-19 14:27:44 UTC [15973]: [3-1] user=postgres,db=softwareheritage ERROR:  record with incorrect prev-link 5403A/2E2F1829 at 18607/9189A578 
2019-02-19 14:27:44 UTC [15973]: [4-1] user=postgres,db=softwareheritage LOG:  could not receive data from client: Connection reset by peer

@zack, this is now deployed.

thanks! can you ping me when this fix (T1303) is deployed?

After a few days of rest:

As other loader failure report, here is a better output:

Next step would be to simply extract the listing and reschedule those.

\o/

The low hanging fruits (i.e. stuff that is still live on GitHub) have been reimported.

Those need investigation, report...

@ardumont: can you file a separate task for migrating existing snapshot in the archive? (I suspect you've clearer than me in mind which snapshots need to be migrated…) TIA

It's currently running.

I'm closing this as it was about defining the naming convention and we have done so. I'm going to file a task about documenting it as part of the data model documentation.

It's currently running.

As per oral discussion, from the development point of view, we should now be good:

• the default branch for snapshots is defined to be HEAD.
  • if the concept of HEAD exists with the same name in the upstream VCS (f.e. git, svn), this branch should be a literal pointer to the corresponding archived object
  • if the concept of HEAD doesn't exist with the same name in the upstream VCS (f.e. mercurial), this branch should be an alias pointing at the default branch, named using the upstream VCS context (f.e. in the mercurial case, that would be an alias for the tip of the default branch)
  • if the concept of a default branch/version doesn't exist in the upstream VCS, no HEAD branch should exist in the snapshot

agreed, they should be removed (I've updated the task title accordingly)

All the old visits have now been migrated to snapshots.