The "mapping version field" is the most fleshed out proposal as it would be my preference. My rationale for it against changing extid_type for backwards incompatible changes is that the extid_type is a property of the external artifact, while the mapping version is a property of our archiving infrastructure.

I'm "naturally averse" to the idea of wiping the extid entries (and it would probably be kind of a mess from a kafka perspective), but I'm not against it on principle either.

and it would probably be kind of a mess from a kafka perspective

Ah, good point.

I'm fine with options 1 and 2.

I've the feeling that option (1) will lead in the long run to an explosion on the size of the mapping which will make us eventually converge (slowly) toward option (3).

Option (3) would also be the first one I'd consider here, because the main use case (in my mind at least) is external lookup of an extid to something in the archive. So if that mapping change, but always give back an object in the archive (pointed by a SWHID), that doesn't seem a problem to me. It also doesn't seem to me to be a problem for the use case of avoiding re-archiving stuff (identified by extids) that has been archived in the past. So what kind of "relevant information" will we actually lose if we go with (3)?

Admittedly, (3) should be ideally implemented in a way that guarantees that extid that were resolvable in previous versions of the mapping will always be resolvable in future versions, which I don't know how easy/realistic will be in practice to guarantee. But the simplicity of (3) still has a lot of appeal, even if this property cannot be strongly guaranteed.

So if that mapping change, but always give back an object in the archive (pointed by a SWHID)

(3) should be ideally implemented in a way that guarantees that extid that were resolvable in previous versions of the mapping will always be resolvable in future versions

I don't understand. Option 3 is to remove relations between extids and SWHID, so it won't be resolvable anymore.

So what kind of "relevant information" will we actually lose if we go with (3)?

IMO, none, because it's only used by loader. However, it will become useful to other people when we implement T2430 (which depends on storing Disarchive data along with the extid)

(3) should be ideally implemented in a way that guarantees that extid that were resolvable in previous versions of the mapping will always be resolvable in future versions

I don't understand. Option 3 is to remove relations between extids and SWHID, so it won't be resolvable anymore.

We clarified this yesterday on IRC, but for the record: @vlorentz was pointing at the race condition between when old mappings are removed and new ones are not created yet; I was considering the fact that eventually all old extids will be resolvable again. There is also a way to avoid this race condition all together: making the extid→swhid direction of the mapping append/update only, but inhibit deletes. That way extids all always resolvable and will point to the new swhids as soon as modified loaders are rerun.

(We also discussed a bunch of other things, which I presume @olasd will eventually summarize here. Not sure we have a decision yet.)

For now I went the simplest way I could think of, which is:

• add a extid_version field to the ExtID object, defaulting to 0 (and backwards compatible, in terms of deduplication id, with the old objects without the field) - D6019
• in storage, store this field without using it at all: all queries in both directions still return all extid objects associated with the queried value, regardless of version. - D6023

In production, we currently have ~41 million entries in the extid table. I'm not sure it's worth much more effort to cut down the volume of these. Clients (i.e. loaders) can filter out the objects with versions they know they don't understand, which will make them parse and load the objects again.

The worst case scenario is having 41 million redundant entries, which is probably small enough. And I assume most objects will really be /new/ as we finally deploy the new mercurial loader and load all the bitbucket objects.

We should also reconsider this if we end up having a lot of churn in the "mapping version", but I don't really see that happening: this will only happen when we have strongly backwards incompatible versions of a loader happening, which is not something that we'll do a lot (and even then, we'll only load "live" origins again, so the duplication should stay under control).

Shipped the following modules to solve that problem:

• swh.model v2.7.0
• swh.storage v0.35.0
• swh.loader.mercurial v2.1.0

Now on to deployment for staging first then production if all is well.

Deployed \o/
Closing.