The fix is deployed on webapp1 and solved the problem.

The storage version v0.21.1 is deployed in staging, the problem looks fixed :

❯ curl -s  https://webapp.staging.swh.network/api/1/origin/https://gitlab.com/miwc/miwc.github.io.git/visit/latest/\?require_snapshot\=true | jq ''
{
  "origin": "https://gitlab.com/miwc/miwc.github.io.git",
  "date": "2020-12-07T18:21:58.967952+00:00",
  "type": "git",
  "visit": 1,
  "status": "full",
  "snapshot": "759b36e0e3e81e8cbf601181829571daa645b5d2",
  "metadata": {},
  "origin_url": "https://webapp.staging.swh.network/api/1/origin/https://gitlab.com/miwc/miwc.github.io.git/get/",
  "snapshot_url": "https://webapp.staging.swh.network/api/1/snapshot/759b36e0e3e81e8cbf601181829571daa645b5d2/"
}

Implemented in T2964

Sounds good to me.

Proposed manifest format:

Question: who should be responsible for filling this table? The loader or the storage (as side effect of revision_add)?

In T2849#54256, @olasd wrote:

In T2849#54254, @zack wrote:

Thanks, this would be an important new feature. Some comments/random thoughts below.

Further design point, which I think it's already implicit in what you wrote, but that it'd be useful to make explicit:

• this new mapping table is needed "only" to speed up things, if we lose it, it will just mean we will be slower in doing future archival (at least for a while), but won't be the end of the world

assuming this is true, loaders will need to be designed with graceful degradation for the incompleteness (or entire disappearance) of the mapping table.

Agreed. Missing objects in the mapping table would just mean that the loader does a bit more work to re-create the associated objects, and, hopefully, generate the same SWHIDs again (which would then be used to populate the mapping table).

Ok so the plan is a first step as simple as possible, implementing what @olasd proposed in the task, put this table in the storage, and provide a simple batch get API endpoint.

In T2849#54254, @zack wrote:

Thanks, this would be an important new feature. Some comments/random thoughts below.

Further design point, which I think it's already implicit in what you wrote, but that it'd be useful to make explicit:

• this new mapping table is needed "only" to speed up things, if we lose it, it will just mean we will be slower in doing future archival (at least for a while), but won't be the end of the world

assuming this is true, loaders will need to be designed with graceful degradation for the incompleteness (or entire disappearance) of the mapping table.

Thanks, this would be an important new feature. Some comments/random thoughts below.

In T2849#53968, @olasd wrote:

In T2849#53961, @marmoute wrote:

scope: mercurial will only need the mapping for revisions;

Mercurial could also use that mapping for every object. However, just keeping that mapping for revision is already providing a very large complexity boost (or reduction I should says) so that is "good enough" for us. I strongly suspect the same will apply to baazar.

Ack. I think having this for all objects in the short term would make it "too large", and having it for "objects with a history where computing the swhid is (deeply) recursive and expensive" (i.e. revisions and releases) is a decent balance.

In T2849#53961, @marmoute wrote:

scope: mercurial will only need the mapping for revisions;

Mercurial could also use that mapping for every object. However, just keeping that mapping for revision is already providing a very large complexity boost (or reduction I should says) so that is "good enough" for us. I strongly suspect the same will apply to baazar.

scope: mercurial will only need the mapping for revisions;

After T2828, It's more clear of what must be deployed to have the counters working on staging:

• the counters can be intialized via the /stat/refresh endpoint of the storage api (Note: It will create more counters than production as directory_entry_* and revision_history are not counted in production)
• Add a script/service to execute the `swh_update_counter_bucketed` in an infinite loop
• Create the buckets in the object_counts_bucketed
  • per object type : identifier|bucket_start|bucket_end. value and last_update will be updated be the stored procedures.
• configure prometheus sql exporter for db1.staging [1]
• configure profile_exporter on pergamon
  • Update the script to ensure the data are filtered by environments (to avoid staging data to be included in production counts [2])
  • Configure a new cron
    • loading an empty file for historical data
    • creating a new export_file
• update webapp to be able to configure the counter origin

I have generated a flame graph for the django view rendering a directory in a snapshot context and we can clearly see that the lookup_snapshot_sizes function
puts quite a burden on the overall process time.

And now it's here

We don't have database backups going back 10 months (nor 8 months, which was the horizon we would have needed at the time the task was submitted).

We don't have easily accessible backups going back to (before) August 2018, so I don't think we'll be able to recover this data.

This line of reasoning makes sense to me.

The fetch_history table is gone since swh.storage v0.0.155 / swh-storage schema v141.

FTR, olasd, douardda and I discussed an inconsistency in keys used in kafka, and decided to use hashes for all origin/visits/visit statuses; and doing the same for ext metadata in both kafka and the DB solves the issue about defining unicity.