The project.json and a couple of source archives have been downloaded during our early archival of google code, however I don't see a dump of the svn repository.

For now I went the simplest way I could think of, which is:

In D6019#155428, @ardumont wrote:

Relatedly to the need (not this diff), I'm unclear on how to deal with the data
migration now. For current extid clashes (which is the source of the task afaiui), how
can we determine which extid is the oldest one and which are more recent (if more than 2
extids)? [1]

Maybe we actually won't have anything to do in regards to migration.

We could adapt the loader so it detects the clash. That is, the loader finds more than
one version for the extid but without any extid-version. Then considers them moot [1],
bumps a new extid-version to 1 or something (thus recomputing the hashes for those
origins). And maybe with this, the loaders get unstuck and we are back on track.

[1] because it's anterior to this diff and we can't really detect what's what currently
(see caveats described in the task for the other discarded implementation).

Add backwards-compat test

Rebase

Ack (with the view that this will be refactored away in a few diffs' time)

make the handling of CTEs more modular

We've done the following:

In the global read index, I would consider storing, for each object, alongside the shard id, the length and offset of the object (which are comparatively cheap to store). This way, the per-shard index only gets used for standalone operation, which would (probably?) be an edge case. As I don't really grasp the specifics of the hashing algorithm, I have a hard time understanding how the actual performance will look like. log2(number objects in bucket) can still be a fairly small number of reads compared to 2 reads and a potentially expensive hashing algorithm?

With the available explanations I don't understand how this format works.

In T3408#67374, @vlorentz wrote:

And swh-graph too, please

Thanks!

I also wonder if we have a somewhat common approach to handle the SVN externals as well.

I think this is worthwhile in general, at least for repositories that are still live.

I've updated the listers with no credentials:

In D5977#154053, @ardumont wrote:

Looks good, thanks a lot!

In D5977#153851, @olasd wrote:

Looking at the size of the changelog (2-ish million entries for 50k-ish pages means 50-ish requests), I /think/ the lister could always be running in incremental mode, rather than having to maintain two modes in the long run.

From my quick testing, I have noticed that the changelog methods only returns a limited number of results, so we would need to iterate calls.

After clarification of the rationale on IRC, LGTM (with some marginal changes), thanks.

The "mapping version field" is the most fleshed out proposal as it would be my preference. My rationale for it against changing extid_type for backwards incompatible changes is that the extid_type is a property of the external artifact, while the mapping version is a property of our archiving infrastructure.

In T3127#66684, @anlambert wrote:

As @olasd said in a previous comment, even if we compute the metrics, we will miss counters about origins not tight to a lister
(googlecode and gitorious for instance). So I am thinking again about an hybrid approach using the swh-counters metrics
implemented yersteday which gives a rough estimation on the number of origins by network location (as visit statuses are not
processed, only origins) and the scheduler metrics.

Wow, I'm surprised that the templated garbage in setup.py still yields a working package :-)

Sorry @anlambert, I was late at Monday's meeting and I completely missed this in your weekly plan, I would have pointed this out earlier.

rebase

rebase

rebase

Use parametrization to test the logic of multiple cooldowns

Only vary the backoff test timestamp by 1 second

Nice, thanks.

Update backoff checking logic to vary the grab timestamp instead of the configured backoff

This has all landed during the early 2021 sprint.

This has all been landed during the sprint at the beginning of 2021.

Summary of the data available in the listed_origins table, broken down by lister and "known state" of origins:

Now that the branch structure has landed, I've deployed this latest version. After some cleanup of the duplicate extids left over from an earlier deployment, everything seems to be fine and the loader is ready for production.

The migration is complete. All staff have joined the new channels and most have registered an account on libera.chat.