Rebase on D7570

Improve docs, rename variable to disable_logging_events

Rebase

Rebase on top of D7569

Rebase below D7566/D7567/D7570

Rebase

Looks like the number of affected revisions is fluctuating a bit:

In T1739#82920, @vlorentz wrote:

The original idea for this was to have separate tasks to fetch metadata, so that loaders did not have forge-specific code to fetch metadata.

However, the idea of loading metadata from loader is more appealing the more I think about it:

1. Metadata are fetched at about the same time as we snapshot code; which would allow showing more consistent states of repositories
2. Active repositories automatically have their metadata fetched more often than inactive ones
3. We don't have one more moving part to monitor and schedule
4. This allows the Git loader to know a new repo is a "forge fork" of another one before it starts loading, so it can do an incremental load

Thanks!

In D7507#196438, @vsellier wrote:

I tried to be more selective as you proposed but I'm not sure if it's possible. It seems the
html reports are not managed as build artifacts but as an output of the html publisher plugin.
The only way I've found is to change the keep-all[1] property to false but in this case, only one
report will be kept, which is even worse.

Rebase, again, on top of D7499.

• Rebase
• Remove spurious print

In D7499#196252, @vlorentz wrote:

It makes me a little uncomfortable that the validator needs access to the same secret key that is used for generating signatures; because the validator runs in a frontend app, and a leak of that secret key allows someone to spoof any address.

(specific targets if we're looking at less indiscriminate removals: docs diffs html output, coverage reports, ccm reports, mochawesome reports, duplicated junit-formatted test reports)

I don't like the idea of removing all old build logs indiscriminately, but it seems that noone else cares enough to do a more targeted analysis of what should be removed or not, so *shrug*.

• Rebase
• Add coverage for the html-only case
• Add coverage for the "multiple ambiguous parts length" behavior

In D7502#196144, @anlambert wrote:

I thought we could introduce flake8-bugbear for line length limits, as suggested by black's docs: https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length, but I can't seem to find a way to *only* enable B950 and not the other bugbear warnings. sigh.

I forgot to run make check again, good catch ! Indeed using flake8-bugbear seems the best way to proceed regarding line length warnings.
I managed to activate only that warning

rebase and apply @ardumont's comment

rebase

rebase

In D7502#196088, @olasd wrote:

Thanks for looking into this!

(sorry, typing... please hold)

rebase

rebase

The duplication between db_transaction and db_transaction_generator looks a bit silly, but I'm not sure there's much to do about it, so this lgtm, thanks.

So, from what I understand, we're currently generating releases combining intrinsic data (the target directory hash) and extrinsic data provided by the package index (release name and message).

Checking the last duration column here https://jenkins.softwareheritage.org/view/swh%20master/ I think we're good to go. Thank you!

I think 30 minutes is a lot of time to waste on an actual hang, but it's clear that this set of jobs is already on the edge.

swh.storage 1.2.0 increased a bunch of timeouts and made some queries smarter so it's entirely plausible that the number of errors has dropped drastically.

What exact metric are you saying isn't updating? In your diff, the swh_storage_request_duration_seconds_count{endpoint="index"} metric seems to be increasing normally

Thank you!

Thanks!

jfc.

SGTM, thanks!

That will apply to all jobs, which I don't think we want to do.

I have the feeling that, in terms of API extensibility, we'll want to be returning both the OriginVisit and its latest OriginVisitStatus.

So, I assume this will bump the swh.model dependency to 6.0 before it lands?

LGTM, thanks

Awesome, thanks!

In D7370#191858, @ardumont wrote:

(as a side note, I don't really understand why these views are split in two modules)

It's currently consistently separated in "public" views and "admin" views.

I think we should really block the views as well as the links when the feature is not enabled.

Looks fine, thanks!

In T3311#80997, @olasd wrote:

I'm not comfortable always creating high priority tasks in this context either, as I'm not sure what the throttling implications are when we inevitably end up on a repository that references a commit in a submodule that doesn't exist.

In D7332#190996, @vlorentz wrote:

FYI, @douardda added a comment on T3311 (I think it's the best place to discuss the design)

I think the approach in D7332 is interesting, but it feels a bit expensive to be doing it for every instance of a .gitmodules file found in any new directory for all git repos that are being loaded, as well as doing it again for the top level of any known branch in the git snapshot being loaded currently.

Cool, thanks.

Cypress parallel testing references:

One last rebase for the road?

In T3841#80779, @olasd wrote:

I think it's fine to remove the entries when we don't need them anymore (i.e. the object has been restored). Worst case, it'll be re-added at the next iteration of the script :-)

You'll need a column for which datastore has the corrupted object.

Use str.partition instead of str.split