This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

https://grafana.softwareheritage.org/d/t0eircZ4k/vlorentz-scrubber-metrics

I've deployed the quiesced scrubbers (v0.1.0) on both staging and production.

Here are the results of the queries.
You can directly paste the json in the search profiler to see the result.
(Be careful some are quite huge)

the content of this comment was moved to the task description

WIP document: https://hedgedoc.softwareheritage.org/GYxIMBMXSRGVNxcCBrfA3w#

Theses errors will be caught by the alert created in T3222

Deployed.

Trying to bisect the issue and failed (multiple dimensions got the best of me, master ok, debian build in stable ko... i need to improve my tooling there).

So, the debian/unstable broke which is fixed now [1] (it wasd missing one new deps to
test the migration part).

Landed. Remains to deploy. I'll attend to that tomorrow.

It'd be best if we distinguish directly from the listing view such details.

In T3252#63370, @anlambert wrote:

In T3252#63315, @zack wrote:

Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).

+1, great idea !

yes, great ;)

@rdicosmo great summary, I'm certainly on that page :)

Thanks to all of you for this dicussion and proposals.

thanks !

In T3252#63410, @ardumont wrote:

+1, can you create a task about it ? This could be handled by a GSOC student who chooses to
work on the webapp.

sure done respectively T3256 then T3257.

+1, can you create a task about it ? This could be handled by a GSOC student who chooses to
work on the webapp.

In T3252#63374, @zack wrote:

but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.

Adding an email field is a poor UX solution (it needs to be reentered every time or saved in a cookie) which we used for the vault at the time because we didn't have user registration.
Now that we have user registration we can just tell users that if they want to be notified, they should login. (Which is indeed something independent from requiring user registration for being able to submit.) That will encourage users to register to have added-value functionalitlies, like notifications.
And then we should go back to all places that could use notifications (vault, save code now, deposit, "save again" button) and uniform things.

but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.

In T3252#63334, @ardumont wrote:

As a first step towards giving more feedback for users who submitted wrong origins for
ingestion (e.g. organization links, tarballs with wrong visit type, link to html page
probably for listing, etc...). We could allow the operator which rejects the origins a
free form input field so they could explain the reason of the rejection. It'd be less
brutal a rejection.

This does not require the user registration part discussed above nor does it exclude it.

Bonus point for this, it's an easy hack ;)

As an incremental step after that, we could make that a configurable predefined template
selection box of rejecting reasons as I don't think there are so many different reasons
after all (unsupported for now, not an origin of type <type>, not a repository link,
...). Drawing stats from the first implementation could help in designing the initial
templates of rejection.

Which could be another easy hack once the first part is done (if we want).

As suggested to @anlambert recently (@antoine, given it a bit more thought and added the
second incremental part since then thus the ping ;)

In T3252#63315, @zack wrote:

Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).

In T3252#63314, @zack wrote:

It would be desirable to provide the user with feedback that helps fix the issue.

Totally.

Now that we have a decent user registration system I think we should consider:

1. requiring user registration for submitting save code now requests (which will also provide an audit trail for users that repeatedly submit bogus if not actively harmful requests)
2. send by default email notifications about the outcome of save code now requests, both successes and failures, with the possibility of disabling email notifications in the user profile

This will make the overall UX of interacting with the archive feel much more "reliable" for users, whereas right now it feels much like a leap of faith whether it will work or not, in good part due to the lack of systematic out-of-band notifications.

As a first step towards giving more feedback for users who submitted wrong origins for
ingestion (e.g. organization links, tarballs with wrong visit type, link to html page
probably for listing, etc...). We could allow the operator which rejects the origins a
free form input field so they could explain the reason of the rejection. It'd be less
brutal a rejection.