Deployed.

Trying to bisect the issue and failed (multiple dimensions got the best of me, master ok, debian build in stable ko... i need to improve my tooling there).

So, the debian/unstable broke which is fixed now [1] (it wasd missing one new deps to
test the migration part).

Landed. Remains to deploy. I'll attend to that tomorrow.

Thanks @ardumont ... so it appears that adapting the logic is easy... may you do it?

In T3213#63913, @rdicosmo wrote:

Thanks @ardumont ... so it appears that adapting the logic is easy... may you do it?
@anlambert may you look into the needed modification of the UI, to enable the new type of save code now payloads for selected authenticated users?

Thanks @ardumont ... so it appears that adapting the logic is easy... may you do it?
@anlambert may you look into the needed modification of the UI, to enable the new type of save code now payloads for selected authenticated users?

All checks green both for production/staging and hg/svn/git,

In T3087#63791, @douardda wrote:

So what about exports of the archive available on git-annex?

For the support of other origin visit types, @ardumont should know better than me how this could be integrated in the scheduler.

On a related note, it may be useful to regularly report requests that did not complete (either as success or failure) in a reasonable amount of time after being scheduled.

So what about exports of the archive available on git-annex?

In T3246#63559, @douardda wrote:

do we also intent to have a takedown topic on kafka?

It'd be best if we distinguish directly from the listing view such details.

In T3252#63370, @anlambert wrote:

In T3252#63315, @zack wrote:

Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).

+1, great idea !

Now that we have authentication and authorization in place, and that Software Heritage ambassadors are coming, we can relax this constraint, allowing specific users the ability to trigger "save code now" also for .tar, .zip, packages etc.

also: what about exports we provide on git annex?

do we also intent to have a takedown topic on kafka?

There is no hg origin for now but this is configurable so as soon as one one is found, this will be checked as well.

In the mean time, this error should not be too blocking on pergamon.

Damn, I tried to deploy the end to end save code now check but computer says no [1]

yes, great ;)

@rdicosmo great summary, I'm certainly on that page :)

Thanks to all of you for this dicussion and proposals.

Great. In addition to the content of the free form field, the standard answer should contain proper boilerplate reminding what is expected in a Save Code Now request, along the lines of what is written in the "Help" tab of https://archive.softwareheritage.org/save/

thanks !

In T3252#63410, @ardumont wrote:

+1, can you create a task about it ? This could be handled by a GSOC student who chooses to
work on the webapp.

sure done respectively T3256 then T3257.

+1, can you create a task about it ? This could be handled by a GSOC student who chooses to
work on the webapp.

In T3252#63374, @zack wrote:

but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.

Adding an email field is a poor UX solution (it needs to be reentered every time or saved in a cookie) which we used for the vault at the time because we didn't have user registration.
Now that we have user registration we can just tell users that if they want to be notified, they should login. (Which is indeed something independent from requiring user registration for being able to submit.) That will encourage users to register to have added-value functionalitlies, like notifications.
And then we should go back to all places that could use notifications (vault, save code now, deposit, "save again" button) and uniform things.

but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.

In T3252#63334, @ardumont wrote:

As a first step towards giving more feedback for users who submitted wrong origins for
ingestion (e.g. organization links, tarballs with wrong visit type, link to html page
probably for listing, etc...). We could allow the operator which rejects the origins a
free form input field so they could explain the reason of the rejection. It'd be less
brutal a rejection.

This does not require the user registration part discussed above nor does it exclude it.

Bonus point for this, it's an easy hack ;)

As an incremental step after that, we could make that a configurable predefined template
selection box of rejecting reasons as I don't think there are so many different reasons
after all (unsupported for now, not an origin of type <type>, not a repository link,
...). Drawing stats from the first implementation could help in designing the initial
templates of rejection.

Which could be another easy hack once the first part is done (if we want).

As suggested to @anlambert recently (@antoine, given it a bit more thought and added the
second incremental part since then thus the ping ;)

In T3252#63315, @zack wrote:

Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).

In T3252#63314, @zack wrote:

It would be desirable to provide the user with feedback that helps fix the issue.

Totally.

Now that we have a decent user registration system I think we should consider:

1. requiring user registration for submitting save code now requests (which will also provide an audit trail for users that repeatedly submit bogus if not actively harmful requests)
2. send by default email notifications about the outcome of save code now requests, both successes and failures, with the possibility of disabling email notifications in the user profile

This will make the overall UX of interacting with the archive feel much more "reliable" for users, whereas right now it feels much like a leap of faith whether it will work or not, in good part due to the lack of systematic out-of-band notifications.

As a first step towards giving more feedback for users who submitted wrong origins for
ingestion (e.g. organization links, tarballs with wrong visit type, link to html page
probably for listing, etc...). We could allow the operator which rejects the origins a
free form input field so they could explain the reason of the rejection. It'd be less
brutal a rejection.

Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).

It would be desirable to provide the user with feedback that helps fix the issue.

Let's go for it, then. May you take this over?

In T2912#63245, @vsellier wrote:

This kind of journal client will be necessary in any case if we want to extend the usage of the counters for other perimeters (metadata count, origin per forge, ...)

I would really like to keep the author counter: how complex is it to add it?

In T2912#63242, @vsellier wrote:

Staging webapp[1] and webapp1 on production [2] are now configured to use swh-counters to display the historical values and the live object counts.

Staging webapp[1] and webapp1 on production [2] are now configured to use swh-counters to display the historical values and the live object counts.

Knobs to adjust the visibility of origins in the archive and in the web API

In T3087#63054, @vsellier wrote:

Are we planning to add a way to notify the mirrors of the takedown notices ?

Are we planning to add a way to notify the mirrors of the takedown notices ?
I'm just thinking if it could be interesting to subscribe the staging environment to it to ensure the content is also removed from it (and also flagged to avoid any further ingestion).

Everything is released correctly and deployed on staging

Reopening as the release is not working on the stable branch