add an anti-Dos limit for edges traversed as a query parameter
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
All Stories
Apr 16 2021
What we can do, however:
Thanks to all of you for this dicussion and proposals.
@Hakimb You rebase is still incorrect. If you look at the content of the diff, there is still some of your initial work in red.
Great. In addition to the content of the free form field, the standard answer should contain proper boilerplate reminding what is expected in a Save Code Now request, along the lines of what is written in the "Help" tab of https://archive.softwareheritage.org/save/
thanks !
Build was aborted
It looks to me like this would be simpler if max_edges was given as a parameter to Traversal, since it's common to most methods. Would that work?
Arf, still two warnings remaining but no clear indication where they come from: https://jenkins.softwareheritage.org/job/DDOC/job/dev/5526/artifact/swh-docs/docs/errors.log/*view*/
fix + rebase
In D5544#140545, @vlorentz wrote:Uh? That wasn't an issue for me
Uh? That wasn't an issue for me
questions inline but nonetheless ok
In D5543#140521, @anlambert wrote:In D5543#140518, @vlorentz wrote:Oh, looks like I forgot to submit my commit for this...
Ack, there is also a remaining warning in swh-core. I guess you must have forgot to push that fix too ?
In D5543#140518, @vlorentz wrote:Oh, looks like I forgot to submit my commit for this...
Oh, looks like I forgot to submit my commit for this...
Build is green
Rebase
In T3252#63410, @ardumont wrote:
+1, can you create a task about it ? This could be handled by a GSOC student who chooses to
work on the webapp.
- unix credentials disabled
- user tenma removed from the groups Staff and Reviewers in phabricator
- VPN credential revoked:
root@louvre:~# cd /etc/openvpn/keys root@louvre:/etc/openvpn/keys# ls dh1024.pem dh2048.pem easyrsa old pki vars x509-types root@louvre:/etc/openvpn/keys# ./easyrsa revoke tenma
Build was aborted
In T3252#63374, @zack wrote:but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.
Adding an email field is a poor UX solution (it needs to be reentered every time or saved in a cookie) which we used for the vault at the time because we didn't have user registration.
Now that we have user registration we can just tell users that if they want to be notified, they should login. (Which is indeed something independent from requiring user registration for being able to submit.) That will encourage users to register to have added-value functionalitlies, like notifications.
And then we should go back to all places that could use notifications (vault, save code now, deposit, "save again" button) and uniform things.
but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.
Build is green
rebase
In D5484#140108, @vlorentz wrote:Nice! Why the green labels/checkboxes/dropdowns?
Build is green
Should I add the names of other contributors as well?
Rebase before pushing
In T3252#63334, @ardumont wrote:As a first step towards giving more feedback for users who submitted wrong origins for
ingestion (e.g. organization links, tarballs with wrong visit type, link to html page
probably for listing, etc...). We could allow the operator which rejects the origins a
free form input field so they could explain the reason of the rejection. It'd be less
brutal a rejection.This does not require the user registration part discussed above nor does it exclude it.
Bonus point for this, it's an easy hack ;)
As an incremental step after that, we could make that a configurable predefined template
selection box of rejecting reasons as I don't think there are so many different reasons
after all (unsupported for now, not an origin of type <type>, not a repository link,
...). Drawing stats from the first implementation could help in designing the initial
templates of rejection.Which could be another easy hack once the first part is done (if we want).
As suggested to @anlambert recently (@antoine, given it a bit more thought and added the
second incremental part since then thus the ping ;)
In T3252#63315, @zack wrote:Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).
In T3252#63314, @zack wrote:It would be desirable to provide the user with feedback that helps fix the issue.
Totally.
Now that we have a decent user registration system I think we should consider:
- requiring user registration for submitting save code now requests (which will also provide an audit trail for users that repeatedly submit bogus if not actively harmful requests)
- send by default email notifications about the outcome of save code now requests, both successes and failures, with the possibility of disabling email notifications in the user profile
This will make the overall UX of interacting with the archive feel much more "reliable" for users, whereas right now it feels much like a leap of faith whether it will work or not, in good part due to the lack of systematic out-of-band notifications.
Deployed, so the queue is now consumed.
Well, that was simple enough.
I'm not a big fan of the random pickup of repo because it adds entropy on the result, but otherwise it's great to be able to monitor this 👍
As a first step towards giving more feedback for users who submitted wrong origins for
ingestion (e.g. organization links, tarballs with wrong visit type, link to html page
probably for listing, etc...). We could allow the operator which rejects the origins a
free form input field so they could explain the reason of the rejection. It'd be less
brutal a rejection.
I'm not a big fan of the random pickup of repo because it adds entropy on the result, but otherwise it's great to be able to monitor this 👍
On a related note, it may be useful to regularly report requests that did not complete (either as success or failure) in a reasonable amount of time after being scheduled.
Build is green
Add same test for regular kafka writer
Apr 15 2021
Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).
It would be desirable to provide the user with feedback that helps fix the issue.
Thanks. Can you please make a release after landing this, so that docs.s.o gets updated?