Page MenuHomeSoftware Heritage
Feed All Stories

Apr 16 2021

anlambert accepted D5545: phabricator: Emphasize the 'arc diff --update' message should not be the default one..
Apr 16 2021, 2:15 PM
anlambert added a revision to T2265: Building the documentation should not show any warning.: D5546: cli: Fix sphinx warning.
Apr 16 2021, 2:14 PM · Easy hack, Documentation
vlorentz requested review of D5545: phabricator: Emphasize the 'arc diff --update' message should not be the default one..
Apr 16 2021, 2:14 PM
Hakimb updated the diff for D5501: add an anti-Dos limit for edges traversed as a query parameter.

add an anti-Dos limit for edges traversed as a query parameter

Apr 16 2021, 2:04 PM
ardumont updated the summary of D5541: Add end to end save code now check.
Apr 16 2021, 1:53 PM
vlorentz added a comment to T2602: Investigate how to upgrade the schema of the Cassandra storage.

What we can do, however:

Apr 16 2021, 1:45 PM · Storage manager
rdicosmo added a comment to T3252: Better handling of erroneous origins submitted to save code now.

Thanks to all of you for this dicussion and proposals.

Apr 16 2021, 1:39 PM · System administration, Save Code Now, Web app
vlorentz added a subtask for T1892: Cassandra as a storage backend: T2602: Investigate how to upgrade the schema of the Cassandra storage.
Apr 16 2021, 1:36 PM · meta-task, Storage manager
vlorentz added a parent task for T2602: Investigate how to upgrade the schema of the Cassandra storage: T1892: Cassandra as a storage backend.
Apr 16 2021, 1:36 PM · Storage manager
vlorentz added a comment to D5501: add an anti-Dos limit for edges traversed as a query parameter.

@Hakimb You rebase is still incorrect. If you look at the content of the diff, there is still some of your initial work in red.

Apr 16 2021, 1:30 PM
rdicosmo added a comment to T3256: Propose reason for rejecting a save code now.

Great. In addition to the content of the free form field, the standard answer should contain proper boilerplate reminding what is expected in a Save Code Now request, along the lines of what is written in the "Help" tab of https://archive.softwareheritage.org/save/

Apr 16 2021, 1:24 PM · Save Code Now, Easy hack, Web app
zack added a comment to T3252: Better handling of erroneous origins submitted to save code now.

thanks !

Apr 16 2021, 1:15 PM · System administration, Save Code Now, Web app
Harbormaster failed remote builds in B20804: Diff 19791 for D5501: add an anti-Dos limit for edges traversed as a query parameter!
Apr 16 2021, 1:11 PM
swh-public-ci added a comment to D5501: add an anti-Dos limit for edges traversed as a query parameter.

Build was aborted

Apr 16 2021, 1:11 PM
seirl added a comment to D5501: add an anti-Dos limit for edges traversed as a query parameter.

It looks to me like this would be simpler if max_edges was given as a parameter to Traversal, since it's common to most methods. Would that work?

Apr 16 2021, 1:09 PM
anlambert added a comment to D5544: docs/cli: Fix sphinx warning.

Arf, still two warnings remaining but no clear indication where they come from: https://jenkins.softwareheritage.org/job/DDOC/job/dev/5526/artifact/swh-docs/docs/errors.log/*view*/

Apr 16 2021, 12:55 PM
Hakimb updated the diff for D5501: add an anti-Dos limit for edges traversed as a query parameter.

fix + rebase

Apr 16 2021, 12:50 PM
anlambert closed D5544: docs/cli: Fix sphinx warning.
Apr 16 2021, 12:44 PM
anlambert committed rDCORE80832699e01d: docs/cli: Fix sphinx warning (authored by anlambert).
docs/cli: Fix sphinx warning
Apr 16 2021, 12:44 PM
anlambert added a comment to D5544: docs/cli: Fix sphinx warning.

Uh? That wasn't an issue for me

Apr 16 2021, 12:43 PM
ardumont closed T3221: elk: automatically limit log retention, a subtask of T3219: No logs are ingested on elasticsearch since 2021-03-26, as Resolved.
Apr 16 2021, 12:34 PM · System administrators
ardumont closed T3221: elk: automatically limit log retention as Resolved.
Apr 16 2021, 12:34 PM · System administration
ardumont closed T3255: save_code_now: Investigate why svn loading tasks are stuck as Resolved.
Apr 16 2021, 12:34 PM · System administration, Web app
vlorentz added a comment to D5544: docs/cli: Fix sphinx warning.

Uh? That wasn't an issue for me

Apr 16 2021, 12:34 PM
ardumont accepted D5544: docs/cli: Fix sphinx warning.

questions inline but nonetheless ok

Apr 16 2021, 12:34 PM
ardumont added inline comments to D5544: docs/cli: Fix sphinx warning.
Apr 16 2021, 12:33 PM
anlambert requested review of D5544: docs/cli: Fix sphinx warning.
Apr 16 2021, 12:31 PM
anlambert added a comment to D5543: multiplexer_objstorage: Fix sphinx warning and napoleon style.

Oh, looks like I forgot to submit my commit for this...

Ack, there is also a remaining warning in swh-core. I guess you must have forgot to push that fix too ?

Apr 16 2021, 12:30 PM
anlambert added a revision to T2265: Building the documentation should not show any warning.: D5544: docs/cli: Fix sphinx warning.
Apr 16 2021, 12:30 PM · Easy hack, Documentation
anlambert closed D5543: multiplexer_objstorage: Fix sphinx warning and napoleon style.
Apr 16 2021, 12:26 PM
anlambert committed rDOBJSe92746e15d8d: multiplexer_objstorage: Fix sphinx warning and napoleon style (authored by anlambert).
multiplexer_objstorage: Fix sphinx warning and napoleon style
Apr 16 2021, 12:26 PM
anlambert added a comment to D5543: multiplexer_objstorage: Fix sphinx warning and napoleon style.

Oh, looks like I forgot to submit my commit for this...

Apr 16 2021, 12:25 PM
vlorentz accepted D5543: multiplexer_objstorage: Fix sphinx warning and napoleon style.

Oh, looks like I forgot to submit my commit for this...

Apr 16 2021, 12:09 PM
anlambert requested review of D5543: multiplexer_objstorage: Fix sphinx warning and napoleon style.
Apr 16 2021, 12:07 PM
anlambert added a revision to T2265: Building the documentation should not show any warning.: D5543: multiplexer_objstorage: Fix sphinx warning and napoleon style.
Apr 16 2021, 12:06 PM · Easy hack, Documentation
swh-public-ci added a comment to D5484: assets/webapp-utils: Add SWHID validation in search inputs.

Build is green

Apr 16 2021, 12:02 PM
ardumont updated the task description for T3257: Template the save code now rejection reason to ease operator workflow.
Apr 16 2021, 11:55 AM · Save Code Now, Easy hack, Web app
anlambert updated the diff for D5484: assets/webapp-utils: Add SWHID validation in search inputs.

Rebase

Apr 16 2021, 11:55 AM
anlambert added a comment to T3252: Better handling of erroneous origins submitted to save code now.

+1, can you create a task about it ? This could be handled by a GSOC student who chooses to
work on the webapp.

sure done respectively T3256 then T3257.

Apr 16 2021, 11:54 AM · System administration, Save Code Now, Web app
ardumont added a comment to T3252: Better handling of erroneous origins submitted to save code now.

+1, can you create a task about it ? This could be handled by a GSOC student who chooses to
work on the webapp.

Apr 16 2021, 11:53 AM · System administration, Save Code Now, Web app
ardumont updated the task description for T3257: Template the save code now rejection reason to ease operator workflow.
Apr 16 2021, 11:52 AM · Save Code Now, Easy hack, Web app
ardumont triaged T3257: Template the save code now rejection reason to ease operator workflow as Normal priority.
Apr 16 2021, 11:52 AM · Save Code Now, Easy hack, Web app
vsellier updated subscribers of T3254: Outboarding of tenma.
  • unix credentials disabled
  • user tenma removed from the groups Staff and Reviewers in phabricator
  • VPN credential revoked:
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ls
dh1024.pem  dh2048.pem	easyrsa  old  pki  vars  x509-types
root@louvre:/etc/openvpn/keys# ./easyrsa revoke tenma
Apr 16 2021, 11:51 AM · System administration
Harbormaster failed remote builds in B20796: Diff 19783 for D5501: add an anti-Dos limit for edges traversed as a query parameter!
Apr 16 2021, 11:50 AM
swh-public-ci added a comment to D5501: add an anti-Dos limit for edges traversed as a query parameter.

Build was aborted

Apr 16 2021, 11:50 AM
vsellier placed T2743: Investigate why logging is inconsistent up for grabs.
Apr 16 2021, 11:48 AM · Vault, Object storage, Core & foundations
vsellier placed T1410: Kill implicit configuration: new configuration scheme up for grabs.
Apr 16 2021, 11:48 AM · Core & foundations
ardumont triaged T3256: Propose reason for rejecting a save code now as Normal priority.
Apr 16 2021, 11:48 AM · Save Code Now, Easy hack, Web app
vsellier placed T2885: investigate the persistent seemingly benign errors on cooking progress up for grabs.
Apr 16 2021, 11:48 AM · Vault
anlambert added a comment to T3252: Better handling of erroneous origins submitted to save code now.
In T3252#63374, @zack wrote:

but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.

Adding an email field is a poor UX solution (it needs to be reentered every time or saved in a cookie) which we used for the vault at the time because we didn't have user registration.
Now that we have user registration we can just tell users that if they want to be notified, they should login. (Which is indeed something independent from requiring user registration for being able to submit.) That will encourage users to register to have added-value functionalitlies, like notifications.
And then we should go back to all places that could use notifications (vault, save code now, deposit, "save again" button) and uniform things.

Apr 16 2021, 11:48 AM · System administration, Save Code Now, Web app
vsellier removed a member for Staff: Tanya.
Apr 16 2021, 11:44 AM
vsellier removed a member for Reviewers: tenma.
Apr 16 2021, 11:43 AM
zack added a comment to T3252: Better handling of erroneous origins submitted to save code now.

but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.

Apr 16 2021, 11:43 AM · System administration, Save Code Now, Web app
vsellier removed a member for Staff: tenma.
Apr 16 2021, 11:41 AM
vsellier committed rSPPRIVC8abf1d0534b8: refresh compared to uncensored data (authored by vsellier).
refresh compared to uncensored data
Apr 16 2021, 11:39 AM
ardumont moved T3255: save_code_now: Investigate why svn loading tasks are stuck from Backlog to deployed/landed/monitoring on the System administration board.
Apr 16 2021, 11:35 AM · System administration, Web app
ardumont accepted D5542: Remove tenma's access.
Apr 16 2021, 11:33 AM
ardumont updated the test plan for D5541: Add end to end save code now check.
Apr 16 2021, 11:32 AM
vsellier added a revision to T3254: Outboarding of tenma: D5542: Remove tenma's access.
Apr 16 2021, 11:31 AM · System administration
vsellier requested review of D5542: Remove tenma's access.
Apr 16 2021, 11:31 AM
swh-public-ci added a comment to D5541: Add end to end save code now check.

Build is green

Apr 16 2021, 11:31 AM
ardumont updated the summary of D5541: Add end to end save code now check.
Apr 16 2021, 11:30 AM
ardumont updated the diff for D5541: Add end to end save code now check.

Update

Apr 16 2021, 11:30 AM
Hakimb updated the diff for D5501: add an anti-Dos limit for edges traversed as a query parameter.

rebase

Apr 16 2021, 11:29 AM
anlambert added a comment to D5484: assets/webapp-utils: Add SWHID validation in search inputs.

Nice! Why the green labels/checkboxes/dropdowns?

Apr 16 2021, 11:17 AM
swh-public-ci added a comment to D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.

Build is green

Apr 16 2021, 11:17 AM
KShivendu added a comment to D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.

Should I add the names of other contributors as well?

Apr 16 2021, 11:15 AM
KShivendu updated the diff for D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.

Rebase before pushing

Apr 16 2021, 11:14 AM
anlambert added a comment to T3252: Better handling of erroneous origins submitted to save code now.

As a first step towards giving more feedback for users who submitted wrong origins for
ingestion (e.g. organization links, tarballs with wrong visit type, link to html page
probably for listing, etc...). We could allow the operator which rejects the origins a
free form input field so they could explain the reason of the rejection. It'd be less
brutal a rejection.

This does not require the user registration part discussed above nor does it exclude it.

Bonus point for this, it's an easy hack ;)

As an incremental step after that, we could make that a configurable predefined template
selection box of rejecting reasons as I don't think there are so many different reasons
after all (unsupported for now, not an origin of type <type>, not a repository link,
...). Drawing stats from the first implementation could help in designing the initial
templates of rejection.

Which could be another easy hack once the first part is done (if we want).

As suggested to @anlambert recently (@antoine, given it a bit more thought and added the
second incremental part since then thus the ping ;)

Apr 16 2021, 11:13 AM · System administration, Save Code Now, Web app
anlambert added a comment to T3252: Better handling of erroneous origins submitted to save code now.
In T3252#63315, @zack wrote:

Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).

Apr 16 2021, 11:11 AM · System administration, Save Code Now, Web app
anlambert added a comment to T3252: Better handling of erroneous origins submitted to save code now.
In T3252#63314, @zack wrote:

It would be desirable to provide the user with feedback that helps fix the issue.

Totally.

Now that we have a decent user registration system I think we should consider:

  1. requiring user registration for submitting save code now requests (which will also provide an audit trail for users that repeatedly submit bogus if not actively harmful requests)
  2. send by default email notifications about the outcome of save code now requests, both successes and failures, with the possibility of disabling email notifications in the user profile

This will make the overall UX of interacting with the archive feel much more "reliable" for users, whereas right now it feels much like a leap of faith whether it will work or not, in good part due to the lack of systematic out-of-band notifications.

Apr 16 2021, 11:11 AM · System administration, Save Code Now, Web app
ardumont added a project to T3255: save_code_now: Investigate why svn loading tasks are stuck: System administration.
Apr 16 2021, 11:10 AM · System administration, Web app
ardumont added a comment to T3255: save_code_now: Investigate why svn loading tasks are stuck.

Deployed, so the queue is now consumed.

Apr 16 2021, 11:09 AM · System administration, Web app
ardumont committed rSPSITEe933748ae1cb: Make high priority queue worker consumes svn tasks (authored by ardumont).
Make high priority queue worker consumes svn tasks
Apr 16 2021, 11:07 AM
ardumont added a comment to T3255: save_code_now: Investigate why svn loading tasks are stuck.

Well, that was simple enough.

Apr 16 2021, 11:04 AM · System administration, Web app
ardumont triaged T3255: save_code_now: Investigate why svn loading tasks are stuck as Normal priority.
Apr 16 2021, 11:01 AM · System administration, Web app
ardumont added a comment to D5541: Add end to end save code now check.

I'm not a big fan of the random pickup of repo because it adds entropy on the result, but otherwise it's great to be able to monitor this 👍

Apr 16 2021, 10:53 AM
vsellier changed the status of T3254: Outboarding of tenma from Open to Work in Progress.
Apr 16 2021, 10:42 AM · System administration
vsellier moved T3203: docs: Document the firewall installation and procedures from Backlog to Weekly backlog on the System administration board.
Apr 16 2021, 10:13 AM · Documentation, System administration
vsellier moved T3242: Decommission ClearlyDefined resources from Backlog to in-progress on the System administration board.
Apr 16 2021, 10:13 AM · System administration
ardumont moved T2117: Save Code Now: End to End monitoring from Backlog to in-progress on the System administration board.
Apr 16 2021, 10:13 AM · System administration, Monitoring, Roadmap 2021
vsellier moved T3243: Replace /dev/sdb and /dev/sdc on storage1.staging from Backlog to in-progress on the System administration board.
Apr 16 2021, 10:12 AM · System administration, Staging environment
ardumont moved T3221: elk: automatically limit log retention from in-progress to deployed/landed/monitoring on the System administration board.
Apr 16 2021, 10:12 AM · System administration
ardumont added inline comments to D5541: Add end to end save code now check.
Apr 16 2021, 9:40 AM
ardumont edited projects for T2117: Save Code Now: End to End monitoring, added: Roadmap 2021, Monitoring, System administration; removed Sprint 2019/12 (Monitor and Conquer).
Apr 16 2021, 9:37 AM · System administration, Monitoring, Roadmap 2021
ardumont added a comment to T3252: Better handling of erroneous origins submitted to save code now.

As a first step towards giving more feedback for users who submitted wrong origins for
ingestion (e.g. organization links, tarballs with wrong visit type, link to html page
probably for listing, etc...). We could allow the operator which rejects the origins a
free form input field so they could explain the reason of the rejection. It'd be less
brutal a rejection.

Apr 16 2021, 9:35 AM · System administration, Save Code Now, Web app
vsellier accepted D5541: Add end to end save code now check.

I'm not a big fan of the random pickup of repo because it adds entropy on the result, but otherwise it's great to be able to monitor this 👍

Apr 16 2021, 9:35 AM
rdicosmo added a comment to T2117: Save Code Now: End to End monitoring.

On a related note, it may be useful to regularly report requests that did not complete (either as success or failure) in a reasonable amount of time after being scheduled.

Apr 16 2021, 9:06 AM · System administration, Monitoring, Roadmap 2021
vlorentz accepted D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.
Apr 16 2021, 9:05 AM
swh-public-ci added a comment to D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.

Build is green

Apr 16 2021, 6:53 AM
KShivendu updated the diff for D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.

Add same test for regular kafka writer

Apr 16 2021, 6:50 AM
KShivendu updated the summary of D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.
Apr 16 2021, 6:48 AM
KShivendu retitled D5529: Add test to ensure that an exception is raised if unique_key isn't implemented from inmemory: Add test to detect exception in object unique_key to Add test to ensure that an exception is raised if unique_key isn't implemented.
Apr 16 2021, 6:47 AM

Apr 15 2021

zack added a comment to T3252: Better handling of erroneous origins submitted to save code now.

Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).

Apr 15 2021, 11:35 PM · System administration, Save Code Now, Web app
zack added a comment to T3252: Better handling of erroneous origins submitted to save code now.

It would be desirable to provide the user with feedback that helps fix the issue.

Apr 15 2021, 11:33 PM · System administration, Save Code Now, Web app
rdicosmo triaged T3252: Better handling of erroneous origins submitted to save code now as Normal priority.
Apr 15 2021, 10:47 PM · System administration, Save Code Now, Web app
zack accepted D5540: docs: Update for new schema.

Thanks. Can you please make a release after landing this, so that docs.s.o gets updated?

Apr 15 2021, 9:42 PM
ardumont added inline comments to D5541: Add end to end save code now check.
Apr 15 2021, 7:25 PM
ardumont updated the summary of D5541: Add end to end save code now check.
Apr 15 2021, 7:13 PM
ardumont updated the test plan for D5541: Add end to end save code now check.
Apr 15 2021, 7:09 PM