The first part got deployed (modification in the webapp routines to update the save code
now statuses).

It is possibly also the only reasonable place where we can have heuristics to
de-duplicate URLs that point to the same repo, e.g., non-canonical GitHub repos URLs.

This will also let us think in terms of reusable widgets. I am not sure how important that would be in our case, but couple of use-cases are

• we can make swh search or the origin browser a public widget, it can be integrated in any external website with just two lines of HTML code.
• Many UI elements are in fact the same kind in SWH (eg: folder/file raw, list view etc). Most this can be re-used inside the system. This will make it efficient and easy to manage.

@zack Thanks for the input. I will start thinking about accessibility as well.
Many pages, including 'swh search', will not function as expected without enabling JS. So, I would say JS is integral.

In T3157#66041, @jayeshv wrote:

This will simplify branches view implementation. You should create a diff to handle it.

Yes. Let me check it's side effects as well. The 'archive.lookup_snapshot' is called from multiple places.

This will simplify branches view implementation. You should create a diff to handle it.

In T3157#66039, @jayeshv wrote:

@anlambert Thank you. I will try to do some refactoring here.

@anlambert Thank you. I will try to do some refactoring here.

There is no need to use JavaScript here from my point of view.

The ideal fix would be to refactor the code to move the entire presentation logic to the template engine. (We already have a check in the template to show "The list of branches is empty" message in case the branch list is empty. But, i think this case will never occur with the current server code)

More answers after further reading and analysis.

This will eventually cause a 404 error when the user search for a non existing branch name in the UI. I don't think this is the intended behavior.

@anlambert The server code seems to have an issue when branches are filtered by name.
"swh.web.common.archive.lookup_snapshot" function is raising an error if the number of searched branches is 0.
raise
This will eventually cause a 404 error when the user search for a non existing branch name in the UI. I don't think this is the intended behavior.

Removing the "easy hack" tag, because that's actually much harder than expected :D

I'm adding here a note about something to consider in terms of pros/cons: accessibility. As for the most part we are archiving textual information, we really want it to be accessible for all users. Right now we go further than that, ensuring that the Web UI can be browser with a textual browser: so, for instance, w3m https://archive.softwareheritage.org/swh:1:cnt:c839dea9e8e6f0528b468214348fee8669b305b2 just works out of the box. I'm not up to date on what's the accessibility impact of current JS frameworks, nor that we should have as a requirement that the archive is browsable without JavaScript enabled (as per today's standards "browsable with free javascript" is probably good enough for us, and we have a curl-able API anyway), but accessibility per se is definitely going to be a requirement.

It's deployed now.

@rdicosmo That's the issue ^ (unrelated to T3361 in the end), the workers were not suscribed to consume from that queue yet.

Can it work with a content SWHID with context?

In T3351#65594, @anlambert wrote:

I made some quick hacks in swh-web to see if it was easily feasible to have standalone source code view that could be embedded into an iframe, the answer is yes !

This is great to know (and see :-)

• The fix was deployed on webapp1 and moma
• The refresh script was manually launched:

root@webapp1:~# /usr/local/bin/refresh-savecodenow-statuses
Successfully updated 140 save request(s).

The previous requests were correctly refreshed and are now displaying the right status.

Will be deployed with version v0.0.310 of the webapp (build in progress)

1. Should we add a filter UI component in the page or just a query param is enough?

2: If we have to add a UI component, do we have a design or example for that? (maybe a generic one for both branches and Releases, the "search branches" widget in github looks nice)

@anlambert I have a few questions regarding this task.
1: Should we add a filter UI component in the page or just a query param is enough?
2: If we have to add a UI component, do we have a design or example for that? (maybe a generic one for both branches and Releases, the "search branches" widget in github looks nice)
3: What about the API /snapshot/<id>, should we add the possibility of this filter there as well?
4: The branches seem to be randomly sorted over Date. Is that for a reason or something we should address?

I made some quick hacks in swh-web to see if it was easily feasible to have standalone source code view that could be embedded into an iframe, the answer is yes !

Great news!

Great to see this progress (and kermit archived :-))

In T3213#65579, @anlambert wrote:

The feature has been implemented and looks ready for production use.

I just tested it using the Web API and the docker environment for a real world example: the Kermit Software Source Code Archive.

The feature has been implemented and looks ready for production use.

That will be helpful in general (to answer questions likes: which endpoint is over/underused for specific use cases) and also in view of seeing who over/underuses rate limits (e.g., to identify the need of having more generous rate limits for specific use cases).

right, ok. We actually do not have any of those in production (i don't recall having seen those at all, i don't even know what that would look like).

In T3313#65313, @ardumont wrote:

Django webapp logs are currently not really well managed.
They are currently dumped into a non rotated logfile on moma and only contain info related to requests with errors (meaning general access is not logged at all).
We were considering with @vsellier to redirect those logs into systemd in order to ease their ingestion by Logstash.
Logs format and levels should also be changed in order to track authenticated users access.

I recall some work on this has been done for that, currently apache logs are pushed through
logstash to elasticsearch (but i don't think the format got changed yet though).

There are boards on grafana gunicor-swh-webapp [1], gunicorn-swh-deposit [2] which exploits those logs.

[1] http://kibana0.internal.softwareheritage.org:5601/goto/5242ef5e080731a742603d76e4c8f7d7

[2] http://kibana0.internal.softwareheritage.org:5601/goto/cdce946ede05a52a927415feb74f8284

Django webapp logs are currently not really well managed.
They are currently dumped into a non rotated logfile on moma and only contain info related to requests with errors (meaning general access is not logged at all).
We were considering with @vsellier to redirect those logs into systemd in order to ease their ingestion by Logstash.
Logs format and levels should also be changed in order to track authenticated users access.

Currently, where are the django web app logs stored and for how long are they kept?

@anlambert @vsellier: question about this, in order to document the status quo.
Currently, where are the django web app logs stored and for how long are they kept?

Those flaky tests are hard to reproduce and fix so I think a good workaround would be to enable Test retries in cypress.

In T3202#65231, @rdicosmo wrote:

In T3202#65230, @anlambert wrote:

If we want non staff users to give a try to the tour before official release, we could take advantage of authentication here and activate the guided tour only for users with a dedicated permission.

Please do. We will take advantage of the growing cohort of SWH Ambassadors to get a fresh feedback from real users that are not part of the team, and that come from different backgrounds.

In T3202#65230, @anlambert wrote:

If we want non staff users to give a try to the tour before official release, we could take advantage of authentication here and activate the guided tour only for users with a dedicated permission.

In T3202#65229, @zack wrote:

In T3202#65224, @anlambert wrote:

What I can do is enabling the guided tour by configuration. This way we can deactivate it in production
until we got something stable and usable while we can test the feature on staging.

I suggest to have a round of internal user testing. It has been enabled in staging, if I understand correctly, but nobody in the team knew about it. It would be useful to send out a message to swh-devel (the list), saying it is available there, encouraging trying it out, and gather feedback. Then we can either push in production or iterate a bit more if (pertinent) comments emerge.

It would be even better if we could make non staffers play with it and give feedback, but I don't think that's currently doable due to the VPN and stuff. (That's fine.)

In T3202#65224, @anlambert wrote:

What I can do is enabling the guided tour by configuration. This way we can deactivate it in production
until we got something stable and usable while we can test the feature on staging.

In T3202#65224, @anlambert wrote:

Is the Help page linked from some other place? (i.e.: are we risking 404s if we dump it?)

I mean dumping the link not the page but I could move it in the footer to still reach the page.

Is the Help page linked from some other place? (i.e.: are we risking 404s if we dump it?)

In T3202#65222, @anlambert wrote:

After some brainstorming on the subject, I was thinking to launch the guided tour through the Help link in the left sidebar and thus dump the Help page.

After some brainstorming on the subject, I was thinking to launch the guided tour through the Help link in the left sidebar and thus dump the Help page.

Final step: bump django-webpack-loader in debian/control file for swh-web and check package build still passes.

Packages have been built and uploaded.

I have created the django-webpack-loader repository on the forge with debian branches to build the package on unstable and buster (I reused same configuration as official debian package, no changes were needed apart the changelog).