@rdicosmo great summary, I'm certainly on that page :)

thanks !

but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.

Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).

It would be desirable to provide the user with feedback that helps fix the issue.

Thanks. Can you please make a release after landing this, so that docs.s.o gets updated?

Sure! My apologies @Hakimb, but it's thank to your work that we have realized what was the right fate for this task.

@seirl, @vlorentz: I see your point, and I agree. We should never have used /nested/paths for this API.
Maybe we should just reconsider this and, one @Hakimb is ready with a new traversal language proposal, we can map it to a better REST API that uses query parameters, and deal properly with 4xx return codes.

In T2981#63164, @Hakimb wrote:

questions:

1/ So for the "filter that applies to visits that return nodes one by one" part, we are talking about: neighbors, walk, visit/nodes only?

Most of my comments are minor/nice to have, although I'd like to be able to pass queries directly on the CLI.

@vsellier: ack on the outboarding, that is actionable as of now.

Thanks for this!

ok, so @Hakimb: go for no default value. If the query param is not passed, the visit will not stop before the end. If it's given, it will stop once the limit is reached. Call the query param ?max_edges. You will find that the java code already keeps track of the number of edges traversed, so you should just need to compare with that.

To complement what @vlorentz mentioned, we should actually stop the visit after the maximum number of edges has been reached, because it is keep doing the visit (no matter how many results are returned after it) that can DoS the swh-graph backend.

(good catch also for the missing "$@" in the last invocation)

@ardumont we briefly discussed this a while ago with @olasd. I think the proposed solution was indeed to have a separate queue (and workers) for "save code now" request, but not necessarily one separate queue per loader, because the current priority system wasn't considered to be "fast enough". Maybe we can discuss this briefly with him and synthesize here what you come up with?

In D5427#137845, @vlorentz wrote:

I don't think that's good enough. We should have an overview of swh-graph's design that doesn't require reading all the code in an unspecified order.
And reading the code does not give a rationale for the decision.

No, swh identify is correct, as all SWH CLI commands register as sub-commands of the main swh executable.

also, can you add tests verifying that calling the API without an argument does in fact return 400 error?

duplicate with T3160

@anlambert it looks like we're thinking at the same placement for the link that open the permalink box. The main difference seems to be "modal popup" v. "drop-down section" (that makes the rest of the page scroll down). Maybe you can just try both and see what looks best?

Adding both something (the animation) and an optional checkbox to hide (because it is potentially annoying in the long run) does not sound like a great UX.

awesome, thanks @joenio ! you can also drop by our other devel communication channel if you want to discuss this in other ways: https://www.softwareheritage.org/community/developers/

Hey, yes, we want to have one, but nobody is working it at the moment, and we rather have someone knowledgeable with that ecosystem to work on it. So, if you're interested, you're more than welcome to help there! (And thank you in advance.)

reopening, as ideally we'd like to have run the entire ORC export once to completion before closing

Now that this is (optionally) done by swh-web, I don't think we want to implement it in swh-graph too.

While you are at it, and as a minor point, please also double check your commit message, it doesn't match our conventions (e.g., it is in passive voice, while it shouldn't).

Please do not claim tasks @shivam2003, just submit a patch fixing the issue when you have one. Thanks.

@shashikant231 please do not claim tasks. Just submit a diff fixing the issue when you have one. Thanks.

@shashikant231 please do not claim tasks, thanks.

Dear @Kaustuv942, sure, patches welcome. We do not use task claiming for non regular contributors though, just submit a patch when you have one.

In D5213#132637, @KShivendu wrote:

In my opinion, the textual output doesn't print anything other than the directory structure so instead of removing the whole root path we can just put the directory name (it looks better).
[...]
But the ndjson output must be stripped.

In T3111#60314, @anlambert wrote:

Done ! Hopefully teachpress (WP plugin to display publications list) has a bibtex import that works like a charm.

Thanks, this fix looks good.

@vlorentz: can you have a look at this? it's related to the recent changes around the CoreSWHID class, maybe just a release of swh-scanner is missing

please keep the file sorted alphabetically

please keep the file sorted alphabetically