In T2265#63605, @anlambert wrote:
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Apr 19 2021
Apr 19 2021
Apr 17 2021
Apr 17 2021
zack committed rDDATASET34da22d6d14e: doc: fix sphinx line continuation glitches in schema.rst (authored by zack).
doc: fix sphinx line continuation glitches in schema.rst
I'm confused about the status of this task. I've just rebuilt the docs for docs.s.o and it says "build succeeded, 83 warnings.". So is the fix for this not yet "deployed" somehow? Also, why is the build succeeding even with all those warnings? Because as long as that's the case, we will for sure keep reintroducing warnings as time goes by.
Apr 16 2021
Apr 16 2021
@rdicosmo great summary, I'm certainly on that page :)
thanks !
but adding an email field (auto filled for registered users) to send a notification after the origin was loaded seems a good tradeoff. To implement the email notification, we will have to add a journal client in swh-web processing origin visit messages.
Apr 15 2021
Apr 15 2021
Oh, and now that we have user profile pages, we should have a list of "my" save code now requests with their status visible in the user profile, for those who want to check synchronously the status of their requests (and might have disabled email notifications).
It would be desirable to provide the user with feedback that helps fix the issue.
Thanks. Can you please make a release after landing this, so that docs.s.o gets updated?
Apr 14 2021
Apr 14 2021
zack closed T1968: existing graph endpoints should not return 404 upon missing arguments as Invalid.
Sure! My apologies @Hakimb, but it's thank to your work that we have realized what was the right fate for this task.
zack updated subscribers of T1968: existing graph endpoints should not return 404 upon missing arguments.
@seirl, @vlorentz: I see your point, and I agree. We should never have used /nested/paths for this API.
Maybe we should just reconsider this and, one @Hakimb is ready with a new traversal language proposal, we can map it to a better REST API that uses query parameters, and deal properly with 4xx return codes.
In T2981#63164, @Hakimb wrote:questions:
1/ So for the "filter that applies to visits that return nodes one by one" part, we are talking about: neighbors, walk, visit/nodes only?
Most of my comments are minor/nice to have, although I'd like to be able to pass queries directly on the CLI.
Apr 12 2021
Apr 12 2021
@vsellier: ack on the outboarding, that is actionable as of now.
Thanks for this!
Apr 8 2021
Apr 8 2021
ok, so @Hakimb: go for no default value. If the query param is not passed, the visit will not stop before the end. If it's given, it will stop once the limit is reached. Call the query param ?max_edges. You will find that the java code already keeps track of the number of edges traversed, so you should just need to compare with that.
To complement what @vlorentz mentioned, we should actually stop the visit after the maximum number of edges has been reached, because it is keep doing the visit (no matter how many results are returned after it) that can DoS the swh-graph backend.
Apr 7 2021
Apr 7 2021
(good catch also for the missing "$@" in the last invocation)
@ardumont we briefly discussed this a while ago with @olasd. I think the proposed solution was indeed to have a separate queue (and workers) for "save code now" request, but not necessarily one separate queue per loader, because the current priority system wasn't considered to be "fast enough". Maybe we can discuss this briefly with him and synthesize here what you come up with?
zack added a comment to D5427: NodeIdMap: use the MPH + mmapped .order to translate SWHID -> node ID.
In D5427#137845, @vlorentz wrote:I don't think that's good enough. We should have an overview of swh-graph's design that doesn't require reading all the code in an unspecified order.
And reading the code does not give a rationale for the decision.
Apr 6 2021
Apr 6 2021
zack committed rMSLD93db6cff6c7b: swh-scanner talk: add links to code and pypi package (authored by zack).
swh-scanner talk: add links to code and pypi package
minor changes and updates for LLW 2021 talk
No, swh identify is correct, as all SWH CLI commands register as sub-commands of the main swh executable.
also, can you add tests verifying that calling the API without an argument does in fact return 400 error?
duplicate with T3160
check-in slide skeleton for LLW 2021
Apr 2 2021
Apr 2 2021
zack added a reviewer for D5411: return a 400 error when accessing endpoints without the arguments: seirl.
@anlambert it looks like we're thinking at the same placement for the link that open the permalink box. The main difference seems to be "modal popup" v. "drop-down section" (that makes the rest of the page scroll down). Maybe you can just try both and see what looks best?
Apr 1 2021
Apr 1 2021
Adding both something (the animation) and an optional checkbox to hide (because it is potentially annoying in the long run) does not sound like a great UX.
Mar 31 2021
Mar 31 2021
docs: drop mention of conffile in quickstart
Mar 30 2021
Mar 30 2021
awesome, thanks @joenio ! you can also drop by our other devel communication channel if you want to discuss this in other ways: https://www.softwareheritage.org/community/developers/
zack renamed T2833: cpan.loader - archive Perl modules from CPAN from [feature request] cpan.loader - preserver Perl modules from CPAN to cpan.loader - preserver Perl modules from CPAN.
zack raised the priority of T2833: cpan.loader - archive Perl modules from CPAN from Wishlist to Normal.
Hey, yes, we want to have one, but nobody is working it at the moment, and we rather have someone knowledgeable with that ecosystem to work on it. So, if you're interested, you're more than welcome to help there! (And thank you in advance.)
Mar 29 2021
Mar 29 2021
CYU talk: use more recent data model slide
check in slides for tomorrow talk at CYU
Mar 27 2021
Mar 27 2021
add recent papers
zack committed R183:04b760d62231: add citation for Apache Gremlin graph traversal language (authored by zack).
add citation for Apache Gremlin graph traversal language
Mar 26 2021
Mar 26 2021
zack reopened T1847: fully automate export of the graph dataset, a subtask of T1848: refresh graph dataset export, as Open.
reopening, as ideally we'd like to have run the entire ORC export once to completion before closing
Mar 23 2021
Mar 23 2021
zack added a project to T3168: Proper deployment of swh-graph with debian package: Compressed graph service.
Mar 22 2021
Mar 22 2021
zack renamed T3161: graph service: add anti-DoS limit on the number of edges traversed from graph service: add limit on the number of edges traversed to graph service: add anti-DoS limit on the number of edges traversed.
zack added a parent task for T3161: graph service: add anti-DoS limit on the number of edges traversed: T2220: swh-graph in production.
zack triaged T3161: graph service: add anti-DoS limit on the number of edges traversed as Normal priority.
Now that this is (optionally) done by swh-web, I don't think we want to implement it in swh-graph too.
Mar 21 2021
Mar 21 2021
While you are at it, and as a minor point, please also double check your commit message, it doesn't match our conventions (e.g., it is in passive voice, while it shouldn't).
Mar 20 2021
Mar 20 2021
LibrePlanet talk: last touches
zack renamed T3160: swh identify: add a -R/--recursive flag from swh identify: add a -R/--recursive to swh identify: add a -R/--recursive flag.
Mar 19 2021
Mar 19 2021
check in slides for LibrePlanet 2021
Please do not claim tasks @shivam2003, just submit a patch fixing the issue when you have one. Thanks.
common: add SwhFS ICSE paper to biblio module
zack committed rMSLDdcf96f56494d: common: revamp some old/common slides to reflect current state (authored by zack).
common: revamp some old/common slides to reflect current state
zack committed rMSLDcd8af720ce3c: common: add swh identify tutorial/example to SWHID module (authored by zack).
common: add swh identify tutorial/example to SWHID module
zack committed rMSLDc2d00871acb0: common: add (minimal) slide module for swh-fuse (authored by zack).
common: add (minimal) slide module for swh-fuse
zack committed rMSLDe9f19e6288df: common: add one-slider module about the Merkle structure (authored by zack).
common: add one-slider module about the Merkle structure
zack committed rMSLDb6c2a59d3dc5: common/images: add archive coverage image + links for coverage & growth (authored by zack).
common/images: add archive coverage image + links for coverage & growth
doc: capitalize heading title
zack committed rDDOC54fe755ea8a9: make heading for swh-loader page consisted with other packages (authored by zack).
make heading for swh-loader page consisted with other packages
Mar 18 2021
Mar 18 2021
Mar 16 2021
Mar 16 2021
@shashikant231 please do not claim tasks. Just submit a diff fixing the issue when you have one. Thanks.
@shashikant231 please do not claim tasks, thanks.
zack placed T1968: existing graph endpoints should not return 404 upon missing arguments up for grabs.
Dear @Kaustuv942, sure, patches welcome. We do not use task claiming for non regular contributors though, just submit a patch when you have one.
Mar 15 2021
Mar 15 2021
zack committed rDGRPH58b46f78ee3f: FindEarliestRevision: bug fix: do not follow rev:rev edges (authored by zack).
FindEarliestRevision: bug fix: do not follow rev:rev edges
Mar 14 2021
Mar 14 2021
zack triaged T3126: API: add endpoint to find the earliest revision referencing a dir/cnt node as Normal priority.
Mar 13 2021
Mar 13 2021
Mar 11 2021
Mar 11 2021
zack committed rDGRPHe0ef3b9b124b: FindEarliestRevision: make it work as a *nix filter and add accounting (authored by zack).
FindEarliestRevision: make it work as a *nix filter and add accounting
Mar 10 2021
Mar 10 2021
In D5213#132637, @KShivendu wrote:In my opinion, the textual output doesn't print anything other than the directory structure so instead of removing the whole root path we can just put the directory name (it looks better).
[...]
But the ndjson output must be stripped.
In T3111#60314, @anlambert wrote:Done ! Hopefully teachpress (WP plugin to display publications list) has a bibtex import that works like a charm.
Thanks, this fix looks good.
Mar 8 2021
Mar 8 2021
@vlorentz: can you have a look at this? it's related to the recent changes around the CoreSWHID class, maybe just a release of swh-scanner is missing
Mar 6 2021
Mar 6 2021