Page MenuHomeSoftware Heritage
Create Task
Maniphest T4438

Look into triple-stores suitable as swh-search backends
Closed, MigratedEdits Locked
Actions

Description

Currently we rely on ElasticSearch, by embedding JSON-LD documents and treating them as regular JSON when searching.

This causes issues like T2876 and T4396 because ES expects some sort of strict schema, which goes against JSON-LD's design.

Plus, it does not support any sort of relations between documents, which we would need to run queries against related projects (eg. dependency graphs, fork graphs, "related software")

Therefore, I would like to try using a proper triple-store. Virtuoso in particular looks promising, as it support both SPARQL and full-text search.

Related Objects

Event Timeline

vlorentz triaged this task as Normal priority.Aug 16 2022, 10:24 AM
vlorentz created this task.
vlorentz updated the task description. (Show Details)
vlorentz updated the task description. (Show Details)Aug 16 2022, 10:27 AM
vlorentz renamed this task from Look into Virtuoso as a new backend for swh-search to Look into triple-stores suitable as swh-search backends.Aug 16 2022, 2:51 PM
vlorentz updated the task description. (Show Details)Aug 18 2022, 12:29 PM
vlorentz added a comment.Aug 26 2022, 10:36 AM

Useful benchmark tool: https://github.com/oxigraph/oxigraph/tree/main/bench

vlorentz added a comment.Aug 29 2022, 10:05 AM

Another option I am considering if other options do not scale: repurposing swh-graph (especially now that it supports edge labels) as a triple store; but we would need to either reimplement SPARQL on top or use https://github.com/SoftwareHeritage/swh-graph-tinkerpop + https://tinkerpop.apache.org/docs/current/reference/#sparql-gremlin

vlorentz added a comment.Dec 13 2022, 12:05 PM

Actually the open-source version of Virtuoso is unmaintained, and I couldn't figure out how to use it. Blazegraph and Apache Jena look promising too, though.

vlorentz added a subscriber: KShivendu.Dec 13 2022, 12:24 PM

@KShivendu pointed out that Neo4j may be a strong option too. It doesn't natively support SPARQL, but we could have it as a layer on top of the Tinkerpop API

gitlab-migration closed this task as Migrated.Jan 8 2023, 5:04 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal