(detaching this task from the Cassandra task tree, because it's getting really big)
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Nov 2 2022
Nov 2 2022
Oct 27 2022
Oct 27 2022
Oct 26 2022
Oct 26 2022
Oct 25 2022
Oct 25 2022
Oct 24 2022
Oct 24 2022
vlorentz edited projects for T4654: swh-indexer produces dates not supported by swh-search/ElasticSearch, added: Archive search; removed Restricted Project.
Oct 21 2022
Oct 21 2022
jayeshv closed T4613: Generalize and simplify the query language, a subtask of T3952: Make the search query language a first class citizen , as Invalid.
Oct 19 2022
Oct 19 2022
gitlab-migration changed the status of T4614: Deploy swh-search v0.16.4, a subtask of T4599: Github descriptions are not used to search origins, from Resolved to Migrated.
gitlab-migration changed the status of T4431: Error from swh-search server was not logged/reported by the client from Invalid to Migrated.
gitlab-migration closed T4251: [swh-search] Investigate long search queries response time as Migrated.
gitlab-migration changed the status of T3944: Deploy swh-search v0.13.0, a subtask of T3930: Update swh-search QL examples, from Resolved to Migrated.
gitlab-migration changed the status of T3944: Deploy swh-search v0.13.0, a subtask of T3927: swh-search crashes when "only show origins visited at least once" is not checked, from Resolved to Migrated.
gitlab-migration changed the status of T3943: rebuild python3-tree-sitter for python 3.10, a subtask of T3944: Deploy swh-search v0.13.0, from Resolved to Migrated.
gitlab-migration changed the status of T3943: rebuild python3-tree-sitter for python 3.10 from Resolved to Migrated.
gitlab-migration closed T3559: Enable the swh-search QL in staging, a subtask of T3558: Enable the swh-search QL in production, as Migrated.
gitlab-migration closed T3558: Enable the swh-search QL in production, a subtask of T3952: Make the search query language a first class citizen , as Migrated.
gitlab-migration changed the status of T3484: Fix the release builds for swh-search, a subtask of T3433: Deploy swh.search v0.10/v0.11, from Resolved to Migrated.
gitlab-migration changed the status of T3484: Fix the release builds for swh-search from Resolved to Migrated.
gitlab-migration changed the status of T3433: Deploy swh.search v0.10/v0.11 from Resolved to Migrated.
gitlab-migration changed the status of T3398: [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation, a subtask of T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields, from Resolved to Migrated.
gitlab-migration changed the status of T3398: [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation from Resolved to Migrated.
gitlab-migration changed the status of T3392: [staging] Properly recreate the origin_intrinsic_metadata topic, a subtask of T3391: [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation, from Resolved to Migrated.
gitlab-migration changed the status of T3391: [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation, a subtask of T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields, from Resolved to Migrated.
gitlab-migration changed the status of T3391: [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation from Resolved to Migrated.
gitlab-migration changed the status of T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields, a subtask of T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata, from Resolved to Migrated.
gitlab-migration changed the status of T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields from Resolved to Migrated.
gitlab-migration changed the status of T3083: Deploy swh-search v0.7.0/v0.7.1, a subtask of T3076: [swh-search] Improve the index/mapping migration process, from Resolved to Migrated.
gitlab-migration changed the status of T3083: Deploy swh-search v0.7.0/v0.7.1 from Resolved to Migrated.
gitlab-migration changed the status of T3076: [swh-search] Improve the index/mapping migration process, a subtask of T2590: Finish the indexer -> swh-search pipeline, from Resolved to Migrated.
gitlab-migration changed the status of T3076: [swh-search] Improve the index/mapping migration process from Resolved to Migrated.
gitlab-migration changed the status of T3067: elasticsearch cluster disk usage and maintenance from Resolved to Migrated.
gitlab-migration changed the status of T3060: Deploy swh-search v0.6.0 in **staging** from Resolved to Migrated.
gitlab-migration changed the status of T3060: Deploy swh-search v0.6.0 in **staging**, a subtask of T3058: Metadata search is failing with "failed to parse date field", from Resolved to Migrated.
gitlab-migration changed the status of T3060: Deploy swh-search v0.6.0 in **staging**, a subtask of T2590: Finish the indexer -> swh-search pipeline, from Resolved to Migrated.
gitlab-migration changed the status of T3042: swh-search: add statsd/prometheus metrics from Resolved to Migrated.
gitlab-migration changed the status of T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata from Resolved to Migrated.
gitlab-migration changed the status of T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata, a subtask of T3040: [production] Enable swh-search's journal-client for indexed objects, from Resolved to Migrated.
gitlab-migration changed the status of T3040: [production] Enable swh-search's journal-client for indexed objects, a subtask of T2590: Finish the indexer -> swh-search pipeline, from Resolved to Migrated.
gitlab-migration changed the status of T3040: [production] Enable swh-search's journal-client for indexed objects from Resolved to Migrated.
gitlab-migration changed the status of T3037: Reschedule origin-intrinsic-metadata tasks for all origins, a subtask of T2590: Finish the indexer -> swh-search pipeline, from Resolved to Migrated.
gitlab-migration changed the status of T3037: Reschedule origin-intrinsic-metadata tasks for all origins from Resolved to Migrated.
gitlab-migration changed the status of T2944: Deploy swh-search v0.4.1, a subtask of T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit, from Resolved to Migrated.
gitlab-migration changed the status of T2905: Deploy swh-search for production, a subtask of T2182: Switch production swh-web to use swh-search instead of postgresql search., from Resolved to Migrated.
gitlab-migration changed the status of T2905: Deploy swh-search for production, a subtask of T2904: Create a new production webapp using the frozen index on the staging ES, from Resolved to Migrated.
gitlab-migration changed the status of T2905: Deploy swh-search for production from Resolved to Migrated.
gitlab-migration changed the status of T2780: Enable the journal-writer for the swh-idx-storage in production, a subtask of T3037: Reschedule origin-intrinsic-metadata tasks for all origins, from Resolved to Migrated.
gitlab-migration changed the status of T2780: Enable the journal-writer for the swh-idx-storage in production, a subtask of T2590: Finish the indexer -> swh-search pipeline, from Resolved to Migrated.
gitlab-migration changed the status of T2497: Create an ElasticSearch cluster tuned for origin/metadata search, a subtask of T2182: Switch production swh-web to use swh-search instead of postgresql search., from Resolved to Migrated.
gitlab-migration changed the status of T2497: Create an ElasticSearch cluster tuned for origin/metadata search from Resolved to Migrated.
gitlab-migration changed the status of T2182: Switch production swh-web to use swh-search instead of postgresql search. from Resolved to Migrated.
gitlab-migration changed the status of T2182: Switch production swh-web to use swh-search instead of postgresql search., a subtask of T1910: Redesign origin search using a dedicated component (swh-search), from Resolved to Migrated.
gitlab-migration changed the status of T2174: Add debian package for swh-search from Duplicate to Migrated.
gitlab-migration changed the status of T2174: Add debian package for swh-search, a subtask of T2167: Deploy swh-search, from Duplicate to Migrated.
gitlab-migration changed the status of T2167: Deploy swh-search, a subtask of T2183: Switch webapp0 to use swh-search instead of postgresql search., from Resolved to Migrated.
gitlab-migration changed the status of T2167: Deploy swh-search, a subtask of T1910: Redesign origin search using a dedicated component (swh-search), from Resolved to Migrated.
Oct 18 2022
Oct 18 2022
I just checked with another project, and it is indeed in the journal:
Bisecting, it seems the metadata is lost somewhere between the indexer storage:
Oct 17 2022
Oct 17 2022
ardumont closed T4614: Deploy swh-search v0.16.4, a subtask of T4599: Github descriptions are not used to search origins, as Resolved.
ardumont moved T4614: Deploy swh-search v0.16.4 from in-progress to deployed/landed/monitoring on the System administration board.
ardumont changed the status of T4614: Deploy swh-search v0.16.4, a subtask of T4599: Github descriptions are not used to search origins, from Open to Work in Progress.
ardumont moved T4614: Deploy swh-search v0.16.4 from Backlog to Weekly backlog on the System administration board.
Oct 13 2022
Oct 13 2022
In T4627#93288, @KShivendu wrote:P1090 should be helpful
iirc, we made some changes even after this. But the final version shouldn't be too different from this :)
P1090 should be helpful
Oct 12 2022
Oct 12 2022
anlambert renamed T4614: Deploy swh-search v0.16.4 from Deploy swh-search v0.16.3 to Deploy swh-search v0.16.4.
Oct 10 2022
Oct 10 2022
What about having a UI like in Github or Phabricator to create an advanced query?
eg:
https://github.com/search/advanced
Oct 7 2022
Oct 7 2022
Oct 5 2022
Oct 5 2022
Please keep it open since the search isn't fast enough at the moment.
Oct 4 2022
Oct 4 2022
@KShivendu Any news regarding these profiling ?
Oct 3 2022
Oct 3 2022
vlorentz lowered the priority of T4599: Github descriptions are not used to search origins from High to Normal.
Aug 29 2022
Aug 29 2022
Another option I am considering if other options do not scale: repurposing swh-graph (especially now that it supports edge labels) as a triple store; but we would need to either reimplement SPARQL on top or use https://github.com/SoftwareHeritage/swh-graph-tinkerpop + https://tinkerpop.apache.org/docs/current/reference/#sparql-gremlin
Aug 26 2022
Aug 26 2022
Useful benchmark tool: https://github.com/oxigraph/oxigraph/tree/main/bench
Aug 25 2022
Aug 25 2022
vlorentz updated the task description for T4249: Choose/define an ontology to use for indexed extrinsic origin metadata.
Aug 23 2022
Aug 23 2022
vlorentz updated the task description for T4249: Choose/define an ontology to use for indexed extrinsic origin metadata.
vlorentz updated the task description for T4249: Choose/define an ontology to use for indexed extrinsic origin metadata.
vlorentz updated the task description for T4249: Choose/define an ontology to use for indexed extrinsic origin metadata.