Page MenuHomeSoftware Heritage
Create Task
Maniphest T1910

Redesign origin search using a dedicated component (swh-search)
Closed, ResolvedPublic
Actions

Description

Origin search is currently written as a substring/regexp search in postgresql. This is not optimal, and won't work with alternative storage backends (eg. Cassandra)

We should use a component dedicated to text search, eg. ElasticSearch.

Related Objects
Search...

StatusAssignedTask
 OpenNoneT2213 Storage
 Work in ProgressvlorentzT2214 Scale-out graph and database storage in production
 OpenvlorentzT1892 Cassandra as a storage backend
 ResolvedvlorentzT2185 Make webapp0 use Cassandra as storage backend.
 ResolvedvlorentzT2186 Merge swh-storage-cassandra in swh-storage master
 OpenNoneT1523 Search tools on metadata
 ResolvedvlorentzT1117 Origin search is *slow* when you look for very common words
 ResolvedvlorentzT1910 Redesign origin search using a dedicated component (swh-search)
 ResolvedolasdT2052 Publish swh-search on PyPI
 ResolvedvsellierT2182 Switch production swh-web to use swh-search instead of postgresql search.
 ResolvedvlorentzT2183 Switch webapp0 to use swh-search instead of postgresql search.
 ResolvedardumontT2167 Deploy swh-search
 DuplicateolasdT2174 Add debian package for swh-search
 ResolvedvlorentzT2184 Replay origins to ElasticSearch's "origin" index
 ResolvedvsellierT2497 Create an ElasticSearch cluster tuned for origin/metadata search
 ResolvedvlorentzT2590 Finish the indexer -> swh-search pipeline
 ResolvedvlorentzT2651 Make the indexer-storage publish its rows to Kafka
 ResolvedvlorentzT2652 Make the indexer-storage interface use attr classes instead of dicts
 ResolvedvsellierT2816 Enable the journal-writer for the swh-idx-storage in staging
 ResolvedvsellierT2817 Enable the swh-search environment in staging
 ResolvedvlorentzT2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
 ResolvedvsellierT2904 Create a new production webapp using the frozen index on the staging ES
 ResolvedvsellierT2905 Deploy swh-search for production
 ResolvedvlorentzT2936 Update the swh-search journal client to only set "has_visit" on "full" status of the visit
 ResolvedvsellierT2944 Deploy swh-search v0.4.1
 ResolvedardumontT3037 Reschedule origin-intrinsic-metadata tasks for all origins
 ResolvedardumontT2780 Enable the journal-writer for the swh-idx-storage in production
 ResolvedvsellierT3040 [production] Enable swh-search's journal-client for indexed objects
 ResolvedvsellierT3041 [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata
 ResolvedvsellierT3373 Metadata search is failing due to a boolean field in the mapping of the metadata fields
 ResolvedvsellierT3391 [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation
 ResolvedvsellierT3392 [staging] Properly recreate the origin_intrinsic_metadata topic
 ResolvedvsellierT3398 [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation
 ResolvedanlambertT3047 Enable to search in origin metadata with swh-search in webapp
 ResolvedvlorentzT3058 Metadata search is failing with "failed to parse date field"
 ResolvedvsellierT3060 Deploy swh-search v0.6.0 in **staging**
 ResolvedvsellierT3076 [swh-search] Improve the index/mapping migration process
 ResolvedvsellierT3083 Deploy swh-search v0.7.0/v0.7.1

Event Timeline

vlorentz triaged this task as Normal priority.Jul 10 2019, 4:56 PM
vlorentz created this task.
vlorentz added a parent task: T1523: Search tools on metadata.Aug 21 2019, 2:34 PM
vlorentz claimed this task.Sep 3 2019, 1:23 PM
vlorentz added a project: Archive search.Oct 28 2019, 11:46 AM
olasd closed subtask T2052: Publish swh-search on PyPI as Resolved.Nov 4 2019, 12:35 PM
vlorentz changed the task status from Open to Work in Progress.Dec 20 2019, 3:26 PM
ardumont renamed this task from Redesign origin search using a dedicated component to Redesign origin search using a dedicated component (swh-search).Jan 24 2020, 9:28 AM
ardumont changed the status of subtask T2167: Deploy swh-search from Open to Work in Progress.Jan 24 2020, 9:38 AM
ardumont closed subtask T2167: Deploy swh-search as Resolved.Jan 24 2020, 1:46 PM
vlorentz mentioned this in T1117: Origin search is *slow* when you look for very common words.Sep 24 2020, 11:05 AM
vlorentz added a parent task: T1117: Origin search is *slow* when you look for very common words.
vsellier changed the status of subtask T2182: Switch production swh-web to use swh-search instead of postgresql search. from Open to Work in Progress.Feb 11 2021, 9:24 AM
vsellier closed subtask T2182: Switch production swh-web to use swh-search instead of postgresql search. as Resolved.Feb 11 2021, 12:10 PM
KShivendu added a subscriber: KShivendu.Mar 4 2021, 7:10 AM
vlorentz closed this task as Resolved.Mar 25 2021, 11:16 AM
About · Developer information · Legal