Page MenuHomeSoftware Heritage
Create Task
Maniphest T1910

Redesign origin search using a dedicated component (swh-search)
Closed, MigratedEdits Locked
Actions

Description

Origin search is currently written as a substring/regexp search in postgresql. This is not optimal, and won't work with alternative storage backends (eg. Cassandra)

We should use a component dedicated to text search, eg. ElasticSearch.

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT1523 Search tools on metadata
 Migratedgitlab-migrationT1117 Origin search is *slow* when you look for very common words
 Migratedgitlab-migrationT1910 Redesign origin search using a dedicated component (swh-search)
 Migratedgitlab-migrationT2052 Publish swh-search on PyPI
 Migratedgitlab-migrationT2182 Switch production swh-web to use swh-search instead of postgresql search.
 Migratedgitlab-migrationT2183 Switch webapp0 to use swh-search instead of postgresql search.
 Migratedgitlab-migrationT2167 Deploy swh-search
 Migratedgitlab-migrationT2174 Add debian package for swh-search
 Migratedgitlab-migrationT2184 Replay origins to ElasticSearch's "origin" index
 Migratedgitlab-migrationT2497 Create an ElasticSearch cluster tuned for origin/metadata search
 Migratedgitlab-migrationT2590 Finish the indexer -> swh-search pipeline
 Migratedgitlab-migrationT2651 Make the indexer-storage publish its rows to Kafka
 Migratedgitlab-migrationT2652 Make the indexer-storage interface use attr classes instead of dicts
 Migratedgitlab-migrationT2816 Enable the journal-writer for the swh-idx-storage in staging
 Migratedgitlab-migrationT2817 Enable the swh-search environment in staging
 Migratedgitlab-migrationT2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
 Migratedgitlab-migrationT2904 Create a new production webapp using the frozen index on the staging ES
 Migratedgitlab-migrationT2905 Deploy swh-search for production
 Migratedgitlab-migrationT2936 Update the swh-search journal client to only set "has_visit" on "full" status of the visit
 Migratedgitlab-migrationT2944 Deploy swh-search v0.4.1
 Migratedgitlab-migrationT3037 Reschedule origin-intrinsic-metadata tasks for all origins
 Migratedgitlab-migrationT2780 Enable the journal-writer for the swh-idx-storage in production
 Migratedgitlab-migrationT3040 [production] Enable swh-search's journal-client for indexed objects
 Migratedgitlab-migrationT3041 [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata
 Migratedgitlab-migrationT3373 Metadata search is failing due to a boolean field in the mapping of the metadata fields
 Migratedgitlab-migrationT3391 [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3392 [staging] Properly recreate the origin_intrinsic_metadata topic
 Migratedgitlab-migrationT3398 [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3047 Enable to search in origin metadata with swh-search in webapp
 Migratedgitlab-migrationT3058 Metadata search is failing with "failed to parse date field"
 Migratedgitlab-migrationT3060 Deploy swh-search v0.6.0 in **staging**
 Migratedgitlab-migrationT3076 [swh-search] Improve the index/mapping migration process
 Migratedgitlab-migrationT3083 Deploy swh-search v0.7.0/v0.7.1

Event Timeline

vlorentz triaged this task as Normal priority.Jul 10 2019, 4:56 PM
vlorentz created this task.
vlorentz added a parent task: T1523: Search tools on metadata.Aug 21 2019, 2:34 PM
vlorentz claimed this task.Sep 3 2019, 1:23 PM
vlorentz added a project: Archive search.Oct 28 2019, 11:46 AM
olasd closed subtask T2052: Publish swh-search on PyPI as Resolved.Nov 4 2019, 12:35 PM
vlorentz changed the task status from Open to Work in Progress.Dec 20 2019, 3:26 PM
ardumont renamed this task from Redesign origin search using a dedicated component to Redesign origin search using a dedicated component (swh-search).Jan 24 2020, 9:28 AM
ardumont changed the status of subtask T2167: Deploy swh-search from Open to Work in Progress.Jan 24 2020, 9:38 AM
ardumont closed subtask T2167: Deploy swh-search as Resolved.Jan 24 2020, 1:46 PM
vlorentz mentioned this in T1117: Origin search is *slow* when you look for very common words.Sep 24 2020, 11:05 AM
vlorentz added a parent task: T1117: Origin search is *slow* when you look for very common words.
vsellier changed the status of subtask T2182: Switch production swh-web to use swh-search instead of postgresql search. from Open to Work in Progress.Feb 11 2021, 9:24 AM
vsellier closed subtask T2182: Switch production swh-web to use swh-search instead of postgresql search. as Resolved.Feb 11 2021, 12:10 PM
KShivendu added a subscriber: KShivendu.Mar 4 2021, 7:10 AM
vlorentz closed this task as Resolved.Mar 25 2021, 11:16 AM
gitlab-migration changed the status of subtask T2167: Deploy swh-search from Resolved to Migrated.Oct 19 2022, 5:56 PM
gitlab-migration changed the status of subtask T2182: Switch production swh-web to use swh-search instead of postgresql search. from Resolved to Migrated.
vlorentz added a comment.Nov 2 2022, 11:33 AM

(detaching this task from the Cassandra task tree, because it's getting really big)

vlorentz removed a parent task: T1892: Cassandra as a storage backend.Nov 2 2022, 11:33 AM
gitlab-migration changed the status of subtask T2052: Publish swh-search on PyPI from Resolved to Migrated.Sun, Jan 8, 4:28 PM
gitlab-migration changed the task status from Resolved to Migrated.Sun, Jan 8, 9:59 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal