Page MenuHomeSoftware Heritage
Create Task
Maniphest T2167

Deploy swh-search
Closed, MigratedEdits Locked
Actions

Description

RPC server (which should be runnable the same way as our gunicorn RPC servers) + journal client (with CLI command swh search journal-client objects)

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT1523 Search tools on metadata
 Migratedgitlab-migrationT1117 Origin search is *slow* when you look for very common words
 Migratedgitlab-migrationT1910 Redesign origin search using a dedicated component (swh-search)
 Migratedgitlab-migrationT2182 Switch production swh-web to use swh-search instead of postgresql search.
 Migratedgitlab-migrationT2183 Switch webapp0 to use swh-search instead of postgresql search.
 Migratedgitlab-migrationT2167 Deploy swh-search
 Migratedgitlab-migrationT2174 Add debian package for swh-search
 Migratedgitlab-migrationT2497 Create an ElasticSearch cluster tuned for origin/metadata search
 Migratedgitlab-migrationT2590 Finish the indexer -> swh-search pipeline
 Migratedgitlab-migrationT2651 Make the indexer-storage publish its rows to Kafka
 Migratedgitlab-migrationT2652 Make the indexer-storage interface use attr classes instead of dicts
 Migratedgitlab-migrationT2816 Enable the journal-writer for the swh-idx-storage in staging
 Migratedgitlab-migrationT2817 Enable the swh-search environment in staging
 Migratedgitlab-migrationT2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
 Migratedgitlab-migrationT2904 Create a new production webapp using the frozen index on the staging ES
 Migratedgitlab-migrationT2905 Deploy swh-search for production
 Migratedgitlab-migrationT2936 Update the swh-search journal client to only set "has_visit" on "full" status of the visit
 Migratedgitlab-migrationT2944 Deploy swh-search v0.4.1
 Migratedgitlab-migrationT3037 Reschedule origin-intrinsic-metadata tasks for all origins
 Migratedgitlab-migrationT2780 Enable the journal-writer for the swh-idx-storage in production
 Migratedgitlab-migrationT3040 [production] Enable swh-search's journal-client for indexed objects
 Migratedgitlab-migrationT3041 [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata
 Migratedgitlab-migrationT3373 Metadata search is failing due to a boolean field in the mapping of the metadata fields
 Migratedgitlab-migrationT3391 [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3392 [staging] Properly recreate the origin_intrinsic_metadata topic
 Migratedgitlab-migrationT3398 [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3047 Enable to search in origin metadata with swh-search in webapp
 Migratedgitlab-migrationT3058 Metadata search is failing with "failed to parse date field"
 Migratedgitlab-migrationT3060 Deploy swh-search v0.6.0 in **staging**
 Migratedgitlab-migrationT3076 [swh-search] Improve the index/mapping migration process
 Migratedgitlab-migrationT3083 Deploy swh-search v0.7.0/v0.7.1

Event Timeline

vlorentz removed vlorentz as the assignee of this task.Jan 7 2020, 3:55 PM
vlorentz triaged this task as Normal priority.
vlorentz created this task.
vlorentz mentioned this in D2505: Disable search in swh-web config..Jan 7 2020, 7:18 PM
olasd closed subtask T2174: Add debian package for swh-search as Resolved.Jan 14 2020, 4:52 PM
vlorentz added a parent task: T2183: Switch webapp0 to use swh-search instead of postgresql search..Jan 15 2020, 3:01 PM
ardumont mentioned this in D2578: Deploy swh search on storage0 and make webapp0 use it for search.Jan 23 2020, 5:10 PM
ardumont changed the task status from Open to Work in Progress.Jan 24 2020, 9:37 AM
ardumont claimed this task.
ardumont mentioned this in rSPPRIVCdb05975be07a: sentry: Add swh-search's dsn setup.Jan 24 2020, 1:23 PM
ardumont closed this task as Resolved.Jan 24 2020, 1:46 PM

Deployed.

From triggering a search on webapp0, we have timeout connection from storage0 now.

We can see that in sentry [1]

[1] https://sentry.softwareheritage.org/organizations/swh/issues/?project=15&query=is%3Aunresolved

So that's progress ;)

gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 5:56 PM
gitlab-migration claimed this task.
gitlab-migration added subscribers: ardumont, gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T2174: Add debian package for swh-search from Duplicate to Migrated.Oct 19 2022, 5:56 PM
About · Developer information · Legal