Page MenuHomeSoftware Heritage
Create Task
Maniphest T2590

Finish the indexer -> swh-search pipeline
Closed, MigratedEdits Locked
Actions

Description

I wrote D1959 and D2027 a year ago, they need to be rebased and merge so we can really use swh-search for the archive search in swh-web.

Revisions and Commits

rCDFJ Dockerfiles for Jenkins
D4675rCDFJ32b720b50619 base-buster: Pin elasticsearch to 7.9.3
rDSEA Archive search
D4661rDSEA27f82bafeaa4 cli: Subscribe journal client to origin_intrinsic_metadata topic
D4671rDSEA94d305509e7b cli: Subscribe journal client to origin_visit_status
D4670rDSEA408d51ef5713 cli: Allow topic prefix declaration through cli or configuration
D4669rDSEA8ffcec124b73 cli: Allow object-type declaration through cli or configuration
rDENV Development environment
D4709rDENVf5d483c14997 indexer_storage: Publish indexer computation to journal topics

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT1523 Search tools on metadata
 Migratedgitlab-migrationT1117 Origin search is *slow* when you look for very common words
 Migratedgitlab-migrationT1910 Redesign origin search using a dedicated component (swh-search)
 Migratedgitlab-migrationT2052 Publish swh-search on PyPI
 Migratedgitlab-migrationT2167 Deploy swh-search
 Migratedgitlab-migrationT2174 Add debian package for swh-search
 Migratedgitlab-migrationT2182 Switch production swh-web to use swh-search instead of postgresql search.
 Migratedgitlab-migrationT2590 Finish the indexer -> swh-search pipeline
 Migratedgitlab-migrationT2651 Make the indexer-storage publish its rows to Kafka
 Migratedgitlab-migrationT2652 Make the indexer-storage interface use attr classes instead of dicts
 Migratedgitlab-migrationT2816 Enable the journal-writer for the swh-idx-storage in staging
 Migratedgitlab-migrationT2817 Enable the swh-search environment in staging
 Migratedgitlab-migrationT2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
 Migratedgitlab-migrationT2904 Create a new production webapp using the frozen index on the staging ES
 Migratedgitlab-migrationT2905 Deploy swh-search for production
 Migratedgitlab-migrationT2936 Update the swh-search journal client to only set "has_visit" on "full" status of the visit
 Migratedgitlab-migrationT2944 Deploy swh-search v0.4.1
 Migratedgitlab-migrationT3037 Reschedule origin-intrinsic-metadata tasks for all origins
 Migratedgitlab-migrationT2780 Enable the journal-writer for the swh-idx-storage in production
 Migratedgitlab-migrationT3040 [production] Enable swh-search's journal-client for indexed objects
 Migratedgitlab-migrationT3041 [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata
 Migratedgitlab-migrationT3373 Metadata search is failing due to a boolean field in the mapping of the metadata fields
 Migratedgitlab-migrationT3391 [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3392 [staging] Properly recreate the origin_intrinsic_metadata topic
 Migratedgitlab-migrationT3398 [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3047 Enable to search in origin metadata with swh-search in webapp
 Migratedgitlab-migrationT3058 Metadata search is failing with "failed to parse date field"
 Migratedgitlab-migrationT3060 Deploy swh-search v0.6.0 in **staging**
 Migratedgitlab-migrationT3076 [swh-search] Improve the index/mapping migration process
 Migratedgitlab-migrationT3083 Deploy swh-search v0.7.0/v0.7.1

Event Timeline

vlorentz triaged this task as Normal priority.Sep 14 2020, 5:39 PM
vlorentz created this task.
vlorentz edited projects, added Journal; removed Storage manager.
vlorentz closed subtask T2651: Make the indexer-storage publish its rows to Kafka as Resolved.Nov 16 2020, 1:27 PM
vsellier changed the status of subtask T2817: Enable the swh-search environment in staging from Open to Work in Progress.Nov 26 2020, 5:59 PM
vsellier added a subscriber: vsellier.EditedNov 27 2020, 10:14 AM

this a description of the pipeline to clarify the interaction between the components (source: P883) :

Origin visit - Indexer - swh search pipeline.svg14 KBDownload

PS1: currently, the search.journalclien's code is only listening origin and origin_visit topics, but it seems the everything is almost ready for intinsic_metadata
PS2: The swh.journal.indexed.origin_intrinsic topic name is the one @vlorentz has suggested in T2780

vsellier closed subtask T2816: Enable the journal-writer for the swh-idx-storage in staging as Resolved.Nov 27 2020, 6:20 PM
vlorentz added a revision: D4661: search.cli: Subscribe journal client to origin_intrinsic_metadata topic.Dec 4 2020, 1:41 PM
ardumont added a revision: D4669: cli: Allow object-type declaration through cli or configuration.Dec 7 2020, 8:52 AM
ardumont added a revision: D4670: cli: Allow topic prefix declaration through cli or configuration.
ardumont added a revision: D4671: cli: Subscribe journal client to origin_visit_status.Dec 7 2020, 8:54 AM
ardumont added a revision: D4675: base-buster: Pin elasticsearch to 7.9.3.Dec 7 2020, 1:29 PM
ardumont added a commit: rCDFJ32b720b50619: base-buster: Pin elasticsearch to 7.9.3.Dec 7 2020, 2:08 PM
ardumont added a commit: rDSEA8ffcec124b73: cli: Allow object-type declaration through cli or configuration.Dec 8 2020, 9:48 AM
ardumont added a commit: rDSEA408d51ef5713: cli: Allow topic prefix declaration through cli or configuration.
ardumont added a commit: rDSEA94d305509e7b: cli: Subscribe journal client to origin_visit_status.Dec 8 2020, 9:52 AM
ardumont added a commit: rDSEA27f82bafeaa4: cli: Subscribe journal client to origin_intrinsic_metadata topic.Dec 8 2020, 9:59 AM
ardumont added a revision: D4709: indexer_storage: Publish indexer computation to journal topics.Dec 10 2020, 9:43 AM
ardumont added a commit: rDENVf5d483c14997: indexer_storage: Publish indexer computation to journal topics.Dec 10 2020, 9:50 AM
vsellier added a subtask: T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types.Dec 10 2020, 12:31 PM
vsellier closed subtask T2817: Enable the swh-search environment in staging as Resolved.Dec 10 2020, 3:29 PM
vlorentz closed subtask T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types as Resolved.Dec 14 2020, 10:54 AM
vlorentz reopened subtask T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types as Open.Dec 14 2020, 10:56 AM
vlorentz closed subtask T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types as Resolved.Dec 22 2020, 10:52 AM
vsellier changed the status of subtask T2904: Create a new production webapp using the frozen index on the staging ES from Open to Work in Progress.Dec 23 2020, 10:04 AM
vsellier closed subtask T2904: Create a new production webapp using the frozen index on the staging ES as Resolved.Dec 23 2020, 5:42 PM
vlorentz mentioned this in D2027: Add support for origin_intrinsic_metadata in the journal client..Jan 29 2021, 3:24 PM
ardumont changed the status of subtask T2780: Enable the journal-writer for the swh-idx-storage in production from Open to Work in Progress.Feb 9 2021, 5:51 PM
ardumont changed the status of subtask T3037: Reschedule origin-intrinsic-metadata tasks for all origins from Open to Work in Progress.Feb 11 2021, 9:41 AM
ardumont closed subtask T2780: Enable the journal-writer for the swh-idx-storage in production as Resolved.Feb 11 2021, 9:48 AM
vlorentz changed the task status from Open to Work in Progress.Feb 11 2021, 11:01 AM
ardumont closed subtask T3037: Reschedule origin-intrinsic-metadata tasks for all origins as Resolved.Feb 12 2021, 6:32 PM
anlambert closed subtask T3047: Enable to search in origin metadata with swh-search in webapp as Resolved.Feb 17 2021, 12:12 PM
vlorentz closed subtask T3058: Metadata search is failing with "failed to parse date field" as Resolved.Feb 18 2021, 12:13 PM
vsellier created subtask T3060: Deploy swh-search v0.6.0 in **staging**.Feb 18 2021, 3:41 PM
vsellier closed subtask T3060: Deploy swh-search v0.6.0 in **staging** as Resolved.Mar 1 2021, 10:55 AM
vsellier changed the status of subtask T3076: [swh-search] Improve the index/mapping migration process from Open to Work in Progress.Mar 1 2021, 3:54 PM
vsellier closed subtask T3076: [swh-search] Improve the index/mapping migration process as Resolved.Mar 5 2021, 10:35 AM
vlorentz added a parent task: T1117: Origin search is *slow* when you look for very common words.Apr 1 2021, 10:51 AM
vsellier closed subtask T3040: [production] Enable swh-search's journal-client for indexed objects as Resolved.Sep 8 2021, 3:24 PM
vlorentz closed this task as Resolved.Sep 8 2021, 3:35 PM
gitlab-migration changed the status of subtask T2780: Enable the journal-writer for the swh-idx-storage in production from Resolved to Migrated.Oct 19 2022, 6:00 PM
gitlab-migration changed the status of subtask T3037: Reschedule origin-intrinsic-metadata tasks for all origins from Resolved to Migrated.
gitlab-migration changed the status of subtask T3040: [production] Enable swh-search's journal-client for indexed objects from Resolved to Migrated.
gitlab-migration changed the status of subtask T3060: Deploy swh-search v0.6.0 in **staging** from Resolved to Migrated.
gitlab-migration changed the status of subtask T3076: [swh-search] Improve the index/mapping migration process from Resolved to Migrated.
gitlab-migration changed the status of subtask T2651: Make the indexer-storage publish its rows to Kafka from Resolved to Migrated.Jan 8 2023, 4:31 PM
gitlab-migration changed the status of subtask T3058: Metadata search is failing with "failed to parse date field" from Resolved to Migrated.
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 10:01 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T2816: Enable the journal-writer for the swh-idx-storage in staging from Resolved to Migrated.Jan 8 2023, 10:01 PM
gitlab-migration changed the status of subtask T2817: Enable the swh-search environment in staging from Resolved to Migrated.
gitlab-migration changed the status of subtask T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types from Resolved to Migrated.
gitlab-migration changed the status of subtask T2904: Create a new production webapp using the frozen index on the staging ES from Resolved to Migrated.
gitlab-migration changed the status of subtask T3047: Enable to search in origin metadata with swh-search in webapp from Resolved to Migrated.
About · Developer information · Legal