Page MenuHomeSoftware Heritage
Create Task
Maniphest T2944

Deploy swh-search v0.4.1
Closed, MigratedEdits Locked
Actions

Description

Deploy the new 0.4.1 version with the fix for the has_visit status.

According D4818, the basic migration can be done like this :

Migration:

  • curl -XPOST "https://localhost:9200/origin/_update_by_query" -d'{"script": {"inline": "ctx._source.has_visits = false", "lang": "painless"}}'
  • reset the offset of the swh-search client on the swh.journal.objects.origin_visit_status topic

This is correct, but doing like this, the search will not be completely accurate until the backfill of the origin_visit_status topic is done.
It's not important currently as the swh-search is not really used in production, but it's the opportunity to test a way to do such reindexation smoothly.

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT1523 Search tools on metadata
 Migratedgitlab-migrationT1117 Origin search is *slow* when you look for very common words
 Migratedgitlab-migrationT1910 Redesign origin search using a dedicated component (swh-search)
 Migratedgitlab-migrationT2182 Switch production swh-web to use swh-search instead of postgresql search.
 Migratedgitlab-migrationT2590 Finish the indexer -> swh-search pipeline
 Migratedgitlab-migrationT2651 Make the indexer-storage publish its rows to Kafka
 Migratedgitlab-migrationT2652 Make the indexer-storage interface use attr classes instead of dicts
 Migratedgitlab-migrationT2816 Enable the journal-writer for the swh-idx-storage in staging
 Migratedgitlab-migrationT2817 Enable the swh-search environment in staging
 Migratedgitlab-migrationT2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
 Migratedgitlab-migrationT2904 Create a new production webapp using the frozen index on the staging ES
 Migratedgitlab-migrationT2905 Deploy swh-search for production
 Migratedgitlab-migrationT2936 Update the swh-search journal client to only set "has_visit" on "full" status of the visit
 Migratedgitlab-migrationT2944 Deploy swh-search v0.4.1
 Migratedgitlab-migrationT3037 Reschedule origin-intrinsic-metadata tasks for all origins
 Migratedgitlab-migrationT2780 Enable the journal-writer for the swh-idx-storage in production
 Migratedgitlab-migrationT3040 [production] Enable swh-search's journal-client for indexed objects
 Migratedgitlab-migrationT3041 [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata
 Migratedgitlab-migrationT3373 Metadata search is failing due to a boolean field in the mapping of the metadata fields
 Migratedgitlab-migrationT3391 [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3392 [staging] Properly recreate the origin_intrinsic_metadata topic
 Migratedgitlab-migrationT3398 [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3047 Enable to search in origin metadata with swh-search in webapp
 Migratedgitlab-migrationT3058 Metadata search is failing with "failed to parse date field"
 Migratedgitlab-migrationT3060 Deploy swh-search v0.6.0 in **staging**
 Migratedgitlab-migrationT3076 [swh-search] Improve the index/mapping migration process
 Migratedgitlab-migrationT3083 Deploy swh-search v0.7.0/v0.7.1

Event Timeline

vsellier triaged this task as Normal priority.Jan 7 2021, 6:39 PM
vsellier created this task.
ardumont renamed this task from Deploy T2936 in staging to Deploy swh-search v0.4.1 in staging.Jan 11 2021, 10:49 AM
ardumont moved this task from Backlog to Weekly backlog on the System administration board.
vlorentz reassigned this task from vlorentz to vsellier.Jan 11 2021, 1:48 PM
vsellier mentioned this in T2978: Deploy visit-stats journal client on staging.Jan 22 2021, 11:12 AM
vsellier renamed this task from Deploy swh-search v0.4.1 in staging to Deploy swh-search v0.4.1.Jan 25 2021, 3:32 PM
vsellier changed the task status from Open to Work in Progress.
vsellier moved this task from Weekly backlog to in-progress on the System administration board.
vsellier added a comment.Jan 25 2021, 4:07 PM

Regarding the index rebuilding process, using a naive approach with aliases with the old and the new index[1] returns duplicated results when the search is done.
Using an alias with only the old index, rebuilding a new index and switching the alias to the new index[2] can be a first approach with the default the old index will not be updated until the alias is switched to the new index.
It also requires the swh-search code is able to use different names for the read and write operations.

[1]


[2]

vsellier added a comment.EditedJan 25 2021, 5:44 PM

Staging

We are proceeding to a complete index rebuilding

  • debian packages upgraded on search0.staging
  • journal client stopped
  • 'origin' index removed on elasticsearch and recreated
% curl -XDELETE http://$ES_SERVER/origin  
 % sudo swh search --config-file /etc/softwareheritage/search/server.yml  initialize
  • Journal client offset reset :
export SERVER=journal0.internal.staging.swh.network:9092
./kafka-consumer-groups.sh --bootstrap-server $SERVER --delete --group swh.search.journal_client 
Deletion of requested consumer groups ('swh.search.journal_client') was successful.
  • swh-search-journal-client restarted
vsellier added a comment.Jan 26 2021, 9:16 AM

The filter on visited origins is working correctly on staging. The has_visit flag looks good.
For example for the https://www.npmjs.com/package/@ehmicky/dev-tasks origin

{
  "_index" : "origin",
  "_type" : "_doc",
  "_id" : "019bd314416108304165e82dd92e00bc9ea85a53",
  "_score" : 60.56421,
  "_source" : {
    "url" : "https://www.npmjs.com/package/@ehmicky/dev-tasks",
    "sha1" : "019bd314416108304165e82dd92e00bc9ea85a53"
  },
  "sort" : [
    60.56421,
    "019bd314416108304165e82dd92e00bc9ea85a53"
  ]
}
swh=> select * from origin join origin_visit_status on id=origin where id=469380;
   id   |                       url                        | origin | visit |             date              | status  | metadata |                  snapshot                  | type 
--------+--------------------------------------------------+--------+-------+-------------------------------+---------+----------+--------------------------------------------+------
 469380 | https://www.npmjs.com/package/@ehmicky/dev-tasks | 469380 |     1 | 2021-01-25 13:30:47.221937+00 | created |          |                                            | npm
 469380 | https://www.npmjs.com/package/@ehmicky/dev-tasks | 469380 |     1 | 2021-01-25 13:41:59.435579+00 | partial |          | \xe3f24413d81fd3e9c309686fcfb6c8f5eb549acf | npm
vsellier added a comment.Jan 26 2021, 9:39 AM

Production

  • puppet disabled
  • Services stopped :
root@search1:~# systemctl stop swh-search-journal-client@objects.service 
root@search1:~# systemctl stop gunicorn-swh-search
  • Index deleted and recreated
% export ES_SERVER=search-esnode1.internal.softwareheritage.org:9200
% curl -s http://$ES_SERVER/_cat/indices\?v 
health status index  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin Mq8dnlpuRXO4yYoC6CTuQw  90   1  151716299     38861934    260.8gb          131gb
% curl -XDELETE http://$ES_SERVER/origin
{"acknowledged":true}%    
% swh search --config-file /etc/softwareheritage/search/server.yml  initialize
INFO:elasticsearch:PUT http://search-esnode1.internal.softwareheritage.org:9200/origin [status:200 request:2.216s]
INFO:elasticsearch:PUT http://search-esnode3.internal.softwareheritage.org:9200/origin/_mapping [status:200 request:0.151s]
Done.
% curl -s http://$ES_SERVER/_cat/indices\?v                                        
health status index  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin yFaqPPCnRFCnc5AA6Ah8lw  90   1          0            0     36.5kb         18.2kb
  • journal client's consumer group delete:
% export SERVER=kafka1.internal.softwareheritage.org:9092  
% ./kafka-consumer-groups.sh --bootstrap-server ${SERVER} --delete --group swh.search.journal_client
Deletion of requested consumer groups ('swh.search.journal_client') was successful.
  • journal client restarted
  • puppet enabled

The journal client is in progress :

% curl -s http://$ES_SERVER/_cat/indices\?v 
health status index  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin yFaqPPCnRFCnc5AA6Ah8lw  90   1      42184            0     45.6mb         42.7mb
ardumont mentioned this in T2993: Deploy visit-stats journal client on production.Jan 26 2021, 10:00 AM
vsellier added a comment.Jan 26 2021, 10:31 AM

Upgrading the index configuration to speedup the indexation :

% cat >/tmp/config.json <<EOF
{
  "index" : {
"translog.sync_interval" : "60s",
"translog.durability": "async",
"refresh_interval": "60s"
  }
}
EOF
% export ES_SERVER=192.168.100.81:9200
% export INDEX=origin            
% curl -s -H "Content-Type: application/json" -XPUT http://${ES_SERVER}/${INDEX}/_settings -d @/tmp/config.json 
{"acknowledged":true}%
vsellier added a comment.Jan 27 2021, 10:00 AM

To decrease the time to recover the lag, several journal client were launched in // with :

/usr/bin/swh search --config-file /etc/softwareheritage/search/journal_client_objects.yml journal-client objects

the memory of search1 server had to be increased to avoid OOM killer: rSPREa7c9c625d98c136ca3d75e649c05bcd4de9aa19b

vsellier moved this task from in-progress to deployed/landed/monitoring on the System administration board.Jan 27 2021, 12:44 PM
vsellier added a comment.Jan 29 2021, 2:44 PM

The journal_client has almost ingested the topics[1] it listens. It took some more time because a backfill of the origin_visit_status was launched for T2993.
It should be done by the end of the day.

[1] https://grafana.softwareheritage.org/goto/EQE43DYMz

vsellier closed this task as Resolved.Feb 1 2021, 10:06 AM

The backfill is done.

vsellier moved this task from deployed/landed/monitoring to done on the System administration board.Feb 18 2021, 9:27 AM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:00 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal