Deploy swh-search v0.7.0/v0.7.1
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	vsellier
	Mar 4 2021, 12:08 PM

Description

It will introduce the aliases usage

First in staging and then in production:

stop the swh-search journal clients
upgrade the packages
restart the gunicorn service and make a request
check the alias on elasticsearch
restart the journal client
upgrade any other nodes (webapp1/moma)

Revisions and Commits

rSPSITE puppet-swh-site
	D5198	rSPSITEb4d30523f9bd swh-search: add indexes configuration

Related Objects
Search...

Status	Assigned	Task
Migrated	gitlab-migration	T1523 Search tools on metadata
Migrated	gitlab-migration	T1117 Origin search is slow when you look for very common words
Migrated	gitlab-migration	T1910 Redesign origin search using a dedicated component (swh-search)
Migrated	gitlab-migration	T2052 Publish swh-search on PyPI
Migrated	gitlab-migration	T2167 Deploy swh-search
Migrated	gitlab-migration	T2174 Add debian package for swh-search
Migrated	gitlab-migration	T2182 Switch production swh-web to use swh-search instead of postgresql search.
Migrated	gitlab-migration	T2590 Finish the indexer -> swh-search pipeline
Migrated	gitlab-migration	T3076 [swh-search] Improve the index/mapping migration process
Migrated	gitlab-migration	T3083 Deploy swh-search v0.7.0/v0.7.1

Event Timeline

vsellier changed the task status from Open to Work in Progress.Mar 4 2021, 12:08 PM

vsellier triaged this task as Normal priority.

vsellier created this task.

vsellier renamed this task from Deploy swh-search v0.7.0 to Deploy swh-search v0.7.0/v0.7.1.Mar 4 2021, 4:01 PM

swh-search:v0.7.1 deployed in staging according to the defined plan.
The aliases are well created and used by the services

vsellier@search-esnode0 ~ % curl -XGET -H "Content-Type: application/json" http://192.168.130.80:9200/_cat/indices
green open  origin                      HthJj42xT5uO7w3Aoxzppw 80 0 929692 137147 4gb 4gb
green close origin-backup-20210209-1736 P1CKjXW0QiWM5zlzX46-fg 80 0                      
green close origin-v0.5.0               SGplSaqPR_O9cPYU4ZsmdQ 80 0                      
vsellier@search-esnode0 ~ % curl -XGET -H "Content-Type: application/json" http://192.168.130.80:9200/_cat/aliases
origin-read  origin - - - -
origin-write origin - - - -

Journal clients:

Mar 04 16:22:40 search0 swh[3598137]: INFO:elasticsearch:POST http://search-esnode0.internal.staging.swh.network:9200/origin-write/_bulk [status:200 request:0.013s]
Mar 04 16:22:41 search0 swh[3598137]: INFO:elasticsearch:POST http://search-esnode0.internal.staging.swh.network:9200/origin-write/_bulk [status:200 request:0.012s]

Search:

Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] swh.search.api.server:INFO Initializing indexes with configuration: 
Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] elasticsearch:INFO HEAD http://search-esnode0.internal.staging.swh.network:9200/origin [status:200 request:0.005s]
Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] elasticsearch:INFO HEAD http://search-esnode0.internal.staging.swh.network:9200/origin-read/_alias [status:200 request:0.001s]
Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] elasticsearch:INFO HEAD http://search-esnode0.internal.staging.swh.network:9200/origin-write/_alias [status:200 request:0.001s]
Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] elasticsearch:INFO PUT http://search-esnode0.internal.staging.swh.network:9200/origin/_mapping [status:200 request:0.006s]
Mar 04 16:19:27 search0 python3[3598042]: 2021-03-04 16:19:27 [3598042] elasticsearch:INFO GET http://search-esnode0.internal.staging.swh.network:9200/origin-read/_search?size=100 [status:200 request:0.076s]

vsellier added a revision: D5198: swh-search: add indexes configuration.Mar 5 2021, 9:19 AM

vsellier added a commit: rSPSITEb4d30523f9bd: swh-search: add indexes configuration.Mar 5 2021, 9:36 AM

Deployment in production:

puppet stopped
configuration updated to declare the index, it needs to be done to make swh-search initializing the aliaes before the journal clients starts (not guaranteed with a puppet apply)
package updated
gunicorn-swh-search service restarted:

Mar 05 09:08:46 search1 python3[1881743]: 2021-03-05 09:08:46 [1881743] gunicorn.error:INFO Starting gunicorn 19.9.0
Mar 05 09:08:46 search1 python3[1881743]: 2021-03-05 09:08:46 [1881743] gunicorn.error:INFO Listening at: unix:/run/gunicorn/swh-search/gunicorn.sock (1881743)
Mar 05 09:08:46 search1 python3[1881743]: 2021-03-05 09:08:46 [1881743] gunicorn.error:INFO Using worker: sync
Mar 05 09:08:46 search1 python3[1881748]: 2021-03-05 09:08:46 [1881748] gunicorn.error:INFO Booting worker with pid: 1881748
Mar 05 09:08:46 search1 python3[1881749]: 2021-03-05 09:08:46 [1881749] gunicorn.error:INFO Booting worker with pid: 1881749
Mar 05 09:08:46 search1 python3[1881750]: 2021-03-05 09:08:46 [1881750] gunicorn.error:INFO Booting worker with pid: 1881750
Mar 05 09:08:46 search1 python3[1881751]: 2021-03-05 09:08:46 [1881751] gunicorn.error:INFO Booting worker with pid: 1881751
Mar 05 09:08:53 search1 python3[1881750]: 2021-03-05 09:08:53 [1881750] swh.search.api.server:INFO Initializing indexes with configuration: 
Mar 05 09:08:53 search1 python3[1881750]: 2021-03-05 09:08:53 [1881750] elasticsearch:INFO HEAD http://search-esnode2.internal.softwareheritage.org:9200/origin-production [status:200 request:0.023s]
Mar 05 09:08:54 search1 python3[1881750]: 2021-03-05 09:08:54 [1881750] elasticsearch:INFO PUT http://search-esnode1.internal.softwareheritage.org:9200/origin-production/_alias/origin-read [status:200 request:0.487s]
Mar 05 09:08:54 search1 python3[1881750]: 2021-03-05 09:08:54 [1881750] elasticsearch:INFO PUT http://search-esnode3.internal.softwareheritage.org:9200/origin-production/_alias/origin-write [status:200 request:0.152s]
Mar 05 09:08:54 search1 python3[1881750]: 2021-03-05 09:08:54 [1881750] elasticsearch:INFO PUT http://search-esnode1.internal.softwareheritage.org:9200/origin-production/_mapping [status:200 request:0.009s]

vsellier@search-esnode1 ~ % curl -s http://$ES_SERVER/_cat/indices\?v && echo && curl -s http://$ES_SERVER/_cat/aliases\?v && echo && curl -s http://$ES_SERVER/_cat/health\?v 
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin-production hZfuv0lVRImjOjO_rYgDzg  90   1  153097672    144224208    288.1gb          149gb

alias        index             filter routing.index routing.search is_write_index
origin       origin-production -      -             -              -
origin-read  origin-production -      -             -              -
origin-write origin-production -      -             -              -

epoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1614935530 09:12:10  swh-search green           3         3    180  90    0    0        0             0                  -                100.0%

journal client restarted

Mar 05 09:13:47 search1 swh[1881848]: INFO:elasticsearch:POST http://search-esnode2.internal.softwareheritage.org:9200/origin-write/_bulk [status:200 request:0.086s]
Mar 05 09:13:47 search1 swh[1881848]: INFO:elasticsearch:POST http://search-esnode1.internal.softwareheritage.org:9200/origin-write/_bulk [status:200 request:0.058s]
Mar 05 09:13:47 search1 swh[1881848]: INFO:elasticsearch:POST http://search-esnode3.internal.softwareheritage.org:9200/origin-write/_bulk [status:200 request:0.069s]
Mar 05 09:13:47 search1 swh[1881848]: INFO:elasticsearch:POST http://search-esnode2.internal.softwareheritage.org:9200/origin-write/_bulk [status:200 request:0.048s]

moma and webapp1 updated

vsellier closed this task as Resolved.Mar 5 2021, 10:34 AM

I forgot one step, cleaning the previous alias origin -> origin_production not needed anymore:

vsellier@search-esnode1 ~ % curl -s http://$ES_SERVER/_cat/indices\?v && echo && curl -s http://$ES_SERVER/_cat/aliases\?v && echo && curl -s http://$ES_SERVER/_cat/health\?v  
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin-production hZfuv0lVRImjOjO_rYgDzg  90   1  153130652     26701625    273.4gb        137.3gb

alias        index             filter routing.index routing.search is_write_index
origin-read  origin-production -      -             -              -
origin-write origin-production -      -             -              -

epoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1614937416 09:43:36  swh-search green           3         3    180  90    0    0        0             0                  -                100.0%

ardumont mentioned this in T3433: Deploy swh.search v0.10/v0.11.Jul 19 2021, 2:53 PM

This task has been migrated to GitLab.

Deploy swh-search v0.7.0/v0.7.1Closed, MigratedEdits LockedActions