Page MenuHomeSoftware Heritage

Deploy swh-search v0.7.0/v0.7.1
Closed, MigratedEdits Locked

Description

It will introduce the aliases usage

First in staging and then in production:

  • stop the swh-search journal clients
  • upgrade the packages
  • restart the gunicorn service and make a request
  • check the alias on elasticsearch
  • restart the journal client
  • upgrade any other nodes (webapp1/moma)

Event Timeline

vsellier changed the task status from Open to Work in Progress.Mar 4 2021, 12:08 PM
vsellier triaged this task as Normal priority.
vsellier created this task.
vsellier renamed this task from Deploy swh-search v0.7.0 to Deploy swh-search v0.7.0/v0.7.1.Mar 4 2021, 4:01 PM

swh-search:v0.7.1 deployed in staging according to the defined plan.
The aliases are well created and used by the services

vsellier@search-esnode0 ~ % curl -XGET -H "Content-Type: application/json" http://192.168.130.80:9200/_cat/indices
green open  origin                      HthJj42xT5uO7w3Aoxzppw 80 0 929692 137147 4gb 4gb
green close origin-backup-20210209-1736 P1CKjXW0QiWM5zlzX46-fg 80 0                      
green close origin-v0.5.0               SGplSaqPR_O9cPYU4ZsmdQ 80 0                      
vsellier@search-esnode0 ~ % curl -XGET -H "Content-Type: application/json" http://192.168.130.80:9200/_cat/aliases
origin-read  origin - - - -
origin-write origin - - - -

Journal clients:

Mar 04 16:22:40 search0 swh[3598137]: INFO:elasticsearch:POST http://search-esnode0.internal.staging.swh.network:9200/origin-write/_bulk [status:200 request:0.013s]
Mar 04 16:22:41 search0 swh[3598137]: INFO:elasticsearch:POST http://search-esnode0.internal.staging.swh.network:9200/origin-write/_bulk [status:200 request:0.012s]

Search:

Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] swh.search.api.server:INFO Initializing indexes with configuration: 
Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] elasticsearch:INFO HEAD http://search-esnode0.internal.staging.swh.network:9200/origin [status:200 request:0.005s]
Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] elasticsearch:INFO HEAD http://search-esnode0.internal.staging.swh.network:9200/origin-read/_alias [status:200 request:0.001s]
Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] elasticsearch:INFO HEAD http://search-esnode0.internal.staging.swh.network:9200/origin-write/_alias [status:200 request:0.001s]
Mar 04 15:40:20 search0 python3[3598040]: 2021-03-04 15:40:20 [3598040] elasticsearch:INFO PUT http://search-esnode0.internal.staging.swh.network:9200/origin/_mapping [status:200 request:0.006s]
Mar 04 16:19:27 search0 python3[3598042]: 2021-03-04 16:19:27 [3598042] elasticsearch:INFO GET http://search-esnode0.internal.staging.swh.network:9200/origin-read/_search?size=100 [status:200 request:0.076s]

Deployment in production:

  • puppet stopped
  • configuration updated to declare the index, it needs to be done to make swh-search initializing the aliaes before the journal clients starts (not guaranteed with a puppet apply)
  • package updated
  • gunicorn-swh-search service restarted:
Mar 05 09:08:46 search1 python3[1881743]: 2021-03-05 09:08:46 [1881743] gunicorn.error:INFO Starting gunicorn 19.9.0
Mar 05 09:08:46 search1 python3[1881743]: 2021-03-05 09:08:46 [1881743] gunicorn.error:INFO Listening at: unix:/run/gunicorn/swh-search/gunicorn.sock (1881743)
Mar 05 09:08:46 search1 python3[1881743]: 2021-03-05 09:08:46 [1881743] gunicorn.error:INFO Using worker: sync
Mar 05 09:08:46 search1 python3[1881748]: 2021-03-05 09:08:46 [1881748] gunicorn.error:INFO Booting worker with pid: 1881748
Mar 05 09:08:46 search1 python3[1881749]: 2021-03-05 09:08:46 [1881749] gunicorn.error:INFO Booting worker with pid: 1881749
Mar 05 09:08:46 search1 python3[1881750]: 2021-03-05 09:08:46 [1881750] gunicorn.error:INFO Booting worker with pid: 1881750
Mar 05 09:08:46 search1 python3[1881751]: 2021-03-05 09:08:46 [1881751] gunicorn.error:INFO Booting worker with pid: 1881751
Mar 05 09:08:53 search1 python3[1881750]: 2021-03-05 09:08:53 [1881750] swh.search.api.server:INFO Initializing indexes with configuration: 
Mar 05 09:08:53 search1 python3[1881750]: 2021-03-05 09:08:53 [1881750] elasticsearch:INFO HEAD http://search-esnode2.internal.softwareheritage.org:9200/origin-production [status:200 request:0.023s]
Mar 05 09:08:54 search1 python3[1881750]: 2021-03-05 09:08:54 [1881750] elasticsearch:INFO PUT http://search-esnode1.internal.softwareheritage.org:9200/origin-production/_alias/origin-read [status:200 request:0.487s]
Mar 05 09:08:54 search1 python3[1881750]: 2021-03-05 09:08:54 [1881750] elasticsearch:INFO PUT http://search-esnode3.internal.softwareheritage.org:9200/origin-production/_alias/origin-write [status:200 request:0.152s]
Mar 05 09:08:54 search1 python3[1881750]: 2021-03-05 09:08:54 [1881750] elasticsearch:INFO PUT http://search-esnode1.internal.softwareheritage.org:9200/origin-production/_mapping [status:200 request:0.009s]
vsellier@search-esnode1 ~ % curl -s http://$ES_SERVER/_cat/indices\?v && echo && curl -s http://$ES_SERVER/_cat/aliases\?v && echo && curl -s http://$ES_SERVER/_cat/health\?v 
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin-production hZfuv0lVRImjOjO_rYgDzg  90   1  153097672    144224208    288.1gb          149gb

alias        index             filter routing.index routing.search is_write_index
origin       origin-production -      -             -              -
origin-read  origin-production -      -             -              -
origin-write origin-production -      -             -              -

epoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1614935530 09:12:10  swh-search green           3         3    180  90    0    0        0             0                  -                100.0%
  • journal client restarted
Mar 05 09:13:47 search1 swh[1881848]: INFO:elasticsearch:POST http://search-esnode2.internal.softwareheritage.org:9200/origin-write/_bulk [status:200 request:0.086s]
Mar 05 09:13:47 search1 swh[1881848]: INFO:elasticsearch:POST http://search-esnode1.internal.softwareheritage.org:9200/origin-write/_bulk [status:200 request:0.058s]
Mar 05 09:13:47 search1 swh[1881848]: INFO:elasticsearch:POST http://search-esnode3.internal.softwareheritage.org:9200/origin-write/_bulk [status:200 request:0.069s]
Mar 05 09:13:47 search1 swh[1881848]: INFO:elasticsearch:POST http://search-esnode2.internal.softwareheritage.org:9200/origin-write/_bulk [status:200 request:0.048s]
  • moma and webapp1 updated

I forgot one step, cleaning the previous alias origin -> origin_production not needed anymore:

vsellier@search-esnode1 ~ % curl -s http://$ES_SERVER/_cat/indices\?v && echo && curl -s http://$ES_SERVER/_cat/aliases\?v && echo && curl -s http://$ES_SERVER/_cat/health\?v  
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin-production hZfuv0lVRImjOjO_rYgDzg  90   1  153130652     26701625    273.4gb        137.3gb

alias        index             filter routing.index routing.search is_write_index
origin-read  origin-production -      -             -              -
origin-write origin-production -      -             -              -

epoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1614937416 09:43:36  swh-search green           3         3    180  90    0    0        0             0                  -                100.0%