Page MenuHomeSoftware Heritage

Create a new production webapp using the frozen index on the staging ES
Closed, ResolvedPublic

Description

In order to test the swh-search behavior with real data, a webapp will be created and configured to use the production data and the cold swh-search index hosted on the staging ES node.

It's a cold index not updated since some time, but it a first step until a production ES cluster with live data is installed and backfilled with real data

Revisions and Commits

Event Timeline

vsellier triaged this task as Normal priority.Dec 21 2020, 9:59 AM
vsellier created this task.
vsellier changed the task status from Open to Work in Progress.Dec 23 2020, 10:04 AM

In prevision of the deployment, the production index present on the staging's elasticsearch was renamed from origin-production2 to production_origin (a clone operation will be user [1], the original index will be let in place)
[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clone-index.html

vsellier@search-esnode0 ~ % export ES_SERVER=192.168.130.80:9200

vsellier@search-esnode0 ~ % curl -XPOST -s http://${ES_SERVER}/origin-production2/_open\?pretty
{
  "acknowledged" : true,
  "shards_acknowledged" : true
}

vsellier@search-esnode0 ~ % curl  -s http://${ES_SERVER}/_cat/indices\?v
health status index              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin-production2 P5qDO-jmQsmUOY1CI6hUcQ  80   0   91517657            0     65.3gb         65.3gb
green  open   origin             T0AItBcEQqKbS7FlmOs6yA  80   0     408275         1067    274.3mb        274.3mb

# make it read-only
vsellier@search-esnode0 ~ % curl -XPUT -H"Content-Type: application/json" http://${ES_SERVER}/origin-production2/_settings -d'{ "settings": { "index.blocks.write": true } }'
{"acknowledged":true}%  

vsellier@search-esnode0 ~ % curl -XPOST http://${ES_SERVER}/origin-production2/_clone/production_origin\?pretty
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "production_origin"
}

vsellier@search-esnode0 ~ % curl -XPOST -s http://${ES_SERVER}/origin-production2/_close\?pretty
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "indices" : {
    "origin-production2" : {
      "closed" : true
    }
  }
}

vsellier@search-esnode0 ~ % curl -XPUT -H "Content-Type: application/json" http://$ES_SERVER/production_origin/_settings -d '{"index": { "number_of_replicas":0 }}' 
{"acknowledged":true}% 

vsellier@search-esnode0 ~ % curl  -s http://${ES_SERVER}/_cat/indices\?v                                                                                                     
health status index              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  close  origin-production2 P5qDO-jmQsmUOY1CI6hUcQ  80   0                                                  
green  open   origin             T0AItBcEQqKbS7FlmOs6yA  80   0     408338         1084    285.8mb        285.8mb
green open   production_origin  _jEfDEzNQeONKiY3tm3zmw  80   0   91517657            0     65.3gb         65.3gb

The webapp is available at https://webapp1.internal.softwareheritage.org

Reminder: the data on the index are quite old an not updated.
The deployment of a real swh-search will be handled on T2905