In order to test the swh-search behavior with real data, a webapp will be created and configured to use the production data and the cold swh-search index hosted on the staging ES node.
It's a cold index not updated since some time, but it a first step until a production ES cluster with live data is installed and backfilled with real data
| rDSEA Archive search | |||
| D4785 | rDSEA5560ba0ad621 Allow to prefix the index name(s) | ||
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T1523 Search tools on metadata | ||
| Migrated | gitlab-migration | T1117 Origin search is *slow* when you look for very common words | ||
| Migrated | gitlab-migration | T1910 Redesign origin search using a dedicated component (swh-search) | ||
| Migrated | gitlab-migration | T2052 Publish swh-search on PyPI | ||
| Migrated | gitlab-migration | T2167 Deploy swh-search | ||
| Migrated | gitlab-migration | T2174 Add debian package for swh-search | ||
| Migrated | gitlab-migration | T2182 Switch production swh-web to use swh-search instead of postgresql search. | ||
| Migrated | gitlab-migration | T2590 Finish the indexer -> swh-search pipeline | ||
| Migrated | gitlab-migration | T2904 Create a new production webapp using the frozen index on the staging ES | ||
| Migrated | gitlab-migration | T2905 Deploy swh-search for production | ||
| Migrated | gitlab-migration | T2936 Update the swh-search journal client to only set "has_visit" on "full" status of the visit | ||
| Migrated | gitlab-migration | T2944 Deploy swh-search v0.4.1 |
In prevision of the deployment, the production index present on the staging's elasticsearch was renamed from origin-production2 to production_origin (a clone operation will be user [1], the original index will be let in place)
[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clone-index.html
vsellier@search-esnode0 ~ % export ES_SERVER=192.168.130.80:9200
vsellier@search-esnode0 ~ % curl -XPOST -s http://${ES_SERVER}/origin-production2/_open\?pretty
{
"acknowledged" : true,
"shards_acknowledged" : true
}
vsellier@search-esnode0 ~ % curl -s http://${ES_SERVER}/_cat/indices\?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open origin-production2 P5qDO-jmQsmUOY1CI6hUcQ 80 0 91517657 0 65.3gb 65.3gb
green open origin T0AItBcEQqKbS7FlmOs6yA 80 0 408275 1067 274.3mb 274.3mb
# make it read-only
vsellier@search-esnode0 ~ % curl -XPUT -H"Content-Type: application/json" http://${ES_SERVER}/origin-production2/_settings -d'{ "settings": { "index.blocks.write": true } }'
{"acknowledged":true}%
vsellier@search-esnode0 ~ % curl -XPOST http://${ES_SERVER}/origin-production2/_clone/production_origin\?pretty
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "production_origin"
}
vsellier@search-esnode0 ~ % curl -XPOST -s http://${ES_SERVER}/origin-production2/_close\?pretty
{
"acknowledged" : true,
"shards_acknowledged" : true,
"indices" : {
"origin-production2" : {
"closed" : true
}
}
}
vsellier@search-esnode0 ~ % curl -XPUT -H "Content-Type: application/json" http://$ES_SERVER/production_origin/_settings -d '{"index": { "number_of_replicas":0 }}'
{"acknowledged":true}%
vsellier@search-esnode0 ~ % curl -s http://${ES_SERVER}/_cat/indices\?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green close origin-production2 P5qDO-jmQsmUOY1CI6hUcQ 80 0
green open origin T0AItBcEQqKbS7FlmOs6yA 80 0 408338 1084 285.8mb 285.8mb
green open production_origin _jEfDEzNQeONKiY3tm3zmw 80 0 91517657 0 65.3gb 65.3gbThe webapp is available at https://webapp1.internal.softwareheritage.org
Reminder: the data on the index are quite old an not updated.
The deployment of a real swh-search will be handled on T2905