Page MenuHomeSoftware Heritage

Deploy swh search on storage0 and make webapp0 use it for search
ClosedPublic

Authored by ardumont on Jan 23 2020, 5:10 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Summary
  • Add swh-search profile
  • storage0.euwest.azure: Install swh-search profile
  • webapp0: Switch to use swh-search

Related to T2167
Related to T2183

Test Plan

bin/octocatalog-diff moma
~> expectedly, nothing changes (i slightly touched the webapp configuration to reuse conf so making sure)

bin/octocatalog-diff webapp0.softwareheritage.org
~> expectedly, wants to add the search dict configuration on webapp configuration for search on azure's storage0 node (against empty for current production in moma for now)

bin/octocatalog-diff storage0.euwest.azure.internal.softwareheritage.org
~> expectedly, wants to install the rpc server and serve on 5010 port

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 10264
Build 15229: arc lint + arc unit

Event Timeline

ardumont created this revision.Jan 23 2020, 5:10 PM
ardumont edited the test plan for this revision. (Show Details)Jan 24 2020, 8:45 AM
ardumont added a project: Archive search.
ardumont added inline comments.
data/defaults.yaml
4218

@vlorentz are those the correct elasticsearch nodes (esnode[1-3]) which have the data to lookup?

ardumont added inline comments.Jan 24 2020, 9:21 AM
data/defaults.yaml
4218

I don't find the origin index swh-search use there so it seems not.

ardumont added inline comments.Jan 24 2020, 9:42 AM
data/defaults.yaml
4218

Looking back at the task hierarchy, i saw T2184 so that might explain it.

vlorentz accepted this revision.Jan 24 2020, 11:35 AM
vlorentz added inline comments.
data/defaults.yaml
4218

They are the right nodes, and the index exists.

>>> import elasticsearch
>>> es = elasticsearch.Elasticsearch(['http://esnode3.internal.softwareheritage.org:9200'])
>>> list(es.indices.get('ori*'))
['origin']
This revision is now accepted and ready to land.Jan 24 2020, 11:35 AM
ardumont added inline comments.
data/defaults.yaml
4218

@olasd told me the index exists btw ;)
So i need to improve on how i look up those indexes.
everything is fine.

ardumont added inline comments.Jan 24 2020, 11:39 AM
data/defaults.yaml
4218

lolsad

/me rollseyes at himself

/me looks back at [1] to see it's all there!

[1] https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/backend_es.py