Deploy swh search on storage0 and make webapp0 use it for search
ClosedPublic
Actions

Authored by ardumont on Jan 23 2020, 5:10 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Summary

Add swh-search profile
storage0.euwest.azure: Install swh-search profile
webapp0: Switch to use swh-search

Related to T2167
Related to T2183

Test Plan

bin/octocatalog-diff moma
~> expectedly, nothing changes (i slightly touched the webapp configuration to reuse conf so making sure)

bin/octocatalog-diff webapp0.softwareheritage.org
~> expectedly, wants to add the search dict configuration on webapp configuration for search on azure's storage0 node (against empty for current production in moma for now)

bin/octocatalog-diff storage0.euwest.azure.internal.softwareheritage.org
~> expectedly, wants to install the rpc server and serve on 5010 port

Diff Detail

Repository

rSPSITE puppet-swh-site

Branch

staging

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 10264
Build 15229: arc lint + arc unit

Event Timeline

ardumont created this revision.Jan 23 2020, 5:10 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 23 2020, 5:10 PM

Harbormaster completed remote builds in B10264: Diff 9206.Jan 23 2020, 5:10 PM

ardumont edited the test plan for this revision. (Show Details)Jan 24 2020, 8:45 AM

ardumont added a project: Archive search.

ardumont added a subscriber: vlorentz.Jan 24 2020, 9:14 AM

ardumont added inline comments.

data/defaults.yaml
4218	@vlorentz are those the correct elasticsearch nodes (esnode[1-3]) which have the data to lookup?

ardumont added inline comments.Jan 24 2020, 9:21 AM

data/defaults.yaml
4218	I don't find the `origin` index swh-search use there so it seems not.

ardumont added inline comments.Jan 24 2020, 9:42 AM

data/defaults.yaml
4218	Looking back at the task hierarchy, i saw T2184 so that might explain it.

vlorentz accepted this revision.Jan 24 2020, 11:35 AM

vlorentz added inline comments.

data/defaults.yaml
4218	They are the right nodes, and the index exists. >>> import elasticsearch >>> es = elasticsearch.Elasticsearch(['http://esnode3.internal.softwareheritage.org:9200']) >>> list(es.indices.get('ori*')) ['origin']

This revision is now accepted and ready to land.Jan 24 2020, 11:35 AM

ardumont added a subscriber: olasd.Jan 24 2020, 11:37 AM

ardumont added inline comments.

data/defaults.yaml
4218	@olasd told me the index exists btw ;) So i need to improve on how i look up those indexes. everything is fine.

ardumont added inline comments.Jan 24 2020, 11:39 AM

data/defaults.yaml
4218	lolsad /me rollseyes at himself /me looks back at [1] to see it's all there! [1] https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/backend_es.py