Page MenuHomeSoftware Heritage

api/server: insert statsd probes for every API endpoint
ClosedPublic

Authored by douardda on Mar 19 2019, 12:32 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Summary

Depends on D1276

Diff Detail

Repository
rDOBJS Object storage
Branch
master
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 4726
Build 6299: tox-on-jenkinsJenkins
Build 6298: arc lint + arc unit

Event Timeline

vlorentz added inline comments.
swh/objstorage/api/server.py
29

await is not a function, is it?

Where are stored those statsd information again?

olasd requested changes to this revision.Mar 19 2019, 2:56 PM
olasd added a subscriber: olasd.

Where are stored those statsd information again?

In production, there's a statsd instance deployed on each machine, that gets scraped every minute by prometheus.

The data can then be used in a grafana dashboard.

swh/objstorage/api/server.py
26–29

Rather than having such a high number of different metrics, please use a single metric name and add the function name as a label: swh_objstorage_request_count and swh_objstorage_request_duration_seconds, both with an endpoint label.

This way we don't need to update a potential dashboard when we add a new function.

I think we'll also want to set up a few static labels to be able to distinguish several instances of the objstorage on the host.

53

Accumulating counters for volumes usually end with <unit>_total (https://prometheus.io/docs/practices/naming/)

swh_objstorage_add_bytes_total or swh_objstorage_in_bytes_total would probably make sense.

76

Same remark: swh_objstorage_get_bytes_total or swh_objstorage_out_bytes_total

This revision now requires changes to proceed.Mar 19 2019, 2:56 PM
douardda added inline comments.
swh/objstorage/api/server.py
26–29

I think we'll also want to set up a few static labels to be able to distinguish several instances of the objstorage on the host.

these ones can be configured vis static tags from the STATSD_TAGS env var rather than adding code, can't they?

53

agreed

Where are stored those statsd information again?

nowhere, unless one have a running statsd server. In our case, we do have a promotheus-statsd server running. So the answer to this question is: in promotheus (in prod).

Updated according to comments

olasd added inline comments.
swh/objstorage/api/server.py
26–29

On the server-side, yeah, that'll work fine.

This revision is now accepted and ready to land.Mar 20 2019, 4:27 PM

closed by 31e221d5aecc0e259f8831059dfc17883ad44441