Page MenuHomeSoftware Heritage

api/server: insert statsd probes for every API endpoint
ClosedPublic

Authored by douardda on Mar 19 2019, 12:32 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Summary

Depends on D1276

Diff Detail

Branch
master
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 4747
Build 6325: arc lint + arc unit

Event Timeline

vlorentz added inline comments.
swh/objstorage/api/server.py
28

await is not a function, is it?

Where are stored those statsd information again?

olasd requested changes to this revision.Mar 19 2019, 2:56 PM
olasd added a subscriber: olasd.

Where are stored those statsd information again?

In production, there's a statsd instance deployed on each machine, that gets scraped every minute by prometheus.

The data can then be used in a grafana dashboard.

swh/objstorage/api/server.py
25–28

Rather than having such a high number of different metrics, please use a single metric name and add the function name as a label: swh_objstorage_request_count and swh_objstorage_request_duration_seconds, both with an endpoint label.

This way we don't need to update a potential dashboard when we add a new function.

I think we'll also want to set up a few static labels to be able to distinguish several instances of the objstorage on the host.

54

Accumulating counters for volumes usually end with <unit>_total (https://prometheus.io/docs/practices/naming/)

swh_objstorage_add_bytes_total or swh_objstorage_in_bytes_total would probably make sense.

78

Same remark: swh_objstorage_get_bytes_total or swh_objstorage_out_bytes_total

This revision now requires changes to proceed.Mar 19 2019, 2:56 PM
douardda added inline comments.
swh/objstorage/api/server.py
25–28

I think we'll also want to set up a few static labels to be able to distinguish several instances of the objstorage on the host.

these ones can be configured vis static tags from the STATSD_TAGS env var rather than adding code, can't they?

54

agreed

Where are stored those statsd information again?

nowhere, unless one have a running statsd server. In our case, we do have a promotheus-statsd server running. So the answer to this question is: in promotheus (in prod).

Updated according to comments

olasd added inline comments.
swh/objstorage/api/server.py
25–28

On the server-side, yeah, that'll work fine.

This revision is now accepted and ready to land.Mar 20 2019, 4:27 PM

closed by 31e221d5aecc0e259f8831059dfc17883ad44441