Differential D8959

github: Export statsd metrics about API requests and token usage
ClosedPublic
Actions

Authored by vlorentz on Dec 15 2022, 12:21 PM.

Details

Reviewers

olasd

Group Reviewers

Reviewers

Maniphest Tasks

T4728: Add monitoring of API token usage

Commits

rDCORE311113c6b7f0: github: Export statsd metrics about API requests and token usage

Summary

We will need to monitored token usage when scheduling recurring fetches
of extrinsic metadata.

Metrics on API requests and sleep time themselves are just an extra
that may be handy in the future.

This is the first half of T4728.

Diff Detail

Repository

rDCORE Foundations and core functionalities

Branch

statsd

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 33236
Build 52111: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 52110: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	4 ms	Jenkins > .tox..tox.py3-core-db-server-github-slow-cover.lib.python3.7.site-packages.swh.core.tests.test_statsd::test_tags_from_environment_warning monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fab5afde860> def test_tags_from_environment_warning(monkeypatch):
	1 ms	Jenkins > .tox..tox.py3-core-db-server-github-slow-cover.lib.python3.7.site-packages.swh.core.api.tests.test_async::test_async_decode_data_empty_cases[None]
	1 ms	Jenkins > .tox..tox.py3-core-db-server-github-slow-cover.lib.python3.7.site-packages.swh.core.api.tests.test_async::test_async_decode_data_empty_cases[]
	1 ms	Jenkins > .tox..tox.py3-core-db-server-github-slow-cover.lib.python3.7.site-packages.swh.core.api.tests.test_async::test_async_decode_data_empty_cases[data2]
	1 ms	Jenkins > .tox..tox.py3-core-db-server-github-slow-cover.lib.python3.7.site-packages.swh.core.api.tests.test_async::test_async_decode_data_empty_cases[data3]
		View Full Test Results (1 Failed · 317 Passed · 1 Skipped)

Event Timeline

vlorentz created this revision.Dec 15 2022, 12:21 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptDec 15 2022, 12:21 PM

vlorentz added a task: T4728: Add monitoring of API token usage.Dec 15 2022, 12:22 PM

Build has FAILED

Patch application report for D8959 (id=32281)

Rebasing onto 306a74d01a...

Current branch diff-target is up to date.

Changes applied before test

commit e932c3978a51c9826c65afc34d7a5c096df63984
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 15 12:21:23 2022 +0100

    github: Export statsd metrics about API requests and token usage
    
    We will need to monitored token usage when scheduling recurring fetches
    of extrinsic metadata.
    
    Metrics on API requests and sleep time themselves are just an extra
    that may be handy in the future.

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/492/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/492/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 15 2022, 12:23 PM

Harbormaster failed remote builds in B33236: Diff 32281!

f-string

Build is green

Patch application report for D8959 (id=32282)

Rebasing onto 306a74d01a...

Current branch diff-target is up to date.

Changes applied before test

commit bf5d7728766aa00ad444cc8481ae333adb2c664a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 15 12:21:23 2022 +0100

    github: Export statsd metrics about API requests and token usage
    
    We will need to monitored token usage when scheduling recurring fetches
    of extrinsic metadata.
    
    Metrics on API requests and sleep time themselves are just an extra
    that may be handy in the future.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/493/ for more details.

Harbormaster completed remote builds in B33237: Diff 32282.Dec 15 2022, 12:30 PM

vlorentz requested review of this revision.Dec 15 2022, 12:30 PM

vlorentz added inline comments.Dec 15 2022, 12:59 PM

swh/core/github/pytest_plugin.py
154–158	this was the branch for cases 1 and 3
160	and that was case 2

Thanks.

This looks sensible. When this is deployed, we'll need to monitor how the label set grows over time, of course!

This revision is now accepted and ready to land.Dec 20 2022, 6:01 PM

metametrics :)

rebase

This revision was landed with ongoing or failed builds.Dec 21 2022, 3:04 PM

Closed by commit rDCORE311113c6b7f0: github: Export statsd metrics about API requests and token usage (authored by vlorentz). · Explain Why

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDCORE311113c6b7f0: github: Export statsd metrics about API requests and token usage.

Build is green

Patch application report for D8959 (id=32345)

Rebasing onto bff1e71c7c...

Current branch diff-target is up to date.

Changes applied before test

commit 311113c6b7f0122a9e13022fb6d7eb7ec9b23d25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 15 12:21:23 2022 +0100

    github: Export statsd metrics about API requests and token usage
    
    We will need to monitored token usage when scheduling recurring fetches
    of extrinsic metadata.
    
    Metrics on API requests and sleep time themselves are just an extra
    that may be handy in the future.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/494/ for more details.

Harbormaster completed remote builds in B33332: Diff 32345.Dec 21 2022, 3:06 PM

vlorentz added a child revision: D8975: Add missing _seconds_total unit to swh_outbound_api_sleep metric.Dec 21 2022, 3:59 PM