Page MenuHomeSoftware Heritage

metrics: Add gauge counting save origin requests created from webhooks
ClosedPublic

Authored by anlambert on Nov 28 2022, 3:17 PM.

Details

Summary

It enables to have some statistics in grafana about the number of save code
now requests created from forge webhook receivers.

Related to T4548

Depends on D8889

Diff Detail

Repository
rDWAPPS Web applications
Branch
save-code-now-webhook-metric
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 33028
Build 51763: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 51762: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8890 (id=32042)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..48f7dfef
Fast-forward
 swh/web/metrics/prometheus.py                       | 17 +++++++++++++++++
 swh/web/metrics/tests/test_metrics.py               | 18 ++++++++++++++++--
 .../0013_saveoriginrequest_from_webhook.py          | 21 +++++++++++++++++++++
 swh/web/save_code_now/models.py                     |  2 ++
 swh/web/save_code_now/origin_save.py                |  5 +++++
 swh/web/save_code_now/tests/test_migrations.py      | 19 ++++++++++++++++++-
 swh/web/save_code_now/tests/test_origin_save.py     |  1 +
 swh/web/save_origin_webhooks/generic_receiver.py    |  2 +-
 swh/web/save_origin_webhooks/tests/utils.py         |  6 ++++++
 swh/web/utils/typing.py                             |  2 ++
 10 files changed, 89 insertions(+), 4 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_from_webhook.py
Changes applied before test
commit 48f7dfefdba629854110dd410a903f11e9358753
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit ec3dc8b253eddb0cb44a5825d13244f147d1659d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add from_webhook field to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/36/ for more details.

This revision is now accepted and ready to land.Nov 29 2022, 11:56 AM

Update: Add label for webhook origin in prometheus gauge

Build is green

Patch application report for D8890 (id=32070)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..76347f26
Fast-forward
 swh/web/metrics/prometheus.py                      | 24 +++++++++++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 28 ++++++++++++++++++++--
 .../0013_saveoriginrequest_webhook_info.py         | 26 ++++++++++++++++++++
 swh/web/save_code_now/models.py                    |  4 ++++
 swh/web/save_code_now/origin_save.py               |  9 +++++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 +++++++++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 ++
 swh/web/save_origin_webhooks/generic_receiver.py   |  8 ++++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 +++++
 swh/web/utils/typing.py                            |  4 ++++
 11 files changed, 129 insertions(+), 5 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit 76347f26d11228f90999af790aa9f84cdc11575c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit dd363e38058609eefb8cfe29a9c186a834c1477e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit 08a2f3dc54122a7683d6d35c8c9f5db076f0b121
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/38/ for more details.

olasd added inline comments.
swh/web/metrics/prometheus.py
102–106

Do we have access to a shorter list of valid (visit_type, supported_forge_type) tuples? Most forges will only accept one or a few visit types and it'd be nice to avoid generating a bunch of metrics that are guaranteed to be zero, if possible.

swh/web/metrics/prometheus.py
102–106

Apart webhooks received from sourceforge, all visit types will be git. I think we can drop the visit_type label for that gauge then as that info is not really of interest here.

Drop visit_type label in prometheus gauge.

Build is green

Patch application report for D8890 (id=32073)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..1c67ce1a
Fast-forward
 swh/web/metrics/prometheus.py                      | 23 ++++++++++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 27 ++++++++++++++++++++--
 .../0013_saveoriginrequest_webhook_info.py         | 26 +++++++++++++++++++++
 swh/web/save_code_now/models.py                    |  4 ++++
 swh/web/save_code_now/origin_save.py               | 10 ++++++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 ++++++++++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 ++
 swh/web/save_origin_webhooks/generic_receiver.py   |  8 ++++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 +++++
 swh/web/utils/typing.py                            |  4 ++++
 11 files changed, 128 insertions(+), 5 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit 1c67ce1aa160acba65ebfc30f0ea33fdf89af944
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit 71ff8eae8660453b61f231cb590765d768fd1019
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit 4bf0cd9bd08c75c083625e76c503708cb3c759f9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/40/ for more details.

Build has FAILED

Patch application report for D8890 (id=32078)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..581e782b
Fast-forward
 swh/web/api/apidoc.py                              |  1 +
 swh/web/metrics/prometheus.py                      | 23 +++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 27 ++++++++++++-
 swh/web/save_code_now/api_views.py                 | 45 +++++++++++++++++++---
 .../0013_saveoriginrequest_webhook_info.py         | 26 +++++++++++++
 swh/web/save_code_now/models.py                    |  4 ++
 swh/web/save_code_now/origin_save.py               | 10 +++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 +++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 +
 .../save_code_now/tests/test_origin_save_api.py    |  4 ++
 swh/web/save_origin_webhooks/generic_receiver.py   |  9 ++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 +++
 swh/web/utils/typing.py                            |  4 ++
 14 files changed, 173 insertions(+), 11 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit 581e782b29fd0df5afd9316b96e691a5232d6067
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit 50ad6e9f97344aee8826bc96cd2ff6507c76e41e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit fb878e1a2cb5c041b1d3250701c273915fa3a2b6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/42/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/42/console

Build is green

Patch application report for D8890 (id=32078)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..581e782b
Fast-forward
 swh/web/api/apidoc.py                              |  1 +
 swh/web/metrics/prometheus.py                      | 23 +++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 27 ++++++++++++-
 swh/web/save_code_now/api_views.py                 | 45 +++++++++++++++++++---
 .../0013_saveoriginrequest_webhook_info.py         | 26 +++++++++++++
 swh/web/save_code_now/models.py                    |  4 ++
 swh/web/save_code_now/origin_save.py               | 10 +++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 +++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 +
 .../save_code_now/tests/test_origin_save_api.py    |  4 ++
 swh/web/save_origin_webhooks/generic_receiver.py   |  9 ++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 +++
 swh/web/utils/typing.py                            |  4 ++
 14 files changed, 173 insertions(+), 11 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit 581e782b29fd0df5afd9316b96e691a5232d6067
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit 50ad6e9f97344aee8826bc96cd2ff6507c76e41e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit fb878e1a2cb5c041b1d3250701c273915fa3a2b6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/43/ for more details.

Build is green

Patch application report for D8890 (id=32102)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..ce8da30b
Fast-forward
 docs/uri-scheme-api-origin.rst                     | 14 ++++++++
 swh/web/api/apidoc.py                              |  1 +
 swh/web/metrics/prometheus.py                      | 23 ++++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 27 ++++++++++++--
 swh/web/save_code_now/api_views.py                 | 42 ++++++++++++++++++----
 .../0013_saveoriginrequest_webhook_info.py         | 26 ++++++++++++++
 swh/web/save_code_now/models.py                    |  4 +++
 swh/web/save_code_now/origin_save.py               | 10 ++++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 ++++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 ++
 .../save_code_now/tests/test_origin_save_api.py    |  4 +++
 swh/web/save_origin_webhooks/generic_receiver.py   |  9 ++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 ++++
 swh/web/utils/typing.py                            |  4 +++
 15 files changed, 184 insertions(+), 11 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit ce8da30b98426c47aba1f9c0feb0ff30d71ceb3a
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit b4fc49b3fe92651f950a83705d5d952ec488ad8c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit 12a291957fa06e3b6fde1ebd3a65cad8e3b03016
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/48/ for more details.