Page MenuHomeSoftware Heritage
Differential D8890

metrics: Add gauge counting save origin requests created from webhooks
ClosedPublic
Actions

Authored by anlambert on Nov 28 2022, 3:17 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4548: Add a public API endpoint and documentation to trigger Save Code Now from webhook
Commits
rDWAPPSce8da30b9842: metrics/tests/test_app: Fix test name
rDWAPPSb4fc49b3fe92: metrics: Add gauge counting save origin requests created from webhooks
Summary

It enables to have some statistics in grafana about the number of save code
now requests created from forge webhook receivers.

Related to T4548

Depends on D8889

Diff Detail

Repository
rDWAPPS Web applications
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Nov 28 2022, 3:17 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 28 2022, 3:17 PM
swh-public-ci added a comment.Nov 28 2022, 3:28 PM

Build is green

Patch application report for D8890 (id=32042)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..48f7dfef
Fast-forward
 swh/web/metrics/prometheus.py                       | 17 +++++++++++++++++
 swh/web/metrics/tests/test_metrics.py               | 18 ++++++++++++++++--
 .../0013_saveoriginrequest_from_webhook.py          | 21 +++++++++++++++++++++
 swh/web/save_code_now/models.py                     |  2 ++
 swh/web/save_code_now/origin_save.py                |  5 +++++
 swh/web/save_code_now/tests/test_migrations.py      | 19 ++++++++++++++++++-
 swh/web/save_code_now/tests/test_origin_save.py     |  1 +
 swh/web/save_origin_webhooks/generic_receiver.py    |  2 +-
 swh/web/save_origin_webhooks/tests/utils.py         |  6 ++++++
 swh/web/utils/typing.py                             |  2 ++
 10 files changed, 89 insertions(+), 4 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_from_webhook.py
Changes applied before test
commit 48f7dfefdba629854110dd410a903f11e9358753
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit ec3dc8b253eddb0cb44a5825d13244f147d1659d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add from_webhook field to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/36/ for more details.

Harbormaster completed remote builds in B32991: Diff 32042.Nov 28 2022, 3:28 PM
anlambert requested review of this revision.Nov 28 2022, 3:28 PM
ardumont accepted this revision.Nov 29 2022, 11:56 AM
This revision is now accepted and ready to land.Nov 29 2022, 11:56 AM
ardumont mentioned this in D8889: save_code_now: Add from_webhook field to SaveOriginRequest model.Nov 29 2022, 12:14 PM
anlambert updated this revision to Diff 32070.Nov 29 2022, 3:04 PM

Update: Add label for webhook origin in prometheus gauge

swh-public-ci added a comment.Nov 29 2022, 3:16 PM

Build is green

Patch application report for D8890 (id=32070)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..76347f26
Fast-forward
 swh/web/metrics/prometheus.py                      | 24 +++++++++++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 28 ++++++++++++++++++++--
 .../0013_saveoriginrequest_webhook_info.py         | 26 ++++++++++++++++++++
 swh/web/save_code_now/models.py                    |  4 ++++
 swh/web/save_code_now/origin_save.py               |  9 +++++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 +++++++++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 ++
 swh/web/save_origin_webhooks/generic_receiver.py   |  8 ++++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 +++++
 swh/web/utils/typing.py                            |  4 ++++
 11 files changed, 129 insertions(+), 5 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit 76347f26d11228f90999af790aa9f84cdc11575c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit dd363e38058609eefb8cfe29a9c186a834c1477e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit 08a2f3dc54122a7683d6d35c8c9f5db076f0b121
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/38/ for more details.

Harbormaster completed remote builds in B33017: Diff 32070.Nov 29 2022, 3:16 PM
olasd added a subscriber: olasd.Nov 29 2022, 3:17 PM
olasd added inline comments.
swh/web/metrics/prometheus.py
102–106

Do we have access to a shorter list of valid (visit_type, supported_forge_type) tuples? Most forges will only accept one or a few visit types and it'd be nice to avoid generating a bunch of metrics that are guaranteed to be zero, if possible.

anlambert added inline comments.Nov 29 2022, 3:24 PM
swh/web/metrics/prometheus.py
102–106

Apart webhooks received from sourceforge, all visit types will be git. I think we can drop the visit_type label for that gauge then as that info is not really of interest here.

anlambert updated this revision to Diff 32073.Nov 29 2022, 3:31 PM

Drop visit_type label in prometheus gauge.

swh-public-ci added a comment.Nov 29 2022, 3:55 PM

Build is green

Patch application report for D8890 (id=32073)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..1c67ce1a
Fast-forward
 swh/web/metrics/prometheus.py                      | 23 ++++++++++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 27 ++++++++++++++++++++--
 .../0013_saveoriginrequest_webhook_info.py         | 26 +++++++++++++++++++++
 swh/web/save_code_now/models.py                    |  4 ++++
 swh/web/save_code_now/origin_save.py               | 10 ++++++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 ++++++++++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 ++
 swh/web/save_origin_webhooks/generic_receiver.py   |  8 ++++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 +++++
 swh/web/utils/typing.py                            |  4 ++++
 11 files changed, 128 insertions(+), 5 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit 1c67ce1aa160acba65ebfc30f0ea33fdf89af944
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit 71ff8eae8660453b61f231cb590765d768fd1019
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit 4bf0cd9bd08c75c083625e76c503708cb3c759f9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/40/ for more details.

Harbormaster completed remote builds in B33022: Diff 32073.Nov 29 2022, 3:55 PM
anlambert updated this revision to Diff 32078.Nov 29 2022, 4:31 PM

Rebase

swh-public-ci added a comment.Nov 29 2022, 4:45 PM

Build has FAILED

Patch application report for D8890 (id=32078)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..581e782b
Fast-forward
 swh/web/api/apidoc.py                              |  1 +
 swh/web/metrics/prometheus.py                      | 23 +++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 27 ++++++++++++-
 swh/web/save_code_now/api_views.py                 | 45 +++++++++++++++++++---
 .../0013_saveoriginrequest_webhook_info.py         | 26 +++++++++++++
 swh/web/save_code_now/models.py                    |  4 ++
 swh/web/save_code_now/origin_save.py               | 10 +++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 +++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 +
 .../save_code_now/tests/test_origin_save_api.py    |  4 ++
 swh/web/save_origin_webhooks/generic_receiver.py   |  9 ++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 +++
 swh/web/utils/typing.py                            |  4 ++
 14 files changed, 173 insertions(+), 11 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit 581e782b29fd0df5afd9316b96e691a5232d6067
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit 50ad6e9f97344aee8826bc96cd2ff6507c76e41e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit fb878e1a2cb5c041b1d3250701c273915fa3a2b6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/42/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/42/console

Harbormaster failed remote builds in B33028: Diff 32078!Nov 29 2022, 4:45 PM
anlambert added a child revision: D8902: save_code_now: Indicate if request was created from webhook in Web UI.Nov 29 2022, 5:00 PM
swh-public-ci added a comment.Nov 29 2022, 5:08 PM

Build is green

Patch application report for D8890 (id=32078)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..581e782b
Fast-forward
 swh/web/api/apidoc.py                              |  1 +
 swh/web/metrics/prometheus.py                      | 23 +++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 27 ++++++++++++-
 swh/web/save_code_now/api_views.py                 | 45 +++++++++++++++++++---
 .../0013_saveoriginrequest_webhook_info.py         | 26 +++++++++++++
 swh/web/save_code_now/models.py                    |  4 ++
 swh/web/save_code_now/origin_save.py               | 10 +++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 +++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 +
 .../save_code_now/tests/test_origin_save_api.py    |  4 ++
 swh/web/save_origin_webhooks/generic_receiver.py   |  9 ++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 +++
 swh/web/utils/typing.py                            |  4 ++
 14 files changed, 173 insertions(+), 11 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit 581e782b29fd0df5afd9316b96e691a5232d6067
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit 50ad6e9f97344aee8826bc96cd2ff6507c76e41e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit fb878e1a2cb5c041b1d3250701c273915fa3a2b6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/43/ for more details.

Harbormaster completed remote builds in B33028: Diff 32078.Nov 29 2022, 5:08 PM
anlambert updated this revision to Diff 32102.Nov 30 2022, 3:40 PM

Rebase

swh-public-ci added a comment.Nov 30 2022, 3:55 PM

Build is green

Patch application report for D8890 (id=32102)

Could not rebase; Attempt merge onto d9ffdd1759...

Updating d9ffdd17..ce8da30b
Fast-forward
 docs/uri-scheme-api-origin.rst                     | 14 ++++++++
 swh/web/api/apidoc.py                              |  1 +
 swh/web/metrics/prometheus.py                      | 23 ++++++++++++
 swh/web/metrics/tests/test_app.py                  |  2 +-
 swh/web/metrics/tests/test_metrics.py              | 27 ++++++++++++--
 swh/web/save_code_now/api_views.py                 | 42 ++++++++++++++++++----
 .../0013_saveoriginrequest_webhook_info.py         | 26 ++++++++++++++
 swh/web/save_code_now/models.py                    |  4 +++
 swh/web/save_code_now/origin_save.py               | 10 ++++++
 swh/web/save_code_now/tests/test_migrations.py     | 21 ++++++++++-
 swh/web/save_code_now/tests/test_origin_save.py    |  2 ++
 .../save_code_now/tests/test_origin_save_api.py    |  4 +++
 swh/web/save_origin_webhooks/generic_receiver.py   |  9 ++++-
 swh/web/save_origin_webhooks/tests/utils.py        |  6 ++++
 swh/web/utils/typing.py                            |  4 +++
 15 files changed, 184 insertions(+), 11 deletions(-)
 create mode 100644 swh/web/save_code_now/migrations/0013_saveoriginrequest_webhook_info.py
Changes applied before test
commit ce8da30b98426c47aba1f9c0feb0ff30d71ceb3a
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Nov 29 15:00:09 2022 +0100

    metrics/tests/test_app: Fix test name

commit b4fc49b3fe92651f950a83705d5d952ec488ad8c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:32:22 2022 +0100

    metrics: Add gauge counting save origin requests created from webhooks
    
    It enables to have some statistics in grafana about the number of save code
    now requests created from forge webhook receivers.
    
    Related to T4548

commit 12a291957fa06e3b6fde1ebd3a65cad8e3b03016
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 28 14:07:22 2022 +0100

    save_code_now: Add webhook info fields to SaveOriginRequest model
    
    It enables to distinguish which requests were created from a forge
    webhook receiver and which forge type sent the webhook.
    
    Related to T4548

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/48/ for more details.

Harbormaster completed remote builds in B33054: Diff 32102.Nov 30 2022, 3:55 PM
Closed by commit rDWAPPSb4fc49b3fe92: metrics: Add gauge counting save origin requests created from webhooks (authored by anlambert). · Explain WhyNov 30 2022, 4:22 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDWAPPSb4fc49b3fe92: metrics: Add gauge counting save origin requests created from webhooks.
anlambert added a commit: rDWAPPSce8da30b9842: metrics/tests/test_app: Fix test name.

Revision Contents
Changeset List

PathSize
swh/
web/
metrics/
23 lines
tests/
27 lines

Diff 32105

View Options

swh/web/metrics/prometheus.py

Loading...
View Options

swh/web/metrics/tests/test_metrics.py

Loading...
About · Developer information · Legal