Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T3165: Generate historical data from the new counters series

Commits

rDCNT1b262c37b768: Manage and expose the historical data

Summary

Convert the former script managed by puppet to a tested implementation of this module
Add the api endpoint needed by the webapp to retrieve the file with static historical data and the live data from prometheus aggregated
Add the api endpoint to build these files, it will be called regularly by a crontab

The build will fail until D5428 will be released

Related to T3165

Diff Detail

Repository

rDCNT Archive counters

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 20469
Build 31769: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 31768: arc lint + arc unit

Event Timeline

vsellier created this revision.Apr 6 2021, 4:25 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptApr 6 2021, 4:25 PM

Build has FAILED

Patch application report for D5429 (id=19402)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.

Changes applied before test

commit 926f3d813d124cf21ea860d112f4657773662996
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

Link to build: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/25/
See console output for more information: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/25/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 6 2021, 4:26 PM

Harbormaster failed remote builds in B20458: Diff 19402!

ardumont added a subscriber: ardumont.Apr 6 2021, 5:08 PM

ardumont added inline comments.

swh/counters/__init__.py
51–57
swh/counters/api/history.py
43 ↗	(On Diff #19402)
47 ↗	(On Diff #19402)
58 ↗	(On Diff #19402)	i gather this many `{{{` is for escape purposes, right?
124 ↗	(On Diff #19402)	probably a duplicate from the main swh.counters.history that i just started reading.

lgtm

couple of remarks.

I'll do another pass once the build is fine (once you release the D5428 diff ;)

swh/counters/api/history.py
124 ↗	(On Diff #19402)	I mean the all file api/history.py.
swh/counters/interface.py
39

ardumont mentioned this in D5428: Allow to use several backends with a RPCServerApp.Apr 6 2021, 5:34 PM

Remove an old unneeded file

Build has FAILED

Patch application report for D5429 (id=19410)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.

Changes applied before test

commit 8bd4c28f1c51ecb4192db8c21a6de121635f6934
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

Link to build: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/26/
See console output for more information: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/26/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 6 2021, 6:52 PM

Harbormaster failed remote builds in B20466: Diff 19410!

fix review's feebacks

Build is green

Patch application report for D5429 (id=19413)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.

Changes applied before test

commit 2925ccd089c17c735682de1fecb8c421a726b523
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/27/ for more details.

Harbormaster completed remote builds in B20469: Diff 19413.Apr 6 2021, 7:31 PM

vsellier requested review of this revision.Apr 6 2021, 7:31 PM

ardumont added inline comments.Apr 6 2021, 7:34 PM

swh/counters/__init__.py
55

fix another typo

Build is green

Patch application report for D5429 (id=19414)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.

Changes applied before test

commit 95683e1fa95a8e9261a83501747cf5661e6c737b
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/28/ for more details.

Harbormaster completed remote builds in B20470: Diff 19414.Apr 6 2021, 7:41 PM

ardumont edited the summary of this revision. (Show Details)Apr 7 2021, 8:55 AM

Looks rather good to me.
Some suggestions inline.

For the sake of the quantity of suggestions, i'll require changes.

Note that I also fixed lots of typos in the diff description,
please amend your commit message accordingly ;)

swh/counters/history.py
106
110	Should work without that change but mypy may complain without it (in the context of the suggested type change)
swh/counters/tests/test_history.py
131	Please add a docstring describing in one line what this test checks. The test name might not be enough.
147	Make that a test utility function `configure_request_mocks` or even a fixture? As you are reusing it in the next test, that will allow us to focus on the test and not its implementation.
161
173
175	with the renaming all around. content is a bit overloaded in our context in general ;)
212	same suggestions as previous test.

This revision now requires changes to proceed.Apr 7 2021, 9:10 AM

fix typos on the commit message

Build is green

Patch application report for D5429 (id=19416)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.

Changes applied before test

commit 0dbf9566b675232bbf438218db098bb7f651c654
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation of this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggregated
    - Add the api endpoint to build these files, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/29/ for more details.

Harbormaster completed remote builds in B20473: Diff 19416.Apr 7 2021, 10:11 AM

update according the review's feedbacks

Build is green

Patch application report for D5429 (id=19429)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.

Changes applied before test

commit 1b262c37b76879f10ea77ae3199f96a7cccb7090
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation of this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggregated
    - Add the api endpoint to build these files, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/30/ for more details.

Harbormaster completed remote builds in B20478: Diff 19429.Apr 7 2021, 12:08 PM

Thanks.

This revision is now accepted and ready to land.Apr 7 2021, 12:09 PM

Closed by commit rDCNT1b262c37b768: Manage and expose the historical data (authored by vsellier). · Explain WhyApr 7 2021, 12:13 PM

This revision was automatically updated to reflect the committed changes.

vsellier added a commit: rDCNT1b262c37b768: Manage and expose the historical data.

Manage and expose the historical data
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D5429 (id=19402)

Changes applied before test

Patch application report for D5429 (id=19410)

Changes applied before test

Patch application report for D5429 (id=19413)

Changes applied before test

Patch application report for D5429 (id=19414)

Changes applied before test

Patch application report for D5429 (id=19416)

Changes applied before test

Patch application report for D5429 (id=19429)

Changes applied before test

Revision Contents
Changeset List

Diff 19413

MANIFEST.in

requirements-test.txt

swh/counters/init.py

swh/counters/api/server.py

swh/counters/history.py

swh/counters/interface.py

swh/counters/tests/data/content.json

swh/counters/tests/data/foo.json

swh/counters/tests/data/revision.json

swh/counters/tests/test_history.py

swh/counters/tests/test_init.py

Manage and expose the historical dataClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D5429 (id=19402)

Changes applied before test

Patch application report for D5429 (id=19410)

Changes applied before test

Patch application report for D5429 (id=19413)

Changes applied before test

Patch application report for D5429 (id=19414)

Changes applied before test

Patch application report for D5429 (id=19416)

Changes applied before test

Patch application report for D5429 (id=19429)

Changes applied before test

Revision ContentsChangeset List

Diff 19413

MANIFEST.in

requirements-test.txt

swh/counters/__init__.py

swh/counters/api/server.py

swh/counters/history.py

swh/counters/interface.py

swh/counters/tests/data/content.json

swh/counters/tests/data/foo.json

swh/counters/tests/data/revision.json

swh/counters/tests/test_history.py

swh/counters/tests/test_init.py

Manage and expose the historical data
ClosedPublic
Actions

Revision Contents
Changeset List

swh/counters/init.py