Page MenuHomeSoftware Heritage
Differential D5429

Manage and expose the historical data
ClosedPublic
Actions

Authored by vsellier on Apr 6 2021, 4:25 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T3165: Generate historical data from the new counters series
Commits
rDCNT1b262c37b768: Manage and expose the historical data
Summary
  • Convert the former script managed by puppet to a tested implementation of this module
  • Add the api endpoint needed by the webapp to retrieve the file with static historical data and the live data from prometheus aggregated
  • Add the api endpoint to build these files, it will be called regularly by a crontab

The build will fail until D5428 will be released

Related to T3165

Diff Detail

Repository
rDCNT Archive counters
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 20466
Build 31764: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 31763: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
6 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_server::test_server_index
swh_counters_server_config_on_disk = None mocker = <pytest_mock.plugin.MockerFixture object at 0x7fca444c1710>
7 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_server::test_server_make_app_from_config_file
swh_counters_server_config_on_disk = None def test_server_make_app_from_config_file(swh_counters_server_config_on_disk):
112 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_server::test_server_metrics
local_redis = <pytest_redis.executor.RedisExecutor: "/usr/bin/r..." 0x7fca444cce48> tmp_path = PosixPath('/tmp/pytest-of-jenkins/pytest-0/test_server_metrics0') monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fca45d56e80>
9,050 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_cli::test__journal_client__key_received
7 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_cli::test__journal_client__missing_journal_config_keys
View Full Test Results (3 Failed · 34 Passed)

Event Timeline

vsellier created this revision.Apr 6 2021, 4:25 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 6 2021, 4:25 PM
swh-public-ci added a comment.Apr 6 2021, 4:26 PM

Build has FAILED

Patch application report for D5429 (id=19402)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 926f3d813d124cf21ea860d112f4657773662996
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

Link to build: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/25/
See console output for more information: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/25/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 6 2021, 4:26 PM
Harbormaster failed remote builds in B20458: Diff 19402!
ardumont added a subscriber: ardumont.Apr 6 2021, 5:08 PM
ardumont added inline comments.
swh/counters/__init__.py
51–57
swh/counters/api/history.py
43 ↗(On Diff #19402)
47 ↗(On Diff #19402)
58 ↗(On Diff #19402)

i gather this many {{{ is for escape purposes, right?

124 ↗(On Diff #19402)

probably a duplicate from the main swh.counters.history that i just started reading.

ardumont added a comment.Apr 6 2021, 5:25 PM

lgtm

couple of remarks.

I'll do another pass once the build is fine (once you release the D5428 diff ;)

swh/counters/api/history.py
124 ↗(On Diff #19402)

I mean the all file api/history.py.

swh/counters/interface.py
39
ardumont mentioned this in D5428: Allow to use several backends with a RPCServerApp.Apr 6 2021, 5:34 PM
vsellier updated this revision to Diff 19410.Apr 6 2021, 6:51 PM

Remove an old unneeded file

swh-public-ci added a comment.Apr 6 2021, 6:52 PM

Build has FAILED

Patch application report for D5429 (id=19410)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 8bd4c28f1c51ecb4192db8c21a6de121635f6934
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

Link to build: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/26/
See console output for more information: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/26/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 6 2021, 6:52 PM
Harbormaster failed remote builds in B20466: Diff 19410!
vsellier updated this revision to Diff 19413.Apr 6 2021, 7:30 PM

fix review's feebacks

swh-public-ci added a comment.Apr 6 2021, 7:31 PM

Build is green

Patch application report for D5429 (id=19413)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 2925ccd089c17c735682de1fecb8c421a726b523
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/27/ for more details.

Harbormaster completed remote builds in B20469: Diff 19413.Apr 6 2021, 7:31 PM
vsellier requested review of this revision.Apr 6 2021, 7:31 PM
ardumont added inline comments.Apr 6 2021, 7:34 PM
swh/counters/__init__.py
54
vsellier updated this revision to Diff 19414.Apr 6 2021, 7:40 PM

fix another typo

swh-public-ci added a comment.Apr 6 2021, 7:41 PM

Build is green

Patch application report for D5429 (id=19414)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 95683e1fa95a8e9261a83501747cf5661e6c737b
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/28/ for more details.

Harbormaster completed remote builds in B20470: Diff 19414.Apr 6 2021, 7:41 PM
ardumont edited the summary of this revision. (Show Details)Apr 7 2021, 8:55 AM
ardumont requested changes to this revision.Apr 7 2021, 9:10 AM

Looks rather good to me.
Some suggestions inline.

For the sake of the quantity of suggestions, i'll require changes.

Note that I also fixed lots of typos in the diff description,
please amend your commit message accordingly ;)

swh/counters/history.py
106
110

Should work without that change but mypy may complain without it (in the context of the suggested type change)

swh/counters/tests/test_history.py
131

Please add a docstring describing in one line what this test checks.
The test name might not be enough.

147

Make that a test utility function configure_request_mocks or even a fixture?
As you are reusing it in the next test, that will allow us to focus on the test
and not its implementation.

161
173
175

with the renaming all around.

content is a bit overloaded in our context in general ;)

212

same suggestions as previous test.

This revision now requires changes to proceed.Apr 7 2021, 9:10 AM
vsellier updated this revision to Diff 19416.Apr 7 2021, 10:10 AM
vsellier edited the summary of this revision. (Show Details)

fix typos on the commit message

swh-public-ci added a comment.Apr 7 2021, 10:11 AM

Build is green

Patch application report for D5429 (id=19416)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 0dbf9566b675232bbf438218db098bb7f651c654
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation of this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggregated
    - Add the api endpoint to build these files, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/29/ for more details.

Harbormaster completed remote builds in B20473: Diff 19416.Apr 7 2021, 10:11 AM
vsellier updated this revision to Diff 19429.Apr 7 2021, 12:07 PM
vsellier marked 7 inline comments as done.

update according the review's feedbacks

swh-public-ci added a comment.Apr 7 2021, 12:08 PM

Build is green

Patch application report for D5429 (id=19429)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 1b262c37b76879f10ea77ae3199f96a7cccb7090
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation of this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggregated
    - Add the api endpoint to build these files, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/30/ for more details.

Harbormaster completed remote builds in B20478: Diff 19429.Apr 7 2021, 12:08 PM
ardumont accepted this revision.Apr 7 2021, 12:09 PM

Thanks.

This revision is now accepted and ready to land.Apr 7 2021, 12:09 PM
Closed by commit rDCNT1b262c37b768: Manage and expose the historical data (authored by vsellier). · Explain WhyApr 7 2021, 12:13 PM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rDCNT1b262c37b768: Manage and expose the historical data.

Revision Contents
Changeset List

PathSize
1 line
1 line
swh/
counters/
33 lines
api/
9 lines
122 lines
15 lines
tests/
data/
32 lines
4 lines
32 lines
212 lines
21 lines

Diff 19410

View Options

MANIFEST.in

Loading...
View Options

requirements-test.txt

Loading...
View Options

swh/counters/__init__.py

Loading...
View Options

swh/counters/api/server.py

Loading...
View Options

swh/counters/history.py

Loading...
View Options

swh/counters/interface.py

Loading...
View Options

swh/counters/tests/data/content.json

Loading...
View Options

swh/counters/tests/data/foo.json

Loading...
View Options

swh/counters/tests/data/revision.json

Loading...
View Options

swh/counters/tests/test_history.py

Loading...
View Options

swh/counters/tests/test_init.py

Loading...
About · Developer information · Legal