Page MenuHomeSoftware Heritage

Manage and expose the historical data
ClosedPublic

Authored by vsellier on Apr 6 2021, 4:25 PM.

Details

Summary
  • Convert the former script managed by puppet to a tested implementation of this module
  • Add the api endpoint needed by the webapp to retrieve the file with static historical data and the live data from prometheus aggregated
  • Add the api endpoint to build these files, it will be called regularly by a crontab

The build will fail until D5428 will be released

Related to T3165

Diff Detail

Repository
rDCNT Archive counters
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 20466
Build 31764: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 31763: arc lint + arc unit

Unit TestsFailed

TimeTest
6 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_server::test_server_index
swh_counters_server_config_on_disk = None mocker = <pytest_mock.plugin.MockerFixture object at 0x7fca444c1710>
7 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_server::test_server_make_app_from_config_file
swh_counters_server_config_on_disk = None def test_server_make_app_from_config_file(swh_counters_server_config_on_disk):
112 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_server::test_server_metrics
local_redis = <pytest_redis.executor.RedisExecutor: "/usr/bin/r..." 0x7fca444cce48> tmp_path = PosixPath('/tmp/pytest-of-jenkins/pytest-0/test_server_metrics0') monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fca45d56e80>
9,050 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_cli::test__journal_client__key_received
7 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.counters.tests.test_cli::test__journal_client__missing_journal_config_keys
View Full Test Results (3 Failed · 34 Passed)

Event Timeline

Build has FAILED

Patch application report for D5429 (id=19402)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 926f3d813d124cf21ea860d112f4657773662996
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

Link to build: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/25/
See console output for more information: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/25/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 6 2021, 4:26 PM
Harbormaster failed remote builds in B20458: Diff 19402!
ardumont added inline comments.
swh/counters/__init__.py
51–57
swh/counters/api/history.py
43 ↗(On Diff #19402)
47 ↗(On Diff #19402)
58 ↗(On Diff #19402)

i gather this many {{{ is for escape purposes, right?

124 ↗(On Diff #19402)

probably a duplicate from the main swh.counters.history that i just started reading.

lgtm

couple of remarks.

I'll do another pass once the build is fine (once you release the D5428 diff ;)

swh/counters/api/history.py
124 ↗(On Diff #19402)

I mean the all file api/history.py.

swh/counters/interface.py
39

Remove an old unneeded file

Build has FAILED

Patch application report for D5429 (id=19410)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 8bd4c28f1c51ecb4192db8c21a6de121635f6934
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

Link to build: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/26/
See console output for more information: https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/26/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 6 2021, 6:52 PM
Harbormaster failed remote builds in B20466: Diff 19410!

Build is green

Patch application report for D5429 (id=19413)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 2925ccd089c17c735682de1fecb8c421a726b523
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/27/ for more details.

swh/counters/__init__.py
54

Build is green

Patch application report for D5429 (id=19414)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 95683e1fa95a8e9261a83501747cf5661e6c737b
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation on this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggragated
    - Add the api endpoint to build these file, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/28/ for more details.

ardumont requested changes to this revision.Apr 7 2021, 9:10 AM

Looks rather good to me.
Some suggestions inline.

For the sake of the quantity of suggestions, i'll require changes.

Note that I also fixed lots of typos in the diff description,
please amend your commit message accordingly ;)

swh/counters/history.py
106
110

Should work without that change but mypy may complain without it (in the context of the suggested type change)

swh/counters/tests/test_history.py
131

Please add a docstring describing in one line what this test checks.
The test name might not be enough.

147

Make that a test utility function configure_request_mocks or even a fixture?
As you are reusing it in the next test, that will allow us to focus on the test
and not its implementation.

161
173
175

with the renaming all around.

content is a bit overloaded in our context in general ;)

212

same suggestions as previous test.

This revision now requires changes to proceed.Apr 7 2021, 9:10 AM
vsellier edited the summary of this revision. (Show Details)

fix typos on the commit message

Build is green

Patch application report for D5429 (id=19416)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 0dbf9566b675232bbf438218db098bb7f651c654
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation of this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggregated
    - Add the api endpoint to build these files, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/29/ for more details.

vsellier marked 7 inline comments as done.

update according the review's feedbacks

Build is green

Patch application report for D5429 (id=19429)

Rebasing onto 2fef22b257...

Current branch diff-target is up to date.
Changes applied before test
commit 1b262c37b76879f10ea77ae3199f96a7cccb7090
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Mar 30 16:02:31 2021 +0200

    Manage and expose the historical data
    
    - Convert the former script managed by puppet to a tested
      implementation of this module
    - Add the api endpoint needed by the webapp to retrieve
      the file with static historical data and the live data
      from prometheus aggregated
    - Add the api endpoint to build these files, it will be called
      regularly by a crontab
    
    Related to T3165

See https://jenkins.softwareheritage.org/job/DCNT/job/tests-on-diff/30/ for more details.

This revision is now accepted and ready to land.Apr 7 2021, 12:09 PM
This revision was automatically updated to reflect the committed changes.