Page MenuHomeSoftware Heritage

Add end to end save code now check
ClosedPublic

Authored by ardumont on Apr 15 2021, 6:52 PM.

Details

Summary

This defines an end to end check which trigger for an dedicated origin with a given type
a save code now requests.

This then regularly polls the status out of the save code now api. Depending on the time
it took to actually ingest and the ingestion status, it will return the status of said
ingestion and roughly the time it took.

The end goal being to deploy such checks into icinga and being notified when a save code
now request takes too long.

Example use:

$ swh icinga_plugins \
  check-savecodenow --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s

Related to T2117

Test Plan

Using the staging webapp api, then checking up until the code worked or being rate limited, whatever came first:

$ swh icinga_plugins check-savecodenow --swh-web-url https://webapp.staging.swh.network origin https://github.com/BRGM/map2loop-2 --visit-type git

SAVECODENOW WARNING - Save code now request for (git, https://github.com/BRGM/map2loop-2) took 51.18s and succeeded.
| 'total_time' = 51.18s

$ swh icinga_plugins check-savecodenow --swh-web-url https://webapp.staging.swh.network origin https://github.com/divnix/devos --visit-type git
SAVECODENOW OK - Save code now request for (git, https://github.com/divnix/devos) took 32.28s and succeeded.
| 'total_time' = 32.28s

$ swh icinga_plugins check-savecodenow --swh-web-url https://webapp.staging.swh.network origin https://github.com/divnix/devos --visit-type git
SAVECODENOW CRITICAL - Save code now request for (git, https://github.com/divnix/devos) took more than 349.80s and has status: succeeded
| 'total_time' = 349.80s    # <-------------------------- pdb session and letting it hang a bit then continue after threshold

$ swh icinga_plugins check-savecodenow --swh-web-url https://webapp.staging.swh.network origin https://github.com/divnix/devos --visit-type git
Traceback (most recent call last):
  File "/home/tony/.virtualenvs/swh/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-core/swh/core/cli/__init__.py", line 185, in main
    return swh(auto_envvar_prefix="SWH")
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-icinga-plugins/swh/icinga_plugins/cli.py", line 86, in check_scn_origin
    sys.exit(SaveCodeNowCheck(ctx.obj).main())
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-icinga-plugins/swh/icinga_plugins/save_code_now.py", line 80, in main
    assert response.status_code == 200, (response, response.text)
AssertionError: (<Response [429]>, '{"exception":"Throttled","reason":"Request was throttled. Expected available in 2058 seconds."}')
$ swh icinga_plugins check-savecodenow --swh-web-url https://archive.softwareheritage.org origin https://github.com/SebaUbuntu/TWRP-device-tree-generator --visit-type git
SAVECODENOW OK - Save code now request for (git, https://github.com/SebaUbuntu/TWRP-device-tree-generator) took 20.66s and succeeded.
| 'total_time' = 20.66s

Diff Detail

Repository
rDICP Icinga plugins
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 20819
Build 32309: Phabricator diff pipeline on jenkinsJenkins console ยท Jenkins
Build 32308: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D5541 (id=19779)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit cbcd8ea35cb1af172d44c8626d8a645d4144226a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    wip: Add end to end save code now check
    
    This is a draft for now as:
    - it's hard-coding a list of origins because reasons [1]
    - it's missing coverage
    
    [1] P1007 i need to investigate
    
    Related to T2117

See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/19/ for more details.

ardumont edited the test plan for this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)

Fix typos and add a missing status check

Build is green

Patch application report for D5541 (id=19780)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit 9635f71d87ab0e04d80a437601df93782f78f91b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    wip: Add end to end save code now check
    
    This is a draft for now as:
    - it's hard-coding a list of origins because reasons [1]
    - it's missing coverage
    
    [1] P1007 i need to investigate
    
    Related to T2117

See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/20/ for more details.

swh/icinga_plugins/save_code_now.py
67

will be replaced when the other part is fixed.

71

We could try and randomly ask for picking up svn or hg mercurial origins which we did not do much over the last 3 months for example especially in staging...
so that ^

vsellier added a subscriber: vsellier.

I'm not a big fan of the random pickup of repo because it adds entropy on the result, but otherwise it's great to be able to monitor this ๐Ÿ‘

swh/icinga_plugins/save_code_now.py
48

lol

This revision is now accepted and ready to land.Apr 16 2021, 9:35 AM
swh/icinga_plugins/save_code_now.py
48

;)

swh/icinga_plugins/tests/test_save_code_now.py
4

/me will play fill in the blank

I'm not a big fan of the random pickup of repo because it adds entropy on the result, but otherwise it's great to be able to monitor this ๐Ÿ‘

yes, i will adapt so the check explicitely requires an origin instead (i'm currenty doing it)
It will keep the code simpler.

And we can always compose with something which will fetch randomly an origin and type from the storage later if need be (outside of the plugin script).
The unix way.

Build is green

Patch application report for D5541 (id=19784)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit efb5b48288a6630cad4a4be0b7b8ade91c3b5af3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which pulls a recent origin from the storage (status
    full from the last 3 months [1]) and triggers a save code now requests for said origin.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/21/ for more details.
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)

Start testing

This refactors common behavior between existing utils test a bit

Build is green

Patch application report for D5541 (id=19799)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit b78c139390e66e24c58565301eac2de3de4346bb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which pulls a recent origin from the storage (status
    full from the last 3 months [1]) and triggers a save code now requests for said origin.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/22/ for more details.

Build is green

Patch application report for D5541 (id=19801)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit 916570f748a191876d14cff8b22c9d9cd7e3d4ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which pulls a recent origin from the storage (status
    full from the last 3 months [1]) and triggers a save code now requests for said origin.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/23/ for more details.

Exceeding threshold scenario

Build is green

Patch application report for D5541 (id=19802)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit 4ad5765a3cdbd5cf4d6db777ab90c75575782cfe
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which pulls a recent origin from the storage (status
    full from the last 3 months [1]) and triggers a save code now requests for said origin.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/24/ for more details.

Last conditional scenario, plain failure

Build is green

Patch application report for D5541 (id=19803)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit 6af2eefdd36834ed5ac294d8e53c72c0edc25142
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which pulls a recent origin from the storage (status
    full from the last 3 months [1]) and triggers a save code now requests for said origin.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/25/ for more details.

Build is green

Patch application report for D5541 (id=19804)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit 45a042fcbfa97920f7c6178e32c1fff02502dfd7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which pulls a recent origin from the storage (status
    full from the last 3 months [1]) and triggers a save code now requests for said origin.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/26/ for more details.
ardumont retitled this revision from wip: Add end to end save code now check to Add end to end save code now check.Apr 16 2021, 3:26 PM
ardumont edited the summary of this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)

Reuse variable instead of duplicating (i forgot to reuse it)

Build is green

Patch application report for D5541 (id=19806)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit 7468d5783794d38ee714a16b4d49eba72d321099
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which pulls a recent origin from the storage (status
    full from the last 3 months [1]) and triggers a save code now requests for said origin.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/27/ for more details.

Build is green

Patch application report for D5541 (id=19808)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit 945e45f941d16cd7a61532573139a03cad4f7ef9
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which trigger for an dedicated origin with a given type
    a save code now requests.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/28/ for more details.

Rework failed scenario docstring

Build is green

Patch application report for D5541 (id=19809)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit ea93e2f170bd0c228fd8500d1adaf31b76becb80
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which trigger for an dedicated origin with a given type
    a save code now requests.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/29/ for more details.

lgtm

swh/icinga_plugins/tests/test_save_code_now.py
49

The return of the random ;)

199

random guy :)

For the fun, 100% coverage on the module now ;)

Build is green

Patch application report for D5541 (id=19810)

Rebasing onto 36f52101f0...

Current branch diff-target is up to date.
Changes applied before test
commit d65fd19afd04e20f1bf9da98cd5565e521ab78bd
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Apr 15 18:44:44 2021 +0200

    Add end to end save code now check
    
    This defines an end to end check which trigger for an dedicated origin with a given type
    a save code now requests.
    
    This then regularly polls the status out of the save code now api. Depending on the time
    it took to actually ingest and the ingestion status, it will return the status of said
    ingestion and roughly the time it took.
    
    The end goal being to deploy such checks into icinga and being notified when a save code
    now request takes too long.
    
    Example use:
$ swh icinga_plugins \
  check-savecodenow --swh-storage-url http://localhost:5002/ \
  --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s
```

Related to T2117
See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/30/ for more details.
This revision was automatically updated to reflect the committed changes.