Page MenuHomeSoftware Heritage

vault: Open the beginning of the tarball to check it makes sense.
ClosedPublic

Authored by vlorentz on Mar 4 2022, 3:50 PM.

Details

Summary

This can catch potential errors like file corruption, wrong encoding,
crashes after response headers were sent, etc.

Diff Detail

Repository
rDICP Icinga plugins
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 27291
Build 42693: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 42692: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D7298 (id=26404)

Could not rebase; Attempt merge onto 27a2ef97c1...

Updating 27a2ef9..ea2fa8b
Fast-forward
 swh/icinga_plugins/tests/test_vault.py   | 111 +++++++++++++++++++++++++------
 swh/icinga_plugins/tests/web_scenario.py |  22 +++---
 swh/icinga_plugins/vault.py              |  61 ++++++++++++++---
 3 files changed, 153 insertions(+), 41 deletions(-)
Changes applied before test
commit ea2fa8bbc392ff24b220ff313f715561282b7740
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 15:50:46 2022 +0100

    vault: Open the beginning of the tarball to check it makes sense.
    
    This can catch potential errors like file corruption, wrong encoding,
    crashes after response headers were sent, etc.

commit a35d451e716c127ce47cd10f41bb0a87f6056d69
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 15:45:54 2022 +0100

    Rename dir_id to DIR_ID
    
    It's a global constant, so it makes more sense to uppercase it.

commit 514a37ff6830e287f85ee2046bdf65b5e2ddbd82
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 15:17:43 2022 +0100

    vault: Check bundle download from the public API.

See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/56/ for more details.

Seems worth a rebase on top of D7264 too, but I guess it'll need some more changes

Build is green

Patch application report for D7298 (id=26410)

Could not rebase; Attempt merge onto c698de2711...

Updating c698de2..408e0fe
Fast-forward
 swh/icinga_plugins/tests/test_vault.py   | 158 +++++++++++++++++++++++++------
 swh/icinga_plugins/tests/web_scenario.py |  22 +++--
 swh/icinga_plugins/vault.py              | 109 ++++++++++++---------
 3 files changed, 207 insertions(+), 82 deletions(-)
Changes applied before test
commit 408e0fe694010d5e7d48caf29f980a73e459c81c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 15:50:46 2022 +0100

    vault: Open the beginning of the tarball to check it makes sense.
    
    This can catch potential errors like file corruption, wrong encoding,
    crashes after response headers were sent, etc.

commit c0de368aa7af5e89a6127ff04fc368a07296b1e4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 16:16:02 2022 +0100

    vault: Check content-type

commit de78c81353f3cd51b7a5ea9054ed8b808fe0f3a1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 16:06:36 2022 +0100

    Rename dir_id to DIR_ID
    
    It's a global constant, so it makes more sense to uppercase it.

commit b84160621e62ee2900e8b8ae19f14d6f5c3ff4e7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 16:04:51 2022 +0100

    vault: Un-nest checks
    
    This will allow more checks later, while keeping the code flow linear.

See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/61/ for more details.

Yep, that looks good to me.

Have you tried it out on staging? swh icinga_plugins check-vault --swh-storage-url http://webapp.internal.staging.swh.network:5002 --swh-web-url https://webapp.staging.swh.network/ directory should work

This revision is now accepted and ready to land.Mar 4 2022, 4:55 PM

Fix crash on tarballs with a root dir as member.

Build is green

Patch application report for D7298 (id=26416)

Could not rebase; Attempt merge onto c698de2711...

Updating c698de2..c12f520
Fast-forward
 swh/icinga_plugins/tests/test_vault.py   | 162 +++++++++++++++++++++++++------
 swh/icinga_plugins/tests/web_scenario.py |  22 +++--
 swh/icinga_plugins/vault.py              | 112 ++++++++++++---------
 3 files changed, 214 insertions(+), 82 deletions(-)
Changes applied before test
commit c12f5202f0be7186591080d8c1533a4bb1a59a20
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 15:50:46 2022 +0100

    vault: Open the beginning of the tarball to check it makes sense.
    
    This can catch potential errors like file corruption, wrong encoding,
    crashes after response headers were sent, etc.

commit c0de368aa7af5e89a6127ff04fc368a07296b1e4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 16:16:02 2022 +0100

    vault: Check content-type

commit de78c81353f3cd51b7a5ea9054ed8b808fe0f3a1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 16:06:36 2022 +0100

    Rename dir_id to DIR_ID
    
    It's a global constant, so it makes more sense to uppercase it.

commit b84160621e62ee2900e8b8ae19f14d6f5c3ff4e7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 4 16:04:51 2022 +0100

    vault: Un-nest checks
    
    This will allow more checks later, while keeping the code flow linear.

See https://jenkins.softwareheritage.org/job/DICP/job/tests-on-diff/62/ for more details.