Page MenuHomeSoftware Heritage

deposit: Add equivalent production integration test to staging
Closed, MigratedEdits Locked

Description

Staging deposit should be tested the same as the production.
So problem after deployment could be detected early enough.

Event Timeline

ardumont triaged this task as Normal priority.Mar 15 2021, 3:56 PM
ardumont created this task.

Deployed (i think):

root@pergamon:~# puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for pergamon.softwareheritage.org
Info: Applying configuration version '1615908229'
Notice: /Stage[main]/Profile::Bind_server::Primary/Resource_record[petitpalais/CNAME]/ensure: created
Notice: /Stage[main]/Profile::Icinga2::Master/Icinga2::Object::Host[pergamon.softwareheritage.org]/Icinga2::Object[icinga2::object::Host::pergamon.softwareheritage.org]/Concat[/etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf]/File[/etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf]/content:
--- /etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf      2020-01-22 13:44:53.110039721 +0000
+++ /tmp/puppet-file20210316-3640122-8dovp5     2021-03-16 15:24:43.620004036 +0000
@@ -16,17 +16,3 @@
   vars.virtual_machine = true
      vars.distro = "Debian"
       }
       -
       -object Service "Check deposit end-to-end" {
       -  import "generic-service-check-e2e"
       -
       -  host_name = "pergamon.softwareheritage.org"
       -  check_command = "check-deposit-cmd"
       -}
       -
       -object Service "Check vault end-to-end" {
       -  import "generic-service-check-e2e"
       -
       -  host_name = "pergamon.softwareheritage.org"
       -  check_command = "check-vault-cmd"
       -}

Info: Computing checksum on file /etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf
Info: /Stage[main]/Profile::Icinga2::Master/Icinga2::Object::Host[pergamon.softwareheritage.org]/Icinga2::Object[icinga2::object::Host::pergamon.softwareheritage.org]/Concat[/etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf]/File[/etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf]: Filebucketed /etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf to puppet with sum 8a4cf069250d00837a192a612d8de9eb
Notice: /Stage[main]/Profile::Icinga2::Master/Icinga2::Object::Host[pergamon.softwareheritage.org]/Icinga2::Object[icinga2::object::Host::pergamon.softwareheritage.org]/Concat[/etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf]/File[/etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf]/content: content changed '{md5}8a4cf069250d00837a192a612d8de9eb' to '{md5}ea8e57bd57ac48c10e98fed058ea977e'
Info: Concat[/etc/icinga2/zones.d/master/pergamon.softwareheritage.org.conf]: Scheduling refresh of Class[Icinga2::Service]
Notice: /Stage[main]/Profile::Icinga2::Master/Profile::Icinga2::Objects::E2e_checks_deposit[End-to-end Deposit Test(s) in staging]/Icinga2::Object::Checkcommand[staging-check-deposit-cmd]/Icinga2::Object[icinga2::object::CheckCommand::staging-check-deposit-cmd]/Concat[/etc/icinga2/conf.d/e2e-checks.conf]/File[/etc/icinga2/conf.d/e2e-checks.conf]/content:
--- /etc/icinga2/conf.d/e2e-checks.conf 2021-01-25 13:12:02.395541078 +0000
+++ /tmp/puppet-file20210316-3640122-lmzx2l     2021-03-16 15:24:46.272007390 +0000
@@ -1,14 +1,27 @@
 # This file is managed by Puppet. DO NOT EDIT.

-object CheckCommand "check-deposit-cmd" {
+object CheckCommand "production-check-deposit-cmd" {
   import "plugin-check-command"

   command = [ "/usr/bin/swh", "icinga_plugins", "-w", 600, "check-deposit", "--server", "https://deposit.softwareheritage.org/1", "--username", "swh", "--password", "girdle-tumble-spiral-valid-robin", "--collection", "swh", "--poll-interval", 1, "single", "--archive", "/usr/share/swh/icinga-plugins/data/deposit/jesuisgpl.tgz", "--metadata", "/usr/share/swh/icinga-plugins/data/deposit/jesuisgpl.tgz.xml", ]
      timeout = 900
       }

-object CheckCommand "check-vault-cmd" {
+object CheckCommand "production-check-vault-cmd" {
   import "plugin-check-command"

   command = [ "/usr/bin/swh", "icinga_plugins", "check-vault", "--swh-storage-url", "http://saam.internal.softwareheritage.org:5002", "--swh-web-url", "https://archive.softwareheritage.org", "directory", ]
    }
    +
    +object CheckCommand "staging-check-deposit-cmd" {
    +  import "plugin-check-command"
    +
    +  command = [ "/usr/bin/swh", "icinga_plugins", "-w", 600, "check-deposit", "--server", "https://deposit.staging.swh.network/1", "--username", "swh", "--password", "girdle-tumble-spiral-valid-robin", "--collection", "swh", "--poll-interval", 1, "single", "--archive", "/usr/share/swh/icinga-plugins/data/deposit/jesuisgpl.tgz", "--metadata", "/usr/share/swh/icinga-plugins/data/deposit/jesuisgpl.tgz.xml", ]
    +  timeout = 900
    +}
    +
    +object CheckCommand "staging-check-vault-cmd" {
    +  import "plugin-check-command"
    +
    +  command = [ "/usr/bin/swh", "icinga_plugins", "check-vault", "--swh-storage-url", "http://storage1.internal.staging.swh.network:5002/", "--swh-web-url", "https://webapp.staging.swh.network", "directory", ]
    +}

Info: Computing checksum on file /etc/icinga2/conf.d/e2e-checks.conf
Info: /Stage[main]/Profile::Icinga2::Master/Profile::Icinga2::Objects::E2e_checks_deposit[End-to-end Deposit Test(s) in staging]/Icinga2::Object::Checkcommand[staging-check-deposit-cmd]/Icinga2::Object[icinga2::object::CheckCommand::staging-check-deposit-cmd]/Concat[/etc/icinga2/conf.d/e2e-checks.conf]/File[/etc/icinga2/conf.d/e2e-checks.conf]: Filebucketed /etc/icinga2/conf.d/e2e-checks.conf to puppet with sum 5781f2c3c5ae4e1706ac55e28150a642
Notice: /Stage[main]/Profile::Icinga2::Master/Profile::Icinga2::Objects::E2e_checks_deposit[End-to-end Deposit Test(s) in staging]/Icinga2::Object::Checkcommand[staging-check-deposit-cmd]/Icinga2::Object[icinga2::object::CheckCommand::staging-check-deposit-cmd]/Concat[/etc/icinga2/conf.d/e2e-checks.conf]/File[/etc/icinga2/conf.d/e2e-checks.conf]/content: content changed '{md5}5781f2c3c5ae4e1706ac55e28150a642' to '{md5}d0c2167b81249c149e5da3b792c781b2'
Info: Concat[/etc/icinga2/conf.d/e2e-checks.conf]: Scheduling refresh of Class[Icinga2::Service]
Notice: /Stage[main]/Profile::Icinga2::Master/Profile::Icinga2::Objects::E2e_checks_deposit[End-to-end Deposit Test(s) in staging]/Icinga2::Object::Service[staging-check-deposit]/Icinga2::Object[icinga2::object::Service::staging-check-deposit]/Concat[/etc/icinga2/zones.d//pergamon.softwareheritage.org.conf]/File[/etc/icinga2/zones.d//pergamon.softwareheritage.org.conf]/ensure: defined content as '{md5}1a8a8dabdde13bcdac3dd6f5aca47c73'
Info: Concat[/etc/icinga2/zones.d//pergamon.softwareheritage.org.conf]: Scheduling refresh of Class[Icinga2::Service]
Info: Class[Icinga2::Service]: Scheduling refresh of Service[icinga2]
Notice: /Stage[main]/Icinga2::Service/Service[icinga2]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 38.88 seconds

Those exported resources are actually a pain to develop and tests...
The feedback loop is bad.

I got multiple typos which passed undetected through octocatalog-diff.
And we cannot test it through our vagrant tool just yet...

After cross-checking, it's not complete just yet...
I don't see the check any longer in the icinga interface [1]

Checking back the diff...

-
-object Service "Check deposit end-to-end" {
-  import "generic-service-check-e2e"
-
-  host_name = "pergamon.softwareheritage.org"
-  check_command = "check-deposit-cmd"
-}
-
-object Service "Check vault end-to-end" {
-  import "generic-service-check-e2e"
-
-  host_name = "pergamon.softwareheritage.org"
-  check_command = "check-vault-cmd"
-}

I don't actually see the equivalent expected staging-check-deposit-cmd,
production-check-deposit-cmd, staging-check-vault-cmd, production-check-vault-cmd appear...
So there is that.

So might be my v2 tryout (D5256) was not too bad after all...

[1] https://icinga.softwareheritage.org/search?q=deposit

ardumont changed the task status from Open to Work in Progress.Mar 16 2021, 4:45 PM
ardumont moved this task from Backlog to in-progress on the System administration board.

Really deployed now with the fix ^

ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.