Differential D7687

staging: Activate metadata loader for git loader
ClosedPublic
Actions

Authored by ardumont on Apr 26 2022, 4:28 PM.

Details

Reviewers

None

Group Reviewers

System administrators

Maniphest Tasks

T4193: staging: Deploy metadata loader

Commits

rSPSITE7d6db46bef09: staging: Activate metadata loader for git loader

Summary

This will install the extra python3-swh.loader.metadata package for staging workers.

By default, this is off for production and on for staging.

When on, it's also installing the extra set of credentials in the loader_git's
configuration to allow the use of such credentials when communicating with the forges.

Related to T4193

Test Plan

Expectedly:

octo-diff on worker.production does nothing
octo-diff on worker.staging wants to do as described.

$SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker0.internal.staging.swh.network
Found host worker0.internal.staging.swh.network
...
*** Running octocatalog-diff on host worker0.internal.staging.swh.network
I, [2022-04-26T16:22:23.543299 #881891]  INFO -- : Catalogs compiled for worker0.internal.staging.swh.network
I, [2022-04-26T16:22:23.943674 #881891]  INFO -- : Diffs computed for worker0.internal.staging.swh.network
diff origin/production/worker0.internal.staging.swh.network current/worker0.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -28,3 +28,105 @@
         - swh.loader.git.tasks.LoadDiskGitRepository
         - swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
      +credentials:
      +  github:
      +    github:
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-0-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/0/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-1-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/1/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-2-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/2/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-3-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/3/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-4-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/4/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-5-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/5/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-6-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/6/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-7-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/7/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-8-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/8/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-9-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/9/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-10-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/10/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-11-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/11/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-12-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/12/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-13-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/13/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github-14-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github/14/username
      +    github.com:
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-0-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/0/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-1-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/1/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-2-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/2/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-3-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/3/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-4-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/4/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-5-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/5/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-6-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/6/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-7-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/7/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-8-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/8/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-9-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/9/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-10-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/10/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-11-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/11/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-12-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/12/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-13-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/13/username
      +    - password: swh-deploy-worker-lister-config-credentials-github-github.com-14-password
      +      username: swh::deploy::worker::lister::config::credentials/github/github.com/14/username
      +  gitlab:
      +    gitlab:
      +    - password: swh-deploy-worker-lister-config-credentials-gitlab-gitlab-0-password
      +      username: swh::deploy::worker::lister::config::credentials/gitlab/gitlab/0/username
      +    gitlab.com:
      +    - password: swh-deploy-worker-lister-config-credentials-gitlab-gitlab.com-0-password
      +      username: swh::deploy::worker::lister::config::credentials/gitlab/gitlab.com/0/username
      +  phabricator:
      +    blender:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-blender-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/blender/0/username
      +    developer.blender.org:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-developer.blender.org-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/developer.blender.org/0/username
      +    forge.softwareheritage.org:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-forge.softwareheritage.org-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/forge.softwareheritage.org/0/username
      +    kde:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-kde-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/kde/0/username
      +    llvm:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-llvm-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/llvm/0/username
      +    phabricator.kde.org:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-phabricator.kde.org-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/phabricator.kde.org/0/username
      +    phabricator.wikimedia.org:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-phabricator.wikimedia.org-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/phabricator.wikimedia.org/0/username
      +    reviews.llvm.org:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-reviews.llvm.org-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/reviews.llvm.org/0/username
      +    swh:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-swh-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/swh/0/username
      +    wikimedia:
      +    - password: swh-deploy-worker-lister-config-credentials-phabricator-wikimedia-0-password
      +      username: swh::deploy::worker::lister::config::credentials/phabricator/wikimedia/0/username
      _
*******************************************
+ Package[python3-swh.loader.metadata] =>
   parameters =>
      "ensure": "installed"
*******************************************
  Profile::Swh::Deploy::Worker::Instance[checker_deposit] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[indexer_content_mimetype] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[indexer_fossology_license] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[indexer_origin_intrinsic_metadata] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[lister] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_archive] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_bzr] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_cran] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_cvs] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_debian] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_deposit] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_git] =>
   parameters =>
     extra_config =>
      + {"credentials"=>{"github"=>{"github"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-github-github-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/0/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-1-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/1/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-2-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/2/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-3-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/3/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-4-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/4/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-5-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/5/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-6-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/6/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-7-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/7/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-8-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/8/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-9-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/9/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-10-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/10/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-11-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/11/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-12-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/12/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-13-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/13/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-14-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/14/username"}], "github.com"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/0/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-1-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/1/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-2-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/2/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-3-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/3/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-4-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/4/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-5-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/5/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-6-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/6/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-7-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/7/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-8-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/8/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-9-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/9/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-10-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/10/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-11-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/11/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-12-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/12/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-13-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/13/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-14-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/14/username"}]}, "gitlab"=>{"gitlab"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-gitlab-gitlab-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/gitlab/gitlab/0/username"}], "gitlab.com"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-gitlab-gitlab.com-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/gitlab/gitlab.com/0/username"}]}, "phabricator"=>{"blender"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-blender-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/blender/0/username"}], "developer.blender.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-developer.blender.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/developer.blender.org/0/username"}], "forge.softwareheritage.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-forge.softwareheritage.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/forge.softwareheritage.org/0/username"}], "kde"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-kde-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/kde/0/username"}], "llvm"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-llvm-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/llvm/0/username"}], "phabricator.kde.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-phabricator.kde.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/phabricator.kde.org/0/username"}], "phabricator.wikimedia.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-phabricator.wikimedia.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/phabricator.wikimedia.org/0/username"}], "reviews.llvm.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-reviews.llvm.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/reviews.llvm.org/0/username"}], "swh"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-swh-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/swh/0/username"}], "wikimedia"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-wikimedia-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/wikimedia/0/username"}]}}}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_high_priority] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_maven] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_mercurial] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_nixguix] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_npm] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_opam] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_pypi] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[loader_svn] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
  Profile::Swh::Deploy::Worker::Instance[vault_cooker] =>
   parameters =>
     extra_config =>
      + {}
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network

Vagrant checks are also ok:

$ vagrant up staging-worker0
...
root@worker0:~# grep -c credentials /etc/softwareheritage/loader_git.yml
85
root@worker0:~# dpkg -l python3-swh.loader.metadata
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                        Version              Architecture Description
+++-===========================-====================-============-=============================================
ii  python3-swh.loader.metadata 0.0.1-1~swh2~bpo10+1 all          Software Heritage Extrinsic Metadata Fetchers