diff --git a/sysadm/puppet/howto-deploy-puppet-change.rst b/sysadm/puppet/howto-deploy-puppet-change.rst index 2c45918..560b943 100644 --- a/sysadm/puppet/howto-deploy-puppet-change.rst +++ b/sysadm/puppet/howto-deploy-puppet-change.rst @@ -1,9 +1,54 @@ .. _puppet_deploy_change: How to deploy a Puppet change ============================= -.. todo:: - This page is a work in progress. Please refer to the `existing documentation - `_ and `the - wiki page `_. +.. admonition:: Intended audience + :class: important + + staff members with enough permissions to deploy + +Once done with the development changes and those are landed + +.. _puppet_deploy_automated: + +Automated +~~~~~~~~~ + +Once your change is landed in the *swh-site* repository, you need to deploy the manifest +on the puppet master. + +Then puppet agent will refresh themselves and apply the changes themselves every 30min. + +.. code:: + + you@pergamon$ sudo swh-puppet-master-deploy + +.. _puppet_semi_automated: + +Semi-automated +~~~~~~~~~~~~~~ + +.. code:: + + you@localhost$ cd puppet-environment + you@localhost$ bin/deploy-on machine1 machine2... + +Note: `puppet-environment `_ + +Remember to pass ``--apt`` to ``bin/deploy-on`` if freshly uploaded Software Heritage +packages are to be deployed. Also, ``bin/deploy-on --help`` is your friend. + +.. _puppet_manual_deployment: + +Manual +~~~~~~ + +.. code:: + + # if a new or updated version debian package needs deploying + you@machine$ sudo apt-get update + # to test/review changes + you@machine$ sudo swh-puppet-test + # to apply + you@machine$ sudo swh-puppet-apply diff --git a/sysadm/puppet/howto-manage-third-party-modules.rst b/sysadm/puppet/howto-manage-third-party-modules.rst index 94ca6d1..3b90917 100644 --- a/sysadm/puppet/howto-manage-third-party-modules.rst +++ b/sysadm/puppet/howto-manage-third-party-modules.rst @@ -1,9 +1,91 @@ -.. _puppet_third_party: +.. _puppet_integration_of_third_party_puppet_modules: How to manage Third-Party modules ================================= -.. todo:: - This page is a work in progress. Please refer to the `existing documentation - `_ and `the - wiki page `_. +Integration of third party puppet modules +----------------------------------------- + +We mirror external repositories to our own forge, to avoid having external dependencies +in our deployment. + +In the *swh-site* ``Puppetfile``, we pin the installation of those modules to the +highest version (that works with our current puppet/facter version), by using the *:ref* +specifier. + +.. _adding_a_new_external_puppet_module: + +Adding a new external puppet module +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the *puppet-environment* repository, the ``bin/import-puppet-module`` takes care of +the following tasks: + +- Getting metadata from the `Puppet forge `_ for the + module (description, upstream git URL) +- Cloning the repository +- Creating a mirror repository on the Software Heritage forge, with the proper + permissions and metadata (notably the *Sync to GitHub* flag) +- Pushing the clone to the forge +- Updating the .mrconfig and .gitignore files + +To be able to use the script, you need: + +- Be a member of the `System Administrators + `_ Phabricator group +- Have the :ref:`Arcanist ` API key setup +- A pair of python dependencies: ``python3-phabricator`` and ``python3-requests`` (pull + them from testing if needed). + +Example usage to pull the `elastic/elasticsearch +`_ module + +:: + + bin/import-module elastic-elasticsearch + git diff # review changes + git add .mrconfig .gitignore + git commit -m "Add the elastic/elasticsearch module" + git push + +Once the module is added, you need to register it in the *swh-site* `Puppetfile +`_. + +You should also check in the module metadata whether any dependencies need importing as +well, which you should do using the same procedure. + +.. _updating_external_puppet_modules: + +Updating external puppet modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There's two sides of this coin: + +.. _updating_our_git_clone_of_external_puppet_modules: + +Updating our git clone of external puppet modules +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The *puppet-environment* ``.mrconfig`` file has a ``pullup`` command which does the +right thing. + +To update all clones: + +:: + + mr -j4 pullup + +.. _upgrading_external_puppet_modules: + +Upgrading external puppet modules +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Upgrading external puppet modules happens manually. + +In the *puppet-environment* repository, the ``bin/check-module-updates`` script compares +the Puppetfile and the local clones and lists the available updates. (depends on ``ruby +r10k``). + +On a staging branch of the *swh-site* repository, update the ``:ref`` value for the +module in the ``Puppetfile`` to the latest tag. You can then run ``octocatalog-diff`` on +a few relevant servers and look for changes. diff --git a/sysadm/puppet/howto-test-changes-in-vagrant.rst b/sysadm/puppet/howto-test-changes-in-vagrant.rst index 075d77f..cc0e258 100644 --- a/sysadm/puppet/howto-test-changes-in-vagrant.rst +++ b/sysadm/puppet/howto-test-changes-in-vagrant.rst @@ -1,8 +1,7 @@ -.. _puppet_vagrant: +.. _puppet_how_to_test_puppet_changes_in_vagrant: How to test Puppet changes in Vagrant ===================================== -.. todo:: - This page is a work in progress. Please refer to the `existing documentation - `_. +.. todo:: This page is a work in progress. Please refer to the `existing documentation + `_. diff --git a/sysadm/puppet/index.rst b/sysadm/puppet/index.rst index a8c6980..1e97b05 100644 --- a/sysadm/puppet/index.rst +++ b/sysadm/puppet/index.rst @@ -1,12 +1,13 @@ Puppet ====== .. toctree:: :titlesonly: + reference-setup tutorial-my-first-change-in-puppet howto-manage-third-party-modules howto-test-changes-in-vagrant howto-deploy-puppet-change reference-architecture reference-best-practices diff --git a/sysadm/puppet/reference-setup.rst b/sysadm/puppet/reference-setup.rst new file mode 100644 index 0000000..b095ba0 --- /dev/null +++ b/sysadm/puppet/reference-setup.rst @@ -0,0 +1,85 @@ +.. _puppet-setup: + +Puppet setup +============ + +.. admonition:: Intended audience + :class: important + + sysadm members + +.. _mutiple_repository_setup: + +Multiple repository setup +------------------------- + +Our puppet environment is split into multiple repos (one repo per module), plus one +"root" repository for multi-repository management. + +First, clone the base repository, containing the configuration file for myrepos and a +README file. + +:: + + $ git clone ssh://git@forge.softwareheritage.org/diffusion/SENV/puppet-environment.git + +Then, use that configuration to clone all the repositories: + +:: + + $ cd puppet-environment + $ readlink -f .mrconfig >> ~/.mrtrust + $ mr up + +(the *mr* command is in the `myrepos Debian package +`_). + +All the swh-specific repositories are in *swh-*-prefixed repositories. The other +repositories come from other sources and have an *upstream* remote allowing updates (the +*origin* remote is always on the swh git server). + +Our puppet workflow is documented in `the README.md file in the puppet-environment +repository +`_. + +.. _configure_octocatalog_diff: + +Configure octocatalog-diff to ease testing +------------------------------------------ + +*puppet-environment* contains the whole scaffolding to be able to use `octocatalog-diff +`_ on our manifests. This allows for +quick(er) local iterations while developing complex puppet manifests. + +Dependencies +~~~~~~~~~~~~ + +You need the following packages installed on your machine: + +- r10k octocatalog-diff +- puppet + +Running +~~~~~~~ + +The ``bin/octocatalog-diff`` script allows diffing the manifests between two +environments (that is, between two branches of the *swh-site* repository. By default it +diffs between ``production`` and ``staging``. + +Default usage: + +.. code:: + + cd puppet-environment + # Diff between branches "staging" and "production" for node "pergamon" + bin/octocatalog-diff pergamon + # Diff between branches "staging_feature" and "production" for node "worker01" + bin/octocatalog-diff --to staging_feature worker01 + +Limitations +~~~~~~~~~~~ + +Our setup for octocatalog-diff doesn't support exported resources, so you won't see your +fancy icinga checks there. For more evolved checks as those, use our `vagrant vms +definitions +`_. diff --git a/sysadm/puppet/tutorial-my-first-change-in-puppet.rst b/sysadm/puppet/tutorial-my-first-change-in-puppet.rst index cbe8d3e..c5cd73c 100644 --- a/sysadm/puppet/tutorial-my-first-change-in-puppet.rst +++ b/sysadm/puppet/tutorial-my-first-change-in-puppet.rst @@ -1,7 +1,83 @@ .. _puppet_tutorial: Tutorial: Making my first change in Puppet ========================================== -.. todo:: - This page is a work in progress. +.. admonition:: Intended audience + :class: important + + staff members + +.. _puppet_development: + +Development +~~~~~~~~~~~ + +The development happens in the swh-site repository. + + +So checkout the swh-site repository, and starts the development in the staging branches. +It's to diff against the production branch: + +.. code:: + + you@localhost$ cd swh-site && git pull + you@localhost$ git checkout production && git merge origin/production + # both staging and production should be in sync + you@localhost$ git checkout staging && git merge origin/staging + # you can now start hacking + you@localhost$ # *hack on puppet Git repo* + you@localhost$ rake validate + you@localhost$ git commit + +Test changes with octocatalog-diff +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As you developed your changes in the staging branch, you can now *diff* against the +production branch: + +.. code:: + + cd puppet-environment + # Diff between branches "staging" and "production" for node "pergamon" + bin/octocatalog-diff pergamon + # Diff between branches "staging_feature" and "production" for node "worker01" + bin/octocatalog-diff --to staging_feature worker01 + +This requires your :ref:`octocatalog setup to be ready `. + +Test changes in Vagrant +~~~~~~~~~~~~~~~~~~~~~~~ + +For more involved checks, we can test the changes using :ref:`vagrant +`. + +Ask for review +~~~~~~~~~~~~~~ + +As for standard development, ask for a review so someone can validate your changes. + +Please, during the diff phase, update the *test plan* paragraph with your +octocatalog-diff output. There can be more than one machines, so please give those +especially the one either impacted or not impacted by changes. + +.. code:: + + # (optional) ask for review + you@localhost$ arc diff... + +Land +~~~~ + +Once you are satisfied with your changes and they passed review, simply push the +changes. Merge both the production and staging branches together. + +.. code:: + + you@localhost$ cd swh-site + # you should develop on the staging branches first + you@localhost$ git checkout staging && git rebase production + # keep in sync the staging and production branches + you@localhost$ git checkout production && git merge staging + # land the changes + you@localhost$ git push origin staging production