puppet-environment contains the whole scaffolding to be able to use [octocatalog-diff][2] on our manifests. This allows for quick(er) local iterations while developing complex puppet manifests.
Dependencies
[2]: https://github.com/github/octocatalog-diff
You need the following packages installed on your machine:
r10k octocatalog-diff puppet
### Running
The `bin/octocatalog-diff` script allows diffing the manifests between two environments (that is, between two branches of the *swh-site* repository. By default it diffs between production and staging.
Default usage:
bin/octocatalog-diff pergamon
Limitations
Our setup for octocatalog-diff doesn't support exported resources, so you won't see your fancy icinga checks there.
Integration of third party puppet modules
-----------------------------------------
We mirror external repositories to our own forge, to avoid having external dependencies in our deployment.
In the `swh-site/Puppetfile`, we pin the installation of those modules to the highest version (that works with our current puppet/facter version), by using the `:ref` specifier.
### Adding a new external puppet module
In the *puppet-environment* repository, the `bin/import-puppet-module` takes care of the following tasks:
* Getting metadata from the Puppet forge for the module (description, upstream git URL)
* Cloning the repository
* Creating a mirror repository on the Software Heritage forge, with the proper permissions and metadata (notably the Sync to GitHub flag)
* Pushing the clone to the forge
* Updating the `.mrconfig` and `.gitignore` files to know the new repository
To be able to use the script, you need to :
* Be a member of the System Administrators Phabricator group
* Have the Arcanist API key setup
* A pair of python dependencies : `python3-phabricator` and `python3-requests` (pull them from testing if needed).
Example usage to pull the `elastic/elasticsearch` module
bin/import-module elastic-elasticsearch
git diff # review changes
git add .mrconfig .gitignore
git commit -m "Add the elastic/elasticsearch module"
git push
Once the module is added, you need to register it in `swh-site/Puppetfile`.
You should also check in the module metadata whether any dependencies need importing as well, which you should do using the same procedure.
### Updating external puppet modules
There's two sides of this coin:
#### Updating our git clone of external puppet modules
The *puppet-environment* `.mrconfig` file has a pullup command which does the right thing.
In the *puppet-environment* repository, the `bin/check-module-updates` script compares the Puppetfile and the local clones and lists the available updates. (depends on `ruby r10k`).
On a staging branch of the *swh-site* repository, update the `:ref` value for the module in the `Puppetfile` to the latest tag. You can then run `octocatalog-diff` on a few relevant servers and look for changes.
Deploy workflow
----------------
### Semi-automated
you@localhost$ # hack on puppet Git repo
you@localhost$ rake validate
you@localhost$ git commit
you@localhost$ git push
you@localhost$ cd puppet-environment
you@localhost$ bin/deploy-on machine1 machine2...
Remember to pass `--apt` to `bin/deploy-on` if freshly uploaded Software Heritage packages are to be deployed. Also, `bin/deploy-on --help` is your friend.
### Manual
you@localhost$ # hack on puppet Git repo
you@localhost$ rake validate
you@localhost$ git commit
you@localhost$ git push
you@pergamon$ sudo swh-puppet-master-deploy
you@machine$ sudo apt-get update # if a new or updated version of a Debian package needs deploying
you@machine$ sudo swh-puppet-test # to test/review changes
you@machine$ sudo swh-puppet-apply # to apply
Local tests with vagrant
------------------------
> **_NOTE_**: The vagrant configuration uses a public image generated with packer.
See the dedicated readme[1] in the packer directory for more information to manage this image.
### Setup
Vagrant and Virtualbox tools must be installed. On a debian based environment,
[a specific debian repository must be configured](https://www.virtualbox.org/wiki/Linux_Downloads):
- 2020-09-17 vagrant (buster) is not working with virtualbox 6.1, so we use
`virtualbox-6.0`
- `nfs-kernel-server` is needed to export and share the local /tmp/puppet to the
vm
- `linux-headers` package is required for the vbox guest additions
-- vagrant-vbguest 0.25.0 makes the first "vagrant up <vm>" fail
### Usage
#### Prepare the puppet environment
The puppet directory structure needs to be prepared before starting a vm.
It can be done with the ``bin/prepare-vagrant-conf`` script. The script must be run each time a new commit is done to refresh the code applied on the vms.
The working directory is ``/tmp/puppet``.
```
bin/prepare-vagrant-conf [-b branch]
```
The ``-b`` parameter allows to create a specific puppet environment based on the branch with the same name.
It allows to test changes on feature branches (The ``environment`` variable in the Vagrantfile has to be updated accordingly).
**_NOTE_**: This command only uses the **committed files**. The pending changes will not be included on the configuration.
(**to be confirmed**) By convention, the vagrant names are prefixed by the environment for the core archive servers:
* staging-webapp0
* staging-worker0
* staging-db0
* production-worker0
* production-db0
* ...
#### Status
The status of all the vms present on the vagranfile can be checked with:
```
vagrant status
```
Example:
```
$ vagrant status
Current machine states:
staging-webapp running (virtualbox)
staging-worker0 running (virtualbox)
prod-worker01 not created (virtualbox)
test poweroff (virtualbox)
This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
```
#### Start a vm
```
vagrant up <server name>
```
For example to start the worker0 of the staging:
```
$ # update the configuration
$ bin/prepare-vagrant-conf
$ # start the vm
$ vagrant up <vm name>
```
If it's the first time this vm is launched, vagrant will apply the puppet configuration to init the server.
#### Apply the puppet configuration
To speedup the tests, a scripts can be used to synchronize the changes made on the working directories and the puppet configuration directories used by vagrant. This script avoid to have to commit each change before being able to test it.
In one terminal:
```
bin/prepare-vagrant-conf
bin/watch-vagrant-conf
```
In another terminal:
```
vagrant provision <vm-name>
```
In this case, the configuration will always be up-to-date with the local directories.
> **_NOTE_**: It works for basic changes on the swh-site and data configurations. For other changes like declaring a new puppet module, the ``prepare-vagrant-conf`` must be called to completely rebuild the configuration.
> **_NOTE_**: The ``watch-vagrant-conf`` script uses inotify and especially the ``inotifywait`` command to detect the changes. The package ``inotify-tools`` needs to be installed on the local environment.
#### connect to a vm
A shell can be opened to a running vm with this command:
```
vagrant ssh <vm name>
```
#### Stop a vm
```
vagrant stop <vm name>
```
#### Update the configuration
If the vagrantfile is updated to change a property of a server, like the memory, cpu configuration or network, the configuration has to be reloaded: