diff --git a/README.md b/README.md index 22091ce..1fd6274 100644 --- a/README.md +++ b/README.md @@ -1,311 +1,345 @@ Software Heritage Puppet environment ==================================== This repository contains the metadata for Software Heritage's puppet infrastructure git repositories. The repositories are managed using [myrepos][1] (see the .mrconfig file), and the `mr` command. [1]: http://myrepos.branchable.com/ As our .mrconfig file contains "untrusted" checkout commands (to setup the upstream repositories of our mirrors of third-party modules), you need to add the .mrconfig file to your ~/.mrtrust file: readlink -f .mrconfig >> ~/.mrtrust You can then checkout the repositories using `mr up`. For periodic updates after initial setup, you can use the `bin/update` helper: cd puppet-environment bin/update Module Layout ------------- We use dynamic environments, and the role/profiles puppet workflow as demonstrated by the following series of articles: - http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/ - http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/ - http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/ - http://garylarizza.com/blog/2014/03/07/puppet-workflow-part-3b/ - http://garylarizza.com/blog/2014/10/24/puppet-workflows-4-using-hiera-in-anger/ Our main manifests are present in the `swh-site` repository. Each branch of that repository corresponds to an environment in the puppet workflow presented. This repository contains the Puppetfile referencing all the modules, the main manifest file `manifests/site.pp`, and the hiera `data` directory, as well as the two "site-modules" for `profile`s and `role`s. Our setup mirrors the git repositories of third-party Puppet modules on the Software Heritage git server --- this is to avoid reliance on 3rd party *hosting* services in order to be able to deploy. We add an upstream remote to the repositories through our mr configuration. Deployment ---------- Deployment happens on the `pergamon.softwareheritage.org` server, using our custom deployment script: sudo /etc/puppet/environments/production/deploy.sh This updates the dynamic environments according to the contents of the branches of the git repository, and the Puppetfile inside. For each third-party module, we pin the module definition in the Puppetfile to a specific tag or revision. Our specific deploy script also fetches private repositories and merges them with the public r10k setup. Local puppet manifest diffing with octocatalog-diff --------------------------------------------------- puppet-environment contains the whole scaffolding to be able to use [octocatalog-diff][2] on our manifests. This allows for quick(er) local iterations while developing complex puppet manifests. Dependencies [2]: https://github.com/github/octocatalog-diff You need the following packages installed on your machine: r10k octocatalog-diff puppet ### Running The `bin/octocatalog-diff` script allows diffing the manifests between two environments (that is, between two branches of the *swh-site* repository. By default it diffs between production and staging. Default usage: bin/octocatalog-diff pergamon Limitations Our setup for octocatalog-diff doesn't support exported resources, so you won't see your fancy icinga checks there. Integration of third party puppet modules ----------------------------------------- We mirror external repositories to our own forge, to avoid having external dependencies in our deployment. In the `swh-site/Puppetfile`, we pin the installation of those modules to the highest version (that works with our current puppet/facter version), by using the `:ref` specifier. ### Adding a new external puppet module In the *puppet-environment* repository, the `bin/import-puppet-module` takes care of the following tasks: * Getting metadata from the Puppet forge for the module (description, upstream git URL) * Cloning the repository * Creating a mirror repository on the Software Heritage forge, with the proper permissions and metadata (notably the Sync to GitHub flag) * Pushing the clone to the forge * Updating the `.mrconfig` and `.gitignore` files to know the new repository To be able to use the script, you need to : * Be a member of the System Administrators Phabricator group * Have the Arcanist API key setup * A pair of python dependencies : `python3-phabricator` and `python3-requests` (pull them from testing if needed). Example usage to pull the `elastic/elasticsearch` module bin/import-module elastic-elasticsearch git diff # review changes git add .mrconfig .gitignore git commit -m "Add the elastic/elasticsearch module" git push Once the module is added, you need to register it in `swh-site/Puppetfile`. You should also check in the module metadata whether any dependencies need importing as well, which you should do using the same procedure. ### Updating external puppet modules There's two sides of this coin: #### Updating our git clone of external puppet modules The *puppet-environment* `.mrconfig` file has a pullup command which does the right thing. To update all clones: mr -j4 pullup #### Upgrading external puppet modules Upgrading external puppet modules happens manually. In the *puppet-environment* repository, the `bin/check-module-updates` script compares the Puppetfile and the local clones and lists the available updates. (depends on `ruby r10k`). On a staging branch of the *swh-site* repository, update the `:ref` value for the module in the `Puppetfile` to the latest tag. You can then run `octocatalog-diff` on a few relevant servers and look for changes. Deploy workflow ---------------- ### Semi-automated you@localhost$ # hack on puppet Git repo you@localhost$ rake validate you@localhost$ git commit you@localhost$ git push you@localhost$ cd puppet-environment you@localhost$ bin/deploy-on machine1 machine2... Remember to pass `--apt` to `bin/deploy-on` if freshly uploaded Software Heritage packages are to be deployed. Also, `bin/deploy-on --help` is your friend. ### Manual you@localhost$ # hack on puppet Git repo you@localhost$ rake validate you@localhost$ git commit you@localhost$ git push you@pergamon$ sudo swh-puppet-master-deploy you@machine$ sudo apt-get update # if a new or updated version of a Debian package needs deploying you@machine$ sudo swh-puppet-test # to test/review changes you@machine$ sudo swh-puppet-apply # to apply Local tests with vagrant ------------------------ > **_NOTE_**: The vagrant configuration uses a public image generated with packer. See the dedicated readme[1] in the packer directory for more information to manage this image. ### Setup -Vagrant and Virtualbox tools must be installed. On a debian based environment, + +Vagrant tools must be installed. + +``` +apt install vagrant nfs-kernel-server +``` + +Note: `nfs-kernel-server` is needed to export and share the local /tmp/puppet + to the vm + +Multiple provisioners exist. We will focus on 2 for now, virtualbox the default +and libvirt for now (we had an history of using this one in the past). + +#### virtualbox + +On a debian based environment, [a specific debian repository must be configured](https://www.virtualbox.org/wiki/Linux_Downloads): ``` -apt install vagrant virtualbox-6.0 nfs-kernel-server linux-headers-$(uname --kernel-release) +apt install virtualbox-6.0 linux-headers-$(uname --kernel-release) ``` Note: - 2020-09-17 vagrant (buster) is not working with virtualbox 6.1, so we use `virtualbox-6.0` -- `nfs-kernel-server` is needed to export and share the local /tmp/puppet to the - vm - `linux-headers` package is required for the vbox guest additions +#### libvirt + +Already part of debian so: + +``` +apt install libvirt-daemon-system qemu-kvm vagrant=2.2.3+dfsg-1 vagrant-libvirt +``` + +Note: +- Use the vagrant packaged packaged in debian stable (not hashicorp's) +- Add your user to the libvirt group. + +``` +sudo usermod --append --groups libvirt `whoami` +``` + +``` +usermod -a -G libvirt $USER +``` + ### Usage #### Prepare the puppet environment The puppet directory structure needs to be prepared before starting a vm. It can be done with the ``bin/prepare-vagrant-conf`` script. The script must be run each time a new commit is done to refresh the code applied on the vms. The working directory is ``/tmp/puppet``. ``` bin/prepare-vagrant-conf [-b branch] ``` The ``-b`` parameter allows to create a specific puppet environment based on the branch with the same name. It allows to test changes on feature branches (The ``environment`` variable in the Vagrantfile has to be updated accordingly). **_NOTE_**: This command only uses the **committed files**. The pending changes will not be included on the configuration. (**to be confirmed**) By convention, the vagrant names are prefixed by the environment for the core archive servers: * staging-webapp0 * staging-worker0 * staging-db0 * production-worker0 * production-db0 * ... #### Status The status of all the vms present on the vagranfile can be checked with: ``` vagrant status ``` Example: ``` $ vagrant status Current machine states: staging-webapp running (virtualbox) staging-worker0 running (virtualbox) +staging-deposit not created (libvirt) prod-worker01 not created (virtualbox) test poweroff (virtualbox) This environment represents multiple VMs. The VMs are all listed above with their current state. For more information about a specific VM, run `vagrant status NAME`. ``` #### Start a vm ``` vagrant up ``` For example to start the worker0 of the staging: ``` $ # update the configuration $ bin/prepare-vagrant-conf $ # start the vm $ vagrant up ``` If it's the first time this vm is launched, vagrant will apply the puppet configuration to init the server. #### Apply the puppet configuration To speedup the tests, a scripts can be used to synchronize the changes made on the working directories and the puppet configuration directories used by vagrant. This script avoid to have to commit each change before being able to test it. In one terminal: ``` bin/prepare-vagrant-conf bin/watch-vagrant-conf ``` In another terminal: ``` vagrant provision ``` In this case, the configuration will always be up-to-date with the local directories. > **_NOTE_**: It works for basic changes on the swh-site and data configurations. For other changes like declaring a new puppet module, the ``prepare-vagrant-conf`` must be called to completely rebuild the configuration. > **_NOTE_**: The ``watch-vagrant-conf`` script uses inotify and especially the ``inotifywait`` command to detect the changes. The package ``inotify-tools`` needs to be installed on the local environment. #### connect to a vm A shell can be opened to a running vm with this command: ``` vagrant ssh ``` #### Stop a vm ``` vagrant stop ``` #### Update the configuration If the vagrantfile is updated to change a property of a server, like the memory, cpu configuration or network, the configuration has to be reloaded: ``` vagrant reload ``` #### Cleanup To recover space, the vms can be destroyed with: ``` vagrant destroy ``` [1]: [packer/README.md][packer/README.md] diff --git a/Vagrantfile b/Vagrantfile index 6534d2a..0c5361b 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -1,226 +1,272 @@ Vagrant.require_version ">= 2.2.0" ENV["LC_ALL"] = "en_US.UTF-8" # Default configuration for all defines node below environment = "staging" environment_path = "/tmp/puppet/environments" manifest_file = "site.pp" manifests_path = "swh-site/manifests" puppet_options = "--fileserverconfig=/etc/puppet/fileserver.conf --verbose" # --debug --trace for more puppet_default_facts = { "vagrant_testing" => "1", "testing" => "vagrant", "deployment" => "staging", "subnet" => "vagrant" } +# used to define the local vm template path +puppet_env_path=ENV["SWH_PUPPET_ENVIRONMENT_HOME"] # Images configuration $global_debian10_box = "debian10-20201006-0832" $global_debian10_box_url = "https://annex.softwareheritage.org/public/isos/virtualbox/debian/swh-debian-10.6-amd64-20201006-0832.box" # For local tests #$global_debian10_box_url = "file:///path/to/packer/builds/swh-debian-10.6-amd64-20201006-0832.box" +# local configuration +$local_debian10_qcow2 = "debian10-20201004-1105" +$local_debian10_qcow2_url = "file://#{puppet_env_path}/packer/builds/swh-debian-10.6-amd64-20201004-1105.qcow2" Vagrant.configure("2") do |global_config| ################ ## STAGING ################ global_config.vm.define :"staging-webapp" do |config| # config.ssh.insert_key = false config.vm.box = $global_debian10_box config.vm.box_url = $global_debian10_box_url config.vm.box_check_update = false config.vm.hostname = "webapp.internal.staging.swh.network" config.vm.network :private_network, ip: "10.168.128.8", netmask: "255.255.255.0" config.vm.synced_folder "/tmp/puppet/", "/tmp/puppet", type: 'nfs' # ssl certificates share config.vm.synced_folder "vagrant/le_certs", "/etc/puppet/le_certs", type: 'nfs' config.vm.provider "virtualbox" do |vb| vb.name = "staging-webapp" vb.gui = false vb.check_guest_additions = false vb.linked_clone = true vb.memory = 512 vb.cpus = 2 end config.vm.provision "puppet" do |puppet| puppet.environment_path = "#{environment_path}" puppet.environment = "#{environment}" puppet.hiera_config_path = "#{puppet.environment_path}/#{puppet.environment}/hiera.yaml" puppet.manifest_file = "#{manifest_file}" puppet.manifests_path = "#{manifests_path}" puppet.options = "#{puppet_options}" puppet.facter = puppet_default_facts end end global_config.vm.define :"staging-deposit" do |config| config.vm.box = $global_debian10_box config.vm.box_url = $global_debian10_box_url config.vm.box_check_update = false config.vm.hostname = "deposit.internal.staging.swh.network" config.vm.network :private_network, ip: "10.168.128.9", netmask: "255.255.255.0" config.vm.synced_folder "/tmp/puppet/", "/tmp/puppet", type: 'nfs' config.vm.provider "virtualbox" do |vb| vb.name = "staging-deposit" vb.gui = false vb.check_guest_additions = false vb.linked_clone = true vb.memory = 512 vb.cpus = 2 end config.vm.provision "puppet" do |puppet| puppet.environment_path = "#{environment_path}" puppet.environment = "#{environment}" puppet.hiera_config_path = "#{puppet.environment_path}/#{puppet.environment}/hiera.yaml" puppet.manifest_file = "#{manifest_file}" puppet.manifests_path = "#{manifests_path}" puppet.options = "#{puppet_options}" puppet.facter = puppet_default_facts end end global_config.vm.define :"staging-worker0" do |config| config.vm.box = $global_debian10_box config.vm.box_url = $global_debian10_box_url config.vm.hostname = "worker0.staging.swh.network" config.vm.network :private_network, ip: "10.168.128.5", netmask: "255.255.255.0" config.vm.synced_folder "/tmp/puppet/", "/tmp/puppet", type: 'nfs' # ssl certificates share config.vm.synced_folder "vagrant/le_certs", "/etc/puppet/le_certs", type: 'nfs' config.vm.provider "virtualbox" do |vb| vb.name = "staging-worker0" vb.gui = false vb.check_guest_additions = false vb.linked_clone = true vb.memory = 4096 vb.cpus = 2 end config.vm.provision "puppet" do |puppet| puppet.environment_path = "#{environment_path}" puppet.environment = "#{environment}" puppet.hiera_config_path = "#{puppet.environment_path}/#{puppet.environment}/hiera.yaml" puppet.manifest_file = "#{manifest_file}" puppet.manifests_path = "#{manifests_path}" puppet.options = "#{puppet_options}" puppet.facter = puppet_default_facts end end ################ # ADMIN ################ global_config.vm.define :"bojimans" do |config| # config.ssh.insert_key = false config.vm.box = $global_debian10_box config.vm.box_url = $global_debian10_box_url config.vm.box_check_update = false config.vm.hostname = "bojimans.internal.softwareheritage.org" config.vm.network :private_network, ip: "10.168.100.199", netmask: "255.255.255.0" config.vm.synced_folder "/tmp/puppet/", "/tmp/puppet", type: 'nfs' # ssl certificates share config.vm.synced_folder "vagrant/le_certs", "/etc/puppet/le_certs", type: 'nfs' config.vm.provider "virtualbox" do |vb| vb.name = "bojimans" vb.gui = false vb.check_guest_additions = false vb.linked_clone = true vb.memory = 512 vb.cpus = 2 end config.vm.provision "puppet" do |puppet| puppet.environment_path = "#{environment_path}" puppet.environment = "#{environment}" puppet.hiera_config_path = "#{puppet.environment_path}/#{puppet.environment}/hiera.yaml" puppet.manifest_file = "#{manifest_file}" puppet.manifests_path = "#{manifests_path}" puppet.options = "#{puppet_options}" puppet.facter = puppet_default_facts end end ################ ## PRODUCTION ################ global_config.vm.define :"prod-worker01" do |config| config.vm.box = $global_debian10_box config.vm.box_url = $global_debian10_box_url config.vm.hostname = "worker01.softwareheritage.org" config.vm.network :private_network, ip: "10.168.100.21", netmask: "255.255.255.0" config.vm.synced_folder "/tmp/puppet/", "/tmp/puppet", type: 'nfs' # ssl certificates share config.vm.synced_folder "vagrant/le_certs", "/etc/puppet/le_certs", type: 'nfs' config.vm.provider "virtualbox" do |vb| vb.name = "worker01" vb.gui = false vb.check_guest_additions = false vb.linked_clone = true vb.memory = 4096 vb.cpus = 2 end config.vm.provision "puppet" do |puppet| puppet.environment_path = "#{environment_path}" puppet.environment = "#{environment}" puppet.hiera_config_path = "#{puppet.environment_path}/#{puppet.environment}/hiera.yaml" puppet.manifest_file = "#{manifest_file}" puppet.manifests_path = "#{manifests_path}" puppet.options = "#{puppet_options}" puppet.facter = puppet_default_facts end end ################ ## MISC ################ global_config.vm.define :test do |config| config.ssh.insert_key = false config.vm.box = $global_debian10_box config.vm.box_url = $global_debian10_box_url config.vm.box_check_update = false config.vm.hostname = "test.softwareheritage.org" config.vm.network :private_network, ip: "10.168.100.30", netmask: "255.255.255.0" config.vm.network :private_network, ip: "10.168.101.30", netmask: "255.255.255.0" config.vm.network "forwarded_port", guest: 10030, host: 22 config.vm.synced_folder "/tmp/puppet/", "/tmp/puppet", type: 'nfs' # ssl certificates share config.vm.synced_folder "vagrant/le_certs", "/etc/puppet/le_certs", type: 'nfs' config.vm.provider "virtualbox" do |vb| vb.name = "test" vb.gui = false vb.check_guest_additions = false vb.linked_clone = true vb.memory = 512 vb.cpus = 2 end config.vm.provision "puppet" do |puppet| puppet.environment_path = "#{environment_path}" puppet.environment = "#{environment}" puppet.hiera_config_path = "#{puppet.environment_path}/#{puppet.environment}/hiera.yaml" puppet.manifest_file = "#{manifest_file}" puppet.manifests_path = "#{manifests_path}" puppet.options = "#{puppet_options}" puppet.facter = puppet_default_facts end end + + unless Vagrant.has_plugin?("libvirt") + $stderr.puts <<-MSG + vagrant-libvirt plugin is required for this. + To install: `$ sudo apt install vagrant-libvirt +MSG + exit 1 + end + + global_config.vm.define :qemutest do |config| + config.ssh.insert_key = false + + config.vm.box = $local_debian10_qcow2 + config.vm.box_url = $local_debian10_qcow2_url + config.vm.hostname = "test.softwareheritage.org" + config.vm.box_check_update = false + config.vm.network :private_network, ip: "10.168.98.30", netmask: "255.255.255.0" + config.vm.network :private_network, ip: "10.168.99.30", netmask: "255.255.255.0" + config.vm.network "forwarded_port", guest: 10030, host: 22 + + config.vm.synced_folder "/tmp/puppet/", "/tmp/puppet", type: 'nfs' + # ssl certificates share + config.vm.synced_folder "vagrant/le_certs", "/etc/puppet/le_certs", type: 'nfs' + + config.vm.provider :libvirt do |provider| + provider.memory = 512 + provider.cpus = 2 + # local test run: https://github.com/vagrant-libvirt/vagrant-libvirt/issues/45 + provider.driver = 'kvm' + end + + config.vm.provision "puppet" do |puppet| + puppet.environment_path = "#{environment_path}" + puppet.environment = "#{environment}" + puppet.hiera_config_path = "#{puppet.environment_path}/#{puppet.environment}/hiera.yaml" + puppet.manifest_file = "#{manifest_file}" + puppet.manifests_path = "#{manifests_path}" + puppet.options = "#{puppet_options}" + puppet.facter = puppet_default_facts + end + end end diff --git a/packer/scripts/post-install.sh b/packer/scripts/post-install.sh index 047e56c..96461dc 100644 --- a/packer/scripts/post-install.sh +++ b/packer/scripts/post-install.sh @@ -1,71 +1,71 @@ #!/bin/bash -eu #### # apt configuration #### cat </etc/apt/sources.list.d/debian.list deb http://deb.debian.org/debian buster main deb-src http://deb.debian.org/debian buster main deb http://deb.debian.org/debian-security/ buster/updates main deb-src http://deb.debian.org/debian-security/ buster/updates main deb http://deb.debian.org/debian buster-updates main deb-src http://deb.debian.org/debian buster-updates main EOF export DEBIAN_FRONTEND=noninteractive apt-get update -apt-get install -y man wget curl telnet net-tools dnsutils traceroute unbound +apt-get install -y man wget curl telnet net-tools dnsutils traceroute unbound gpg #### # allow vagrant user to sudo to root without password #### cat </etc/sudoers.d/vagrant vagrant ALL= NOPASSWD: ALL EOF chmod 0440 /etc/sudoers.d/vagrant #### # Configure authorized_keys to allow vagrant command line to interact with the VM #### mkdir -m 700 /home/vagrant/.ssh curl -s https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant.pub >> /home/vagrant/.ssh/authorized_keys chmod 0600 /home/vagrant/.ssh/authorized_keys chown -R vagrant:vagrant /home/vagrant/.ssh #### # Puppet #### apt-get install -y puppet # Configure the fileserver for the "le_certs" share # In the vm, this directory is a mount of the vagrant/le_certs directory configured on the vagrantfile cat > /etc/puppet/fileserver.conf < /etc/network/interfaces <