• The configuration was applied on moma
• a manual import was performed on worker01 :
  • the /etc/softwareheritage/loader_git.yaml config was updated:

root@worker01:/etc/softwareheritage# diff -U3 /tmp/loader_git.yml loader_git.yml 
--- /tmp/loader_git.yml	2020-11-20 08:43:18.682462213 +0000
+++ loader_git.yml	2020-11-20 08:44:00.150375756 +0000
@@ -13,7 +13,7 @@
   - cls: filter
   - cls: remote
     args:
-      url: http://uffizi.internal.softwareheritage.org:5002/
+      url: http://saam.internal.softwareheritage.org:5002/
 max_content_size: 104857600
 save_data: false
 save_data_path: "/srv/storage/space/data/sharded_packfiles"

• the import was run on the puppet-swh-site repository:

root@worker01:/etc/softwareheritage# sudo -u swhworker SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_git.yml swh loader run git https://github.com/SoftwareHeritage/puppet-swh-site

The first try returns this exception :

swh.core.api.RemoteException: <RemoteException 500 ValueError: ["Storage class azure-prefixed is not available: No module named 'swh.objstorage.backends.azure'"]>

rebase

rebase

In D4534#113059, @olasd wrote:

We use the puppet java module in a bunch of other places, maybe it makes sense to directly import that (which would mean using include ::java)?

Use ::java instead of directly install the jre package

LGTM

I will not land this now, it seems there is another issue with the startup of kafka when the logdir is already existing but empty (i.e. created by puppet). I need to dig further

The systemd configuration looks good.

Looks fine to me except for the vagrant bit.

thanks, it's fixed

fix misdirected removal

up -d is waiting the containers are "started" before returning the hand so you are sure the execs on line 169- can be executed.
You will also miss the return code of the docker-compose up command

Rectification : kafka is installed on the node but it seems the configuration is not complete

The varnish logs should be also ingested to elasticsearch to have fine grained statistics.

• adapt the configuration to be able to test locally without interference with the other environments :

The /etc/hosts files of the vagrant vms are configured to declare local ips for the service they are using [1] . It's not a strong security but it works for the moment.
A strongest security will be put in place when the admin servers will be moved to the admin network, the network could be filtered to ensure such local vms can't interact with real production servers

The network configuration is done and the staging archive and deposit are now exposed publicly. The principal goal of the task is achieve.
The staging VMs could be moved to their dedicated hypervisor when it will be available, finally it's not a mandatory step for this task as we were able to use the existing hypervisors.

The metric are well ingested by prometheus and the hosts availability is checked by icinga.
A basic dashboard was created in grafana[1] with the following information for both firewall :

• uptime
• load
• memory stats
• partitions stats
• network traffic for each interface

LGTM, it works in vagrant with the firewalls configuration :

==> pergamon: Notice: /Stage[main]/Profile::Icinga2::Master/File[/etc/icinga2/zones.d/master/pushkin.internal.softwareheritage.org.conf]/ensure: removed
==> pergamon: Info: /etc/icinga2/zones.d/master: Scheduling refresh of Class[Icinga2::Service]

I have created the diff for information but will land it quickly to fix the prometheus configuration ASAP.

fix formating

replace the lost lookup by an alias

rebase and remove unnecessary spaces

use https

It fixes problems to reach the public ip from the internal network.
Feel free to land it if it looks good to you

• Add webui checks on icinga
• Rename the puppet class to something more generic as it's not only dedicated to prometheus configuration

rebase

This is a schema in complement of the previous ones. It represent a more network oriented interaction between the server and the firewall :

After double(at least) checking the routed on louvre is working well (the packets are not intercepted by the ip masquerade).
The problem was the DNAT rule on the firewall was not applied because the packets are not entering from the vtnet0 interface (they were simply lost). The DNAT rule was updated to be applied on the vtnet1 (VLAN440) and vtnet0 (VLAN1300) interfaces[1]. Pergamon can now reach the reverse proxy on ports 80/443

To solve the monitoring alerts [1], we tried to bypass the restriction between the VLAN210 and the VLAN1300 by adding a route between pergamon and VLAN1300 via the firewall (D4454).
The route is well created on pergamon but it seems to be ignored :

root@pergamon:~# traceroute 128.93.166.2
traceroute to 128.93.166.2 (128.93.166.2), 30 hops max, 60 byte packets
 1  louvre.internal.softwareheritage.org (192.168.100.1)  0.185 ms * *

It's the same for other routes :

root@pergamon:~# traceroute 192.168.130.10
traceroute to 192.168.130.10 (192.168.130.10), 30 hops max, 60 byte packets
 1  louvre.internal.softwareheritage.org (192.168.100.1)  0.168 ms * *
 2  pushkin.internal.softwareheritage.org (192.168.100.129)  0.331 ms  0.316 ms  0.307 ms
 3  pushkin.internal.softwareheritage.org (192.168.100.129)  0.426 ms  0.414 ms  0.400 ms

Fix indentation

A step was achieve in the configuration. The staging services are now accessible from the internet from these addresses :

• webapp : https://webapp.staging.swh.network
• deposit: https://deposit.staging.swh.network

we don't need it because pergamon is not managing the first level of swh.network and declaring such entries avoid puppet to test and update the dns configuration as your paste P862 shows it.

LGTM