diff --git a/docs/images/network.png b/docs/images/network.png deleted file mode 100644 index 0000000000000000000000000000000000000000..0000000000000000000000000000000000000000 GIT binary patch literal 0 Hc$@`_ VMs deployed on the PROXMOX cluster with an `High Availability `_ configuration. - -They are sharing a virtual IP on each VLAN to act as the gateway. Only one of the 2 firewalls is owning all the GW ips at the same time. The owner is called the ``PRIMARY`` - -.. list-table:: - :header-rows: 1 - - * - Nominal Role - - name (link to the inventory) - - login page - * - PRIMARY - - `pushkin `_ - - `https://pushkin.internal.softwareheritage.org `_ - * - BACKUP - - `glyptotek `_ - - `https://glyptotek.internal.softwareheritage.org `_ - - -Access to the gui of the secondary firewall ----------------------------------------------- - -The secondary firewall is not directly reachable for VPN user. -As the OpenVPN service is also running when the firewall is a backup, the packets -coming from tne VPN are routed to the local VPN on the secondary and lost. - -To access to GUI, a tunnel can be used: - - ssh -L 8443:pushkin.internal.softwareheritage.org:443 pergamon.internal.softwareheritage.org - -Once the tunnel is created, the gui is accessible at https://localhost:8443 in any browser - -Configuration backup --------------------- - -The configuration is automatically committed on a `git repository `_. -Each firewall regularly pushes its configuration on a dedicated branch of the repository. - -The configuration is visible on the `System / Configuration / Backups `_ page -of each one. - -Upgrade procedure ------------------ - -Initial status -^^^^^^^^^^^^^^ - -This is the nominal status of the firewalls: - -.. list-table:: - :header-rows: 1 - - * - Firewall - - Status - * - pushkin - - PRIMARY - * - glyptotek - - BACKUP - -Preparation -^^^^^^^^^^^ - -* Connect to the `principal `_ (pushkin here) -* Check the `CARP status `_ to ensure the firewall is the principal (must have the status MASTER for all the IPS) -* Connect to the `backup `_ (glytotek here) -* Check the `CARP status `__ to ensure the firewall is the backup (must have the status BACKUP for all the IPS) -* Ensure the 2 firewalls are in sync: - - * On the principal, go to the `High availability status `_ and force a synchronization - * click on the button on the right of ``Synchronize config to backup`` - -.. image:: ../images/infrastructure/network/sync.png - -* Switch the principal/backup to prepare the upgrade of the master - (The switch is transparent from the user perspective and can be done without service interruption) - - * [1] On the principal, go to the `Virtual IPS status `_ page - * Activate the CARP maintenance mode - - .. image:: ../images/infrastructure/network/carp_maintenance.png - - * check the status of the VIPs, they must be ``BACKUP`` on pushkin and ``PRIMARY`` on glyptotek - - -* wait a few minutes to let the monitoring detect if there are connection issues, check ssh connection on several servers on different VLANs (staging, admin, ...) - -If everything is ok, proceed to the next section. - -Upgrade the first firewall -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Before starting this section, the firewall statuses should be: - -.. list-table:: - :header-rows: 1 - - * - Firewall - - Status - * - pushkin - - BACKUP - * - glyptotek - - PRIMARY - -If not, be sure of what you are doing and adapt the links accordingly - -* [2] go to the `System Firmware: status `_ page (pushkin here) -* Click on the ``Check for upgrades`` button - -.. image:: ../images/infrastructure/network/check_for_upgrade.png - -* follow the interface indication, one or several reboots can be necessary depending to the number of upgrade to apply - -.. image:: ../images/infrastructure/network/proceed_update.png - -* repeat from the ``Check for upgrades`` operation until there is no upgrades to apply -* Switch the principal/backup to restore ``pushkin`` as the principal: - - * on the current backup (pushkin here) go to `Virtual IPS status `_ - * [3] click on `Leave Persistent CARP Maintenance Mode` - - .. image:: ../images/infrastructure/network/reactivate_carp.png - - * refresh the page, the role should have changed from ``BACKUP`` to ``MASTER`` - * check on the other firewall, if the roles is indeed ``BACKUP`` for all the IPs - -* Wait few moment to ensure everything is ok with the new version - -Upgrade the second firewall -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Before starting this section, the firewall statuses should be: - -.. list-table:: - :header-rows: 1 - - * - Firewall - - Status - * - pushkin - - PRIMARY - * - glyptotek - - BACKUP - -If not, be sure of what you are doing and adapt the links accordingly - -* Proceed to the second firewall upgrade - - * perform [1] on the backup (should be ``glyptotek`` here) - * perform [2] on the backup (should be ``glyptotek`` here) - * perform [3] on the backup (should be ``glyptotek`` here) diff --git a/swh/docs/sphinx/conf.py b/swh/docs/sphinx/conf.py --- a/swh/docs/sphinx/conf.py +++ b/swh/docs/sphinx/conf.py @@ -151,6 +151,9 @@ "swh-deposit/metadata": "api/metadata.html", "swh-deposit/specs/blueprint": "../api/use-cases.html", "swh-deposit/user-manual": "api/user-manual.html", + "infrastructure/index.html": "sysadm/network-architecture/index.html", + "infrastructure/network.html": "sysadm/network-architecture/index.html", + "infrastructure/service-urls.html": "sysadm/network-architecture/service-urls.html", "architecture": "architecture/overview.html", "mirror": "architecture/mirror.html", "users": "user", diff --git a/docs/images/infrastructure/network/carp_maintenance.png b/sysadm/images/infrastructure/network/carp_maintenance.png rename from docs/images/infrastructure/network/carp_maintenance.png rename to sysadm/images/infrastructure/network/carp_maintenance.png diff --git a/docs/images/infrastructure/network/check_for_upgrade.png b/sysadm/images/infrastructure/network/check_for_upgrade.png rename from docs/images/infrastructure/network/check_for_upgrade.png rename to sysadm/images/infrastructure/network/check_for_upgrade.png diff --git a/docs/images/infrastructure/network/proceed_update.png b/sysadm/images/infrastructure/network/proceed_update.png rename from docs/images/infrastructure/network/proceed_update.png rename to sysadm/images/infrastructure/network/proceed_update.png diff --git a/docs/images/infrastructure/network/reactivate_carp.png b/sysadm/images/infrastructure/network/reactivate_carp.png rename from docs/images/infrastructure/network/reactivate_carp.png rename to sysadm/images/infrastructure/network/reactivate_carp.png diff --git a/docs/images/infrastructure/network/sync.png b/sysadm/images/infrastructure/network/sync.png rename from docs/images/infrastructure/network/sync.png rename to sysadm/images/infrastructure/network/sync.png diff --git a/sysadm/network-architecture/how-to-access-firewall-settings.rst b/sysadm/network-architecture/how-to-access-firewall-settings.rst --- a/sysadm/network-architecture/how-to-access-firewall-settings.rst +++ b/sysadm/network-architecture/how-to-access-firewall-settings.rst @@ -3,5 +3,52 @@ How to access firewall settings =============================== -.. todo:: - This page is a work in progress. For now, please refer to the existing documentation :ref:`swh-devel:network_configuration`. \ No newline at end of file +.. admonition:: Intended audience + :class: important + + sysadm staff members + +The firewalls are 2 `OPNsense `_ VMs deployed on the PROXMOX +cluster with an `High Availability +`_ +configuration. + +They are sharing a virtual IP on each VLAN to act as the gateway. Only one of the 2 +firewalls is owning all the GW ips at the same time. The owner is called the ``PRIMARY`` + +.. list-table:: + :header-rows: 1 + + * - Nominal Role + - name (link to the inventory) + - login page + * - PRIMARY + - `pushkin `_ + - `https://pushkin.internal.softwareheritage.org `_ + * - BACKUP + - `glyptotek `_ + - `https://glyptotek.internal.softwareheritage.org `_ + +Access to the gui of the secondary firewall +------------------------------------------- + +The secondary firewall is not directly reachable for VPN user. As the OpenVPN service is +also running when the firewall is a backup, the packets coming from tne VPN are routed +to the local VPN on the secondary and lost. + +To access to GUI, a tunnel can be used: + + ssh -L 8443:pushkin.internal.softwareheritage.org:443 pergamon.internal.softwareheritage.org + +Once the tunnel is created, the gui is accessible at https://localhost:8443 in any +browser + +Configuration backup +-------------------- + +The configuration is automatically committed on a `git repository +`_. Each firewall +regularly pushes its configuration on a dedicated branch of the repository. + +The configuration is visible on the `System / Configuration / Backups +`_ page of each one. diff --git a/sysadm/network-architecture/how-to-upgrade-firewall-os.rst b/sysadm/network-architecture/how-to-upgrade-firewall-os.rst --- a/sysadm/network-architecture/how-to-upgrade-firewall-os.rst +++ b/sysadm/network-architecture/how-to-upgrade-firewall-os.rst @@ -3,5 +3,127 @@ How to upgrade firewall OS ========================== -.. todo:: - This page is a work in progress. For now, please refer to the existing documentation :ref:`swh-devel:network_configuration`. \ No newline at end of file +.. admonition:: Intended audience + :class: important + + sysadm staff members. + +Initial status +^^^^^^^^^^^^^^ + +This is the nominal status of the firewalls: + +.. list-table:: + :header-rows: 1 + + * - Firewall + - Status + * - pushkin + - PRIMARY + * - glyptotek + - BACKUP + +Preparation +^^^^^^^^^^^ + +* Connect to the `principal `_ (pushkin + here) +* Check the `CARP status + `_ to ensure the + firewall is the principal (must have the status MASTER for all the IPS) +* Connect to the `backup `_ (glytotek + here) +* Check the `CARP status + `__ to ensure the + firewall is the backup (must have the status BACKUP for all the IPS) +* Ensure the 2 firewalls are in sync: + + * On the principal, go to the `High availability status + `_ and force a + synchronization + * click on the button on the right of ``Synchronize config to backup`` + +.. image:: ../images/infrastructure/network/sync.png + +* Switch the principal/backup to prepare the upgrade of the master (The switch is + transparent from the user perspective and can be done without service interruption) + + * [1] On the principal, go to the `Virtual IPS status + `_ page + * Activate the CARP maintenance mode + + .. image:: ../images/infrastructure/network/carp_maintenance.png + + * check the status of the VIPs, they must be ``BACKUP`` on pushkin and ``PRIMARY`` on glyptotek + + +* wait a few minutes to let the monitoring detect if there are connection issues, check + ssh connection on several servers on different VLANs (staging, admin, ...) + +If everything is ok, proceed to the next section. + +Upgrade the first firewall +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Before starting this section, the firewall statuses should be: + +.. list-table:: + :header-rows: 1 + + * - Firewall + - Status + * - pushkin + - BACKUP + * - glyptotek + - PRIMARY + +If not, be sure of what you are doing and adapt the links accordingly + +* [2] go to the `System Firmware: status + `_ page + (pushkin here) +* Click on the ``Check for upgrades`` button + +.. image:: ../images/infrastructure/network/check_for_upgrade.png + +* follow the interface indication, one or several reboots can be necessary depending to + the number of upgrade to apply + +.. image:: ../images/infrastructure/network/proceed_update.png + +* repeat from the ``Check for upgrades`` operation until there is no upgrades to apply +* Switch the principal/backup to restore ``pushkin`` as the principal: + + * on the current backup (pushkin here) go to `Virtual IPS status + `_ + * [3] click on `Leave Persistent CARP Maintenance Mode` + + .. image:: ../images/infrastructure/network/reactivate_carp.png + + * refresh the page, the role should have changed from ``BACKUP`` to ``MASTER`` + * check on the other firewall, if the roles is indeed ``BACKUP`` for all the IPs + +* Wait few moment to ensure everything is ok with the new version + +Upgrade the second firewall +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Before starting this section, the firewall statuses should be: + +.. list-table:: + :header-rows: 1 + + * - Firewall + - Status + * - pushkin + - PRIMARY + * - glyptotek + - BACKUP + +If not, be sure of what you are doing and adapt the links accordingly + +* Proceed to the second firewall upgrade + + * perform [1] on the backup (should be ``glyptotek`` here) + * perform [2] on the backup (should be ``glyptotek`` here) + * perform [3] on the backup (should be ``glyptotek`` here) diff --git a/sysadm/network-architecture/index.rst b/sysadm/network-architecture/index.rst --- a/sysadm/network-architecture/index.rst +++ b/sysadm/network-architecture/index.rst @@ -8,3 +8,4 @@ reference-network-configuration how-to-access-firewall-settings how-to-upgrade-firewall-os + service-urls diff --git a/sysadm/network-architecture/reference-network-configuration.rst b/sysadm/network-architecture/reference-network-configuration.rst --- a/sysadm/network-architecture/reference-network-configuration.rst +++ b/sysadm/network-architecture/reference-network-configuration.rst @@ -3,6 +3,11 @@ Reference: Network configuration ================================ +.. admonition:: Intended audience + :class: important + + sysadm staff members. + The network is split in several VLANs provided by the INRIA network team: .. thumbnail:: ../images/network.svg @@ -13,21 +18,24 @@ All inter vlan communications are filtered by our firewalls `pushkin` and `glyptotek`. .. todo:: - Check the for more information. + Check the :ref:`firewall settings ` page for more information. VLAN1300 - Public network ~~~~~~~~~~~~~~~~~~~~~~~~~ -The detail of this range is available in this `VLAN1300 inventory page `_ +The detail of this range is available in this `VLAN1300 inventory page +`_ -All the inbound traffic is firewalled by the INRIA gateway. The detail of the opened ports is -visible on the private archive in the file :file:`sysadm/Software_Heritage_VLAN1300_plan.ods` +All the inbound traffic is firewalled by the INRIA gateway. The detail of the opened +ports is visible on the private archive in the file +:file:`sysadm/Software_Heritage_VLAN1300_plan.ods` Some nodes are directly exposed on this network for special needs: * moma: the main archive entry point * production workers: to have different visible ips during forge crawling -* pergamon: act as a reverse proxy for some public sites (debian repository, annex, sentry, ...) +* pergamon: act as a reverse proxy for some public sites (debian repository, annex, + sentry, ...) * forge: needs some special rules VLAN440 - Production network @@ -35,20 +43,22 @@ All the nodes dedicated to the main archive are deployed in this network. -The detail of this range is available in this `VLAN440 inventory page `_ +The detail of this range is available in this `VLAN440 inventory page +`_ -For historical reasons, some admin nodes are deployed in this range (monitoring, ci, ...) -and will be progressively moved into the admin network. +For historical reasons, some admin nodes are deployed in this range (monitoring, ci, +...) and will be progressively moved into the admin network. The internal domain associted to this vlan is ``.internal.staging.swh.network`` VLAN443 - Staging network ~~~~~~~~~~~~~~~~~~~~~~~~~ -All the nodes dedicated to the staging version of the archive are deployed on this network. -POCs and temporary nodes can also take place in the range. +All the nodes dedicated to the staging version of the archive are deployed on this +network. POCs and temporary nodes can also take place in the range. -The detail of this range is visible in this `VLAN443 inventory page `_ +The detail of this range is visible in this `VLAN443 inventory page +`_ The internal domain associted to this vlan is ``.internal.staging.swh.network`` @@ -57,7 +67,8 @@ This network is dedicated for admin and support nodes. -The detail of this range is visible in this `VLAN442 inventory page `_. +The detail of this range is visible in this `VLAN442 inventory page +`_. -The internal domain associted to this vlan is ``.internal.admin.swh.network`` +The internal domain associated to this vlan is ``.internal.admin.swh.network`` diff --git a/docs/infrastructure/service-urls.rst b/sysadm/network-architecture/service-urls.rst rename from docs/infrastructure/service-urls.rst rename to sysadm/network-architecture/service-urls.rst --- a/docs/infrastructure/service-urls.rst +++ b/sysadm/network-architecture/service-urls.rst @@ -1,9 +1,14 @@ +.. _service-url: + Service urls -##################### +============ +.. admonition:: Intended audience + :class: important -This section regroups the urls of the services + Staff members +This section regroups the urls of the services. .. toctree:: :maxdepth: 2