diff --git a/docs/infrastructure/index.rst b/docs/infrastructure/index.rst index 422ac78..300e5c0 100644 --- a/docs/infrastructure/index.rst +++ b/docs/infrastructure/index.rst @@ -1,14 +1,15 @@ .. _infrastructure: Infrastructure ############## -.. keep this in sync with the 'sysadm' section in swh-docs/docs/index.rst +.. keep this in sync with the 'infrastructure' section in swh-docs/docs/index.rst This section regroups the knowledge base and procedures relative to the |swh| infrastructure management. .. toctree:: :maxdepth: 2 :titlesonly: + service-urls network diff --git a/docs/infrastructure/network.rst b/docs/infrastructure/network.rst index fcb9e72..a1e22ea 100644 --- a/docs/infrastructure/network.rst +++ b/docs/infrastructure/network.rst @@ -1,172 +1,172 @@ Network documentation ##################### -.. keep this in sync with the 'sysadm' section in swh-docs/docs/index.rst +.. keep this in sync with the 'infrastructure' section in swh-docs/docs/index.rst This section regroups the knowledge base for our network components. .. toctree:: :maxdepth: 2 :titlesonly: Network architecture ******************** The network is split in several VLANs provided by the INRIA network team: .. thumbnail:: ../images/network.png Firewalls ========= The firewalls are 2 `OPNsense `_ VMs deployed on the PROXMOX cluster with an `High Availability `_ configuration. They are sharing a virtual IP on each VLAN to act as the gateway. Only one of the 2 firewalls is owning all the GW ips at the same time. The owner is called the ``PRIMARY`` .. list-table:: :header-rows: 1 * - Nominal Role - name (link to the inventory) - login page * - PRIMARY - `pushkin `_ - `https://pushkin.internal.softwareheritage.org `_ * - BACKUP - `glyptotek `_ - `https://glyptotek.internal.softwareheritage.org `_ Access to the gui of the secondary firewall ---------------------------------------------- The secondary firewall is not directly reachable for VPN user. As the OpenVPN service is also running when the firewall is a backup, the packets coming from tne VPN are routed to the local VPN on the secondary and lost. To access to GUI, a tunnel can be used: ssh -L 8443:pushkin.internal.softwareheritage.org:443 pergamon.internal.softwareheritage.org Once the tunnel is created, the gui is accessible at https://localhost:8443 in any browser Configuration backup -------------------- The configuration is automatically committed on a `git repository `_. Each firewall regularly pushes its configuration on a dedicated branch of the repository. The configuration is visible on the `System / Configuration / Backups `_ page of each one. Upgrade procedure ----------------- Initial status ^^^^^^^^^^^^^^ This is the nominal status of the firewalls: .. list-table:: :header-rows: 1 * - Firewall - Status * - pushkin - PRIMARY * - glyptotek - BACKUP Preparation ^^^^^^^^^^^ * Connect to the `principal `_ (pushkin here) * Check the `CARP status `_ to ensure the firewall is the principal (must have the status MASTER for all the IPS) * Connect to the `backup `_ (glytotek here) * Check the `CARP status `__ to ensure the firewall is the backup (must have the status BACKUP for all the IPS) * Ensure the 2 firewalls are in sync: * On the principal, go to the `High availability status `_ and force a synchronization * click on the button on the right of ``Synchronize config to backup`` .. image:: ../images/infrastructure/network/sync.png * Switch the principal/backup to prepare the upgrade of the master (The switch is transparent from the user perspective and can be done without service interruption) * [1] On the principal, go to the `Virtual IPS status `_ page * Activate the CARP maintenance mode .. image:: ../images/infrastructure/network/carp_maintenance.png * check the status of the VIPs, they must be ``BACKUP`` on pushkin and ``PRIMARY`` on glyptotek * wait a few minutes to let the monitoring detect if there are connection issues, check ssh connection on several servers on different VLANs (staging, admin, ...) If everything is ok, proceed to the next section. Upgrade the first firewall ^^^^^^^^^^^^^^^^^^^^^^^^^^ Before starting this section, the firewall statuses should be: .. list-table:: :header-rows: 1 * - Firewall - Status * - pushkin - BACKUP * - glyptotek - PRIMARY If not, be sure of what you are doing and adapt the links accordingly * [2] go to the `System Firmware: status `_ page (pushkin here) * Click on the ``Check for upgrades`` button .. image:: ../images/infrastructure/network/check_for_upgrade.png * follow the interface indication, one or several reboots can be necessary depending to the number of upgrade to apply .. image:: ../images/infrastructure/network/proceed_update.png * repeat from the ``Check for upgrades`` operation until there is no upgrades to apply * Switch the principal/backup to restore ``pushkin`` as the principal: * on the current backup (pushkin here) go to `Virtual IPS status `_ * [3] click on `Leave Persistent CARP Maintenance Mode` .. image:: ../images/infrastructure/network/reactivate_carp.png * refresh the page, the role should have changed from ``BACKUP`` to ``MASTER`` * check on the other firewall, if the roles is indeed ``BACKUP`` for all the IPs * Wait few moment to ensure everything is ok with the new version Upgrade the second firewall ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Before starting this section, the firewall statuses should be: .. list-table:: :header-rows: 1 * - Firewall - Status * - pushkin - PRIMARY * - glyptotek - BACKUP If not, be sure of what you are doing and adapt the links accordingly * Proceed to the second firewall upgrade * perform [1] on the backup (should be ``glyptotek`` here) * perform [2] on the backup (should be ``glyptotek`` here) * perform [3] on the backup (should be ``glyptotek`` here) diff --git a/docs/infrastructure/service-urls.rst b/docs/infrastructure/service-urls.rst new file mode 100644 index 0000000..2804b2e --- /dev/null +++ b/docs/infrastructure/service-urls.rst @@ -0,0 +1,191 @@ +Service urls +##################### + + +This section regroups the urls of the services + + +.. toctree:: + :maxdepth: 2 + :titlesonly: + +Staging +------- + +Try to use the staging environment as far as possible for your tests + +Public urls +~~~~~~~~~~~ + ++---------------------------------------+-------------------------------------------+ +| Service | URL | ++=======================================+===========================================+ +| swh-web | https://webapp.staging.swh.network | ++---------------------------------------+-------------------------------------------+ +| swh-deposit | https://deposit.staging.swh.network | ++---------------------------------------+-------------------------------------------+ +| swh-objstorage read-only (for mirror) | https://objstorage.staging.swh.network | ++---------------------------------------+-------------------------------------------+ +| Journal TLS | broker1.journal.softwareheritage.org:9093 | ++---------------------------------------+-------------------------------------------+ + +Internal services +~~~~~~~~~~~~~~~~~ + ++--------------------------+------------------------------------------------------+--------+------------+ +| Service | URL | VPN[1] | Private[2] | ++==========================+======================================================+========+============+ +| swh-storage | http://storage1.internal.staging.swh.network:5002 | | X | ++--------------------------+------------------------------------------------------+--------+------------+ +| swh-storage read-only | http://webapp.internal.staging.swh.network:5002 | X | | ++--------------------------+------------------------------------------------------+--------+------------+ +| swh-objstorage | http://storage1.internal.staging.swh.network:5003 | | X | ++--------------------------+------------------------------------------------------+--------+------------+ +| swh-objstorage read-only | http://objstorage0.internal.staging.swh.network:5003 | X | | ++--------------------------+------------------------------------------------------+--------+------------+ +| swh-scheduler | http://scheduler0.internal.staging.swh.network:5008 | X | | ++--------------------------+------------------------------------------------------+--------+------------+ +| swh-counters | http://counters0.internal.staging.swh.network:5011 | X | | ++--------------------------+------------------------------------------------------+--------+------------+ +| swh-search | http://webapp.internal.staging.swh.network:5010 | X | | ++--------------------------+------------------------------------------------------+--------+------------+ +| swh-search | http://search0.internal.staging.swh.network:5010 | | X | ++--------------------------+------------------------------------------------------+--------+------------+ +| swh-vault | http://vault.internal.staging.swh.network:5005 | | X | ++--------------------------+------------------------------------------------------+--------+------------+ +| Journal plaintext | journal0.internal.staging.swh.network:9092 | | X | ++--------------------------+------------------------------------------------------+--------+------------+ +| Journal internal TLS | journal0.internal.staging.swh.network:9094 | | X | ++--------------------------+------------------------------------------------------+--------+------------+ + +SWH backends +~~~~~~~~~~~~ + ++--------------------+---------------------------------------------------------+--------+------------+ +| Backend | URL | VPN[1] | Private[2] | ++====================+=========================================================+========+============+ +| RabbitMq GUI | http://scheduler0.internal.staging.swh.network:15672 | X | | ++--------------------+---------------------------------------------------------+--------+------------+ +| archive database | db1.internal.staging.swh.network:5432/swh | X | | ++--------------------+---------------------------------------------------------+--------+------------+ +| webapp database | db1.internal.staging.swh.network:5432/swh-web | X | | ++--------------------+---------------------------------------------------------+--------+------------+ +| deposit database | db1.internal.staging.swh.network:5432/swh-deposit | X | | ++--------------------+---------------------------------------------------------+--------+------------+ +| vault database | db1.internal.staging.swh.network:5432/swh-vault | X | | ++--------------------+---------------------------------------------------------+--------+------------+ +| scheduler database | db1.internal.staging.swh.network:5432/swh-scheduler | X | | ++--------------------+---------------------------------------------------------+--------+------------+ +| lister database | db1.internal.staging.swh.network:5432/swh-lister | X | | ++--------------------+---------------------------------------------------------+--------+------------+ +| swh-search ES | http://search-esnode0.internal.staging.swh.network:9200 | | X | ++--------------------+---------------------------------------------------------+--------+------------+ +| Counters redis | counters0.internal.staging.swh.network:6379 | | X | ++--------------------+---------------------------------------------------------+--------+------------+ + +Production +---------- + +.. _public-urls-1: + +Public urls +~~~~~~~~~~~ + ++---------------------------------------+-----------------------------------------------+ +| Service | URL | ++=======================================+===============================================+ +| swh-web | https://archive.softwareheritage.org | ++---------------------------------------+-----------------------------------------------+ +| swh-deposit | https://deposit.softwareheritage.org | ++---------------------------------------+-----------------------------------------------+ +| swh-objstorage read-only (for mirror) | N/A | ++---------------------------------------+-----------------------------------------------+ +| Journal TLS | broker[1-4].journal.softwareheritage.org:9093 | ++---------------------------------------+-----------------------------------------------+ + +.. _internal-services-1: + +Internal services +~~~~~~~~~~~~~~~~~ + ++--------------------------+----------------------------------------------------------------+--------+------------+ +| Service | URL | VPN[1] | Private[2] | ++==========================+================================================================+========+============+ +| swh-web test/validation | https://webapp1.internal.softwareheritage.org | X | | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-storage | http://saam.internal.softwareheritage.org:5002 | | X | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-storage read-only | http://webapp1.internal.softwareheritage.org:5002 | X | | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-storage read-only | http://moma.internal.softwareheritage.org:5002 | X | | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-objstorage | http://saam.internal.softwareheritage.org:5003 | | X | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-objstorage read-only | N/A | | | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-scheduler | http://saatchi.internal.softwareheritage.org:5008 | X | | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-counters | http://counters1.internal.softwareheritage.org:5011 | X | | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-search | http://webapp1.internal.softwareheritage.org:5010 | X | | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-search | http://moma.internal.softwareheritage.org:5010 | X | | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-search | http://search1.internal.softwareheritage.org:5010 | | X | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| swh-vault | http://vangogh.euwest.azure.internal.softwareheritage.org:5005 | | X | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| Journal plaintext | kafka[1-4].internal.softwareheritage.org:9092 | | X | ++--------------------------+----------------------------------------------------------------+--------+------------+ +| Journal internal TLS | kafka[1-4].internal.softwareheritage.org:9094 | X | | ++--------------------------+----------------------------------------------------------------+--------+------------+ + +.. _swh-backends-1: + +SWH backends +~~~~~~~~~~~~ + ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| Backend | URL | VPN[1] | Private[2] | ++==========================+=======================================================================+========+============+ +| RabbitMq GUI | http://saatchi.internal.softwareheritage.org:15672 | X | | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| archive database replica | somerset.internal.softwareheritage.org:5432/softwareheritage | X | | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| archive database main | belvedere.internal.softwareheritage.org:5432/softwareheritage | X | | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| webapp database main | belvedere.internal.softwareheritage.org:5432/swh-web | X | | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| scheduler database | belvedere.internal.softwareheritage.org:5432/swh-scheduler | X | | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| lister database | belvedere.internal.softwareheritage.org:5432/swh-lister | X | | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| deposit database | belvedere.internal.softwareheritage.org:5432/softwareheritage-deposit | X | | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| vault database | belvedere.internal.softwareheritage.org:5432/swh-vault | X | | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| swh-search ES | http://search-esnode[1-3].internal.softwareheritage.org:9200 | | X | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ +| Counters redis | counters1.internal.softwareheritage.org:6379 | | X | ++--------------------------+-----------------------------------------------------------------------+--------+------------+ + +Other tools +----------- + ++-------------------+-------------------------------------------------------+--------------------+--------+------------+ +| Tool | URL | Public | VPN[1] | Private[2] | ++===================+=======================================================+====================+========+============+ +| grafana | https://grafana.softwareheritage.org | X | | | ++-------------------+-------------------------------------------------------+--------------------+--------+------------+ +| Kibana | http://kibana0.internal.softwareheritage.org:5601 | | X | | ++-------------------+-------------------------------------------------------+--------------------+--------+------------+ +| Log Elasticsearch | http://search[1-3].internal.softwareheritage.org:9200 | | X | | ++-------------------+-------------------------------------------------------+--------------------+--------+------------+ +| C.M.A.K. | http://getty.internal.softwareheritage.org:9000 | | X | | ++-------------------+-------------------------------------------------------+--------------------+--------+------------+ +| Sentry | https://sentry.softwareheritage.org | X (authentication) | | | ++-------------------+-------------------------------------------------------+--------------------+--------+------------+ + +[1] VPN: URL only accessible when connected to the SoftwareHeritage VPN + +[2] Private: URL only accessible from the internal network, i.e nor public neither accessible through the VPN.