Page MenuHomeSoftware Heritage
Feed Advanced Search

Wed, Sep 14

vsellier closed T4510: [cassandra] Profile the replayer cpu consumption, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
Wed, Sep 14, 9:47 AM · Storage manager, System administration
vsellier closed T4510: [cassandra] Profile the replayer cpu consumption as Resolved.

I close this issue because after the @vlorentz 's analysis it seems there isn't a lot of things to improve

Wed, Sep 14, 9:47 AM · Storage manager, System administration

Tue, Sep 13

vsellier added a comment to T4510: [cassandra] Profile the replayer cpu consumption.

These are the results of the different algorithms tests for the directory_add (with 20 directory replayers)

  • one-by-one
Tue, Sep 13, 4:23 PM · Storage manager, System administration

Mon, Sep 12

douardda added a comment to T4510: [cassandra] Profile the replayer cpu consumption.

FTR here a dot export of the directory profiling file:

Mon, Sep 12, 5:18 PM · Storage manager, System administration

Fri, Sep 9

vsellier changed the status of T4458: Test reaper to automate the cassandra repair actions, a subtask of T4373: [cassandra] Test the new hardware, from Open to Work in Progress.
Fri, Sep 9, 11:49 AM · Storage manager, System administration
vsellier added a comment to T4510: [cassandra] Profile the replayer cpu consumption.

here some profiling of a couple of replayers:

  • origin-visit:
swh@storage-replayer-origin-visit-76f6bf9d75-znqfs:~$ time python -m cProfile -o /tmp/origin-visit.pyprof /opt/swh/.local/bin/swh storage replay --stop-after-objects 10000
WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.100.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.100.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt' for DCAwareRoundRobinPolicy (via host '192.168.100.181:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
Done.
Fri, Sep 9, 11:27 AM · Storage manager, System administration

Thu, Sep 8

vsellier changed the status of T4510: [cassandra] Profile the replayer cpu consumption from Open to Work in Progress.
Thu, Sep 8, 6:29 PM · Storage manager, System administration
vsellier changed the status of T4510: [cassandra] Profile the replayer cpu consumption, a subtask of T4373: [cassandra] Test the new hardware, from Open to Work in Progress.
Thu, Sep 8, 6:29 PM · Storage manager, System administration
vsellier triaged T4510: [cassandra] Profile the replayer cpu consumption as Normal priority.
Thu, Sep 8, 10:38 AM · Storage manager, System administration

Aug 25 2022

vlorentz added a comment to T3841: regularly scrub all the data stores of swh.

Metrics: https://grafana.softwareheritage.org/goto/s3g4jcWVk?orgId=1

Aug 25 2022, 11:22 AM · meta-task, Roadmap 2022, Storage manager

Aug 18 2022

vsellier closed T4446: Prepare the disks and configure zfs as Resolved.

all server reconfigured and cassandra started on them:

/opt/cassandra/bin/nodetool status
Datacenter: sesi_rocquencourt
=============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens  Owns (effective)  Host ID                               Rack 
UN  192.168.100.184  88.65 KiB  16      34.3%             e0c24d24-6f68-4a26-8561-94e67b58211a  rack1
UN  192.168.100.181  84.71 KiB  16      31.3%             1d9b9e7d-b376-4afe-8f67-482e8412f21b  rack1
UN  192.168.100.186  69.07 KiB  16      34.2%             0dd3426d-9159-47bd-9b4e-065ff0fbb889  rack1
UN  192.168.100.183  69.08 KiB  16      37.1%             78281a92-7fa0-43bd-bc33-c5b419ee8715  rack1
UN  192.168.100.185  69.07 KiB  16      32.2%             abf9b69e-3cec-4ac3-a195-a54481e4d9da  rack1
UN  192.168.100.182  74.05 KiB  16      30.9%             eca5ea5d-8bd5-4301-9a5e-ffa01aa1b7e5  rack1
Aug 18 2022, 7:00 PM · Storage manager, System administration
vsellier closed T4446: Prepare the disks and configure zfs, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
Aug 18 2022, 7:00 PM · Storage manager, System administration
vsellier added a comment to T4446: Prepare the disks and configure zfs.

Recreating the zpool correctly:

# mixedused
ls /dev/disk/by-id/nvme-MO003200KXAVU* | grep -v part | xargs -t zpool create -o ashift=12 -O mountpoint=none mixeduse
zfs create -o mountpoint=/srv/cassandra/instance1/data mixeduse/cassandra-instance1-data
Aug 18 2022, 6:16 PM · Storage manager, System administration
vsellier added a comment to T4446: Prepare the disks and configure zfs.

Testing the performances of the different configuration (on a zfs pool with only one disk):

  • disk block: 512k / zpool ashift:9
zpool create -o ashift=9 -O mountpoint=none mixeduse /dev/disk/by-id/nvme-MO003200KXAVU_SJA4N7938I0405A0U
zfs create -o mountpoint=/srv/cassandra/instance1/data -o atime=off -o relatime=on mixeduse/cassandra-data
cd /srv/cassandra/instance1/data
bonnie++ -d . -m cassandra04 -u nobody                                                                                                                                                                                                                                                 
Using uid:65534, gid:65534.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
cassandra04 515496M  293k  99  1.0g  99  703m  99  661k  99  1.4g  91 13717 463
Latency             48216us    7316us    8224us   23303us    7928us    1606us
Version  2.00       ------Sequential Create------ --------Random Create--------
cassandra04         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 16384  98 +++++ +++ 16384   8 +++++ +++ +++++ +++ 16384  99
Latency              2679us    1207us    4851ms    2850us     138us     301us
1.98,2.00,cassandra04,1,1659338044,515496M,,8192,5,293,99,1080974,99,720299,99,661,99,1488832,91,13717,463,16,,,,,28232,98,+++++,+++,2018,8,+++++,+++,+++++,+++,24821,99,48216us,7316us,8224us,23303us,7928us,1606us,2679us,1207us,4851ms,2850us,138us,301us
Aug 18 2022, 5:49 PM · Storage manager, System administration
vsellier added a comment to T4446: Prepare the disks and configure zfs.

The nvme format command didn't succeed on the write intensive disk. It never exits and the disk become unresponsive after that.

Aug 18 2022, 5:15 PM · Storage manager, System administration
vsellier updated the task description for T4446: Prepare the disks and configure zfs.
Aug 18 2022, 4:47 PM · Storage manager, System administration
vsellier changed the status of T4446: Prepare the disks and configure zfs from Open to Work in Progress.
Aug 18 2022, 4:46 PM · Storage manager, System administration

Aug 17 2022

vlorentz removed a subtask for T4136: Add an "history completeness check": Restricted Maniphest Task.
Aug 17 2022, 4:38 PM · Storage manager
vlorentz closed T4136: Add an "history completeness check", a subtask of T3841: regularly scrub all the data stores of swh, as Resolved.
Aug 17 2022, 4:38 PM · meta-task, Roadmap 2022, Storage manager
vlorentz closed T4136: Add an "history completeness check" as Resolved.
Aug 17 2022, 4:38 PM · Storage manager
vlorentz added a subtask for T4136: Add an "history completeness check": Restricted Maniphest Task.
Aug 17 2022, 3:50 PM · Storage manager
vlorentz added a revision to T4136: Add an "history completeness check": D8248: Add checking and reporting of references to missing objects.
Aug 17 2022, 3:44 PM · Storage manager

Aug 12 2022

vsellier added a revision to T4373: [cassandra] Test the new hardware: D8242: sysadmin: bootstrap the cassandra documentation.
Aug 12 2022, 3:34 PM · Storage manager, System administration
vsellier changed the status of T4391: [cassandra] deploy the replaying stack, a subtask of T4373: [cassandra] Test the new hardware, from Open to Work in Progress.
Aug 12 2022, 11:10 AM · Storage manager, System administration
vsellier added a comment to T4373: [cassandra] Test the new hardware.

The puppet code is ready for review. It was updated to support multi instances deployment in anticipation of T4375.

Aug 12 2022, 11:09 AM · Storage manager, System administration
vsellier added a revision to T4373: [cassandra] Test the new hardware: D8236: cassandra: Refactor the former installation scripts.
Aug 12 2022, 11:01 AM · Storage manager, System administration

Aug 10 2022

vsellier added a comment to T4373: [cassandra] Test the new hardware.

For the record, the issues related to the commitlog_directory configuration:

Aug 10 2022, 11:07 AM · Storage manager, System administration
vsellier added a comment to T4373: [cassandra] Test the new hardware.

After spending some time to successfully start a cassandra cluster of 2 nodes with a declarative configuration), these are the observations:

  • A service can't be used to expose the cassandra ports to the clustrer, the pod address must be used. It's because cassandra use the dns name provided as listen address
  • It should work by setting the listen address to 0.0.0.0 but it's stongly recommanded to not use this in the documentation

Setting listen_address to 0.0.0.0 is always wrong.

  • Using internal pod address will avoid multi DC deployment for the future
Aug 10 2022, 10:30 AM · Storage manager, System administration

Aug 9 2022

vlorentz closed Restricted Maniphest Task, a subtask of T3841: regularly scrub all the data stores of swh, as Resolved.
Aug 9 2022, 10:16 AM · meta-task, Roadmap 2022, Storage manager

Aug 8 2022

vlorentz reopened Restricted Maniphest Task, a subtask of T3841: regularly scrub all the data stores of swh, as Open.
Aug 8 2022, 2:23 PM · meta-task, Roadmap 2022, Storage manager

Aug 5 2022

ardumont added a subtask for T3841: regularly scrub all the data stores of swh: T4371: Deploy swh-scrubber on all storage instances.
Aug 5 2022, 4:31 PM · meta-task, Roadmap 2022, Storage manager
ardumont added a parent task for T4371: Deploy swh-scrubber on all storage instances: T3841: regularly scrub all the data stores of swh.
Aug 5 2022, 4:31 PM · System administration, Archive integrity, Storage manager
vlorentz closed Restricted Maniphest Task, a subtask of T3841: regularly scrub all the data stores of swh, as Resolved.
Aug 5 2022, 3:07 PM · meta-task, Roadmap 2022, Storage manager

Aug 4 2022

ardumont closed T4371: Deploy swh-scrubber on all storage instances as Resolved.
Aug 4 2022, 3:57 PM · System administration, Archive integrity, Storage manager
ardumont moved T4371: Deploy swh-scrubber on all storage instances from in-progress to deployed/landed/monitoring on the System administration board.
Aug 4 2022, 3:49 PM · System administration, Archive integrity, Storage manager
ardumont added a comment to T4371: Deploy swh-scrubber on all storage instances.

Deployed both in staging and production [1]:

Aug 4 2022, 3:49 PM · System administration, Archive integrity, Storage manager
ardumont changed the status of T4371: Deploy swh-scrubber on all storage instances from Open to Work in Progress.
Aug 4 2022, 3:03 PM · System administration, Archive integrity, Storage manager
ardumont added a revision to T4371: Deploy swh-scrubber on all storage instances: D8181: scrubber: Make service parametric on the db instance to scrub.
Aug 4 2022, 3:03 PM · System administration, Archive integrity, Storage manager
ardumont moved T4371: Deploy swh-scrubber on all storage instances from Backlog to Weekly backlog on the System administration board.
Aug 4 2022, 11:34 AM · System administration, Archive integrity, Storage manager

Jul 13 2022

vsellier added a comment to T4373: [cassandra] Test the new hardware.

Unfortunately, the operator test is a failure due to the lack of configuration possibility

  • non blocker, the init containers are OOMkilled during the start, it can be solved by editing the cassandra statefulset created by the operator to extend the limits
  • blocker, it's not possible to configure the commitlog_directory explicitly. it's by default on /var/lib/cassandra/commitlog
    • it's not easy to propagate the host mounts to use 2 mountpoints /srv/cassandra and /srv/cassandra/commitlog without tweaking the kernel / rancher configuration
    • it's not possible to add a second volume on the pod description created by the operator
Jul 13 2022, 10:25 AM · Storage manager, System administration
vlorentz added a subtask for T3841: regularly scrub all the data stores of swh: Restricted Maniphest Task.
Jul 13 2022, 9:40 AM · meta-task, Roadmap 2022, Storage manager

Jul 12 2022

vsellier added a revision to T4373: [cassandra] Test the new hardware: D8116: Deploy the cassandra operator on the production cassandra cluster.
Jul 12 2022, 3:01 PM · Storage manager, System administration
vsellier changed the status of T4374: [cassandra] Test basic topology, a subtask of T4379: [cassandra] create etcd / controlplane servers, from Open to Work in Progress.
Jul 12 2022, 12:10 PM · Storage manager, System administration
vsellier changed the status of T4374: [cassandra] Test basic topology from Open to Work in Progress.
Jul 12 2022, 12:10 PM · Storage manager, System administration
vsellier closed T4389: [cassandra] Configure the monitoring of the cluster, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
Jul 12 2022, 12:09 PM · Storage manager, System administration

Jul 11 2022

vsellier closed T4379: [cassandra] create etcd / controlplane servers as Resolved.

Finally, the cluster is up.
I'm not sure what unstuck the node registration, but I suspect a node with all the roles is needed to bootstrap the cluster.
I tried this initially, it didn't worked, but I'm not sure in which status the cluster was.

Jul 11 2022, 4:33 PM · Storage manager, System administration
vsellier closed T4379: [cassandra] create etcd / controlplane servers, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
Jul 11 2022, 4:33 PM · Storage manager, System administration
vsellier added a revision to T4373: [cassandra] Test the new hardware: D8105: Install zfs and docker on the cassandra node to prepare the cass operator tests.
Jul 11 2022, 9:33 AM · Storage manager, System administration

Jul 7 2022

vsellier added a comment to T4379: [cassandra] create etcd / controlplane servers.

The management nodes were correctly created but it seems rancher is having some issuer to register them in the cluster.

Jul 7 2022, 6:52 PM · Storage manager, System administration
vsellier added a revision to T4379: [cassandra] create etcd / controlplane servers: D8094: Declare the kubernetes cluster and management nodes for cassandra.
Jul 7 2022, 12:11 PM · Storage manager, System administration
vsellier changed the status of T4379: [cassandra] create etcd / controlplane servers, a subtask of T4373: [cassandra] Test the new hardware, from Open to Work in Progress.
Jul 7 2022, 11:56 AM · Storage manager, System administration
vsellier changed the status of T4379: [cassandra] create etcd / controlplane servers from Open to Work in Progress.
Jul 7 2022, 11:56 AM · Storage manager, System administration

Jul 5 2022

vsellier removed a parent task for T4374: [cassandra] Test basic topology: T4373: [cassandra] Test the new hardware.
Jul 5 2022, 5:50 PM · Storage manager, System administration
vsellier removed a subtask for T4373: [cassandra] Test the new hardware: T4374: [cassandra] Test basic topology.
Jul 5 2022, 5:50 PM · Storage manager, System administration
vsellier removed a subtask for T4373: [cassandra] Test the new hardware: T4375: [cassandra] One cassandra per data disk.
Jul 5 2022, 5:50 PM · Storage manager, System administration
vsellier removed a parent task for T4375: [cassandra] One cassandra per data disk: T4373: [cassandra] Test the new hardware.
Jul 5 2022, 5:50 PM · Storage manager, System administration
vsellier added a parent task for T4374: [cassandra] Test basic topology: T4379: [cassandra] create etcd / controlplane servers.
Jul 5 2022, 5:49 PM · Storage manager, System administration
vsellier added a subtask for T4379: [cassandra] create etcd / controlplane servers: T4374: [cassandra] Test basic topology.
Jul 5 2022, 5:49 PM · Storage manager, System administration
vsellier added a parent task for T4375: [cassandra] One cassandra per data disk: T4379: [cassandra] create etcd / controlplane servers.
Jul 5 2022, 5:49 PM · Storage manager, System administration
vsellier added a subtask for T4379: [cassandra] create etcd / controlplane servers: T4375: [cassandra] One cassandra per data disk.
Jul 5 2022, 5:49 PM · Storage manager, System administration
vsellier triaged T4379: [cassandra] create etcd / controlplane servers as Normal priority.
Jul 5 2022, 5:47 PM · Storage manager, System administration
vsellier changed the status of T4373: [cassandra] Test the new hardware from Open to Work in Progress.
Jul 5 2022, 5:41 PM · Storage manager, System administration
ardumont updated the task description for T4373: [cassandra] Test the new hardware.
Jul 5 2022, 10:04 AM · Storage manager, System administration
vsellier triaged T4375: [cassandra] One cassandra per data disk as Normal priority.
Jul 5 2022, 9:52 AM · Storage manager, System administration
vsellier triaged T4374: [cassandra] Test basic topology as Normal priority.
Jul 5 2022, 9:43 AM · Storage manager, System administration
vsellier triaged T4373: [cassandra] Test the new hardware as Normal priority.
Jul 5 2022, 9:36 AM · Storage manager, System administration

Jul 4 2022

ardumont added a project to T4371: Deploy swh-scrubber on all storage instances: System administration.
Jul 4 2022, 10:47 AM · System administration, Archive integrity, Storage manager
ardumont added projects to T4371: Deploy swh-scrubber on all storage instances: Storage manager, Archive integrity.
Jul 4 2022, 10:47 AM · System administration, Archive integrity, Storage manager

Jul 1 2022

douardda triaged T4370: Refactor the origin visit data model (aka get rid of the OriginVisit model object) as High priority.
Jul 1 2022, 4:35 PM · Storage manager, Data Model
douardda triaged T4368: Loosen "foreign key" validation in storages used as mirror ingestion endpoint as High priority.
Jul 1 2022, 11:13 AM · Storage manager
douardda created T4368: Loosen "foreign key" validation in storages used as mirror ingestion endpoint.
Jul 1 2022, 11:13 AM · Storage manager

Jun 23 2022

vlorentz added a comment to T4185: Loader profiling : Add Measure of ignored objects .

the Git loader now exports a swh_loader_filtered_objects_total metric. We should generalize this to other loaders eventually; using one of the options above

Jun 23 2022, 10:40 AM · Storage manager

Jun 10 2022

douardda triaged T4325: Remove (useless) metadata_authority and metadata_fetcher from the journal as Normal priority.
Jun 10 2022, 2:51 PM · Storage manager

Jun 2 2022

vlorentz moved T3841: regularly scrub all the data stores of swh from Backlog to Work in progress on the Roadmap 2022 board.
Jun 2 2022, 9:58 AM · meta-task, Roadmap 2022, Storage manager
vlorentz moved T2214: Scale-out graph and database storage in production from Backlog to Work in progress on the Roadmap 2022 board.
Jun 2 2022, 9:57 AM · meta-task, Roadmap 2022, Roadmap 2021, Storage manager

May 31 2022

douardda closed T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Resolved.
May 31 2022, 7:28 PM · Storage manager
douardda added a revision to T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function : D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.
May 31 2022, 4:43 PM · Storage manager
douardda triaged T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Normal priority.
May 31 2022, 4:23 PM · Storage manager

May 4 2022

olasd added a project to T4185: Loader profiling : Add Measure of ignored objects : Storage manager.
May 4 2022, 10:54 AM · Storage manager

Apr 19 2022

anlambert closed T4090: Add method to efficiently retrieve latest statuses of origin visits as Resolved.

Feature has been implemented and deployed, closing this.

Apr 19 2022, 11:42 AM · Storage manager

Apr 12 2022

ardumont closed T4137: Deploy swh.storage v1.3 as Resolved.
Apr 12 2022, 5:08 PM · System administration, Storage manager
anlambert added a revision to T4090: Add method to efficiently retrieve latest statuses of origin visits : D7559: common/archive: Improve lookup_origin_visits performance.
Apr 12 2022, 4:32 PM · Storage manager
ardumont moved T4137: Deploy swh.storage v1.3 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 12 2022, 3:54 PM · System administration, Storage manager
ardumont added a comment to T4137: Deploy swh.storage v1.3.

done ^

Apr 12 2022, 3:54 PM · System administration, Storage manager
ardumont moved T4137: Deploy swh.storage v1.3 from deployed/landed/monitoring to in-progress on the System administration board.
Apr 12 2022, 3:27 PM · System administration, Storage manager
ardumont added a comment to T4137: Deploy swh.storage v1.3.

New fix needed so another round for v1.3.1

Apr 12 2022, 3:26 PM · System administration, Storage manager
anlambert added a revision to T4090: Add method to efficiently retrieve latest statuses of origin visits : D7552: origin_get_with_statuses: Fix case when fetched visits list is empty.
Apr 12 2022, 12:10 PM · Storage manager

Apr 11 2022

ardumont moved T4137: Deploy swh.storage v1.3 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 11 2022, 4:06 PM · System administration, Storage manager
ardumont updated the task description for T4137: Deploy swh.storage v1.3.
Apr 11 2022, 4:06 PM · System administration, Storage manager
ardumont updated the task description for T4137: Deploy swh.storage v1.3.
Apr 11 2022, 4:03 PM · System administration, Storage manager
ardumont updated the task description for T4137: Deploy swh.storage v1.3.
Apr 11 2022, 4:01 PM · System administration, Storage manager
ardumont updated the task description for T4137: Deploy swh.storage v1.3.
Apr 11 2022, 2:29 PM · System administration, Storage manager
ardumont changed the status of T4137: Deploy swh.storage v1.3 from Open to Work in Progress.
Apr 11 2022, 2:27 PM · System administration, Storage manager
ardumont added a project to T4137: Deploy swh.storage v1.3: System administration.
Apr 11 2022, 2:27 PM · System administration, Storage manager
ardumont updated the task description for T4137: Deploy swh.storage v1.3.
Apr 11 2022, 2:25 PM · System administration, Storage manager
ardumont renamed T4137: Deploy swh.storage v1.3 from storage v1.3 to Deploy swh.storage v1.3.
Apr 11 2022, 2:10 PM · System administration, Storage manager
ardumont triaged T4137: Deploy swh.storage v1.3 as Normal priority.
Apr 11 2022, 2:10 PM · System administration, Storage manager
bchauvet removed a project from T4136: Add an "history completeness check": Roadmap 2022.
Apr 11 2022, 12:02 PM · Storage manager
bchauvet triaged T4136: Add an "history completeness check" as Normal priority.
Apr 11 2022, 12:02 PM · Storage manager

Mar 30 2022

vsellier closed T4117: Storage metrics not refreshed as Resolved.

Thanks olasd for restarting the service following this documentation: https://docs.gunicorn.org/en/stable/signals.html#upgrading-to-a-new-binary-on-the-fly

First, replace the old binary with a new one, then send a USR2 signal to the current master process. It executes a new binary whose PID file is postfixed with .2 (e.g. /var/run/gunicorn.pid.2), which in turn starts a new master process and new worker processes:
At this point, two instances of Gunicorn are running, handling the incoming requests together. To phase the old instance out, you have to send a WINCH signal to the old master process, and its worker processes will start to gracefully shut down.

Mar 30 2022, 11:51 AM · Storage manager, System administration