I just noticed moma seems to have a cron running /usr/local/bin/refresh-savecodenow-statuses, but there are up to four of them running at any one time.
This doesn't seem right; and they probably interfere which each other. (and I can imagine it cascading by becoming slower and slower)
Instead, it should probably be migrated to a systemd timer, so systemd can make sure there aren't ever two instances running at the same time, no matter how slow they are.
Indeed, a quick htop which demonstrates it:
XY48 root 20 0 2392 692 628 S 0.0 0.0 0:00.00 │ └─ /bin/sh -c chronic sh -c '/usr/local/bin/refresh-savecodenow-statuses' XY49 root 20 0 17700 10164 4472 S 0.0 0.0 4:01.25 │ └─ /usr/bin/perl /usr/bin/chronic sh -c /usr/local/bin/refresh-savecodenow-statuses XY50 root 20 0 2392 756 692 S 0.0 0.0 0:00.00 │ └─ /bin/sh -c /usr/local/bin/refresh-savecodenow-statuses XY51 root 20 0 8336 2872 2640 S 0.0 0.0 0:00.00 │ └─ bash /usr/local/bin/refresh-savecodenow-statuses XY52 root 20 0 156M 88576 20360 S 0.0 0.3 0:03.16 │ └─ python3 /usr/bin/django-admin refresh_savecodenow_statuses XY11 root 20 0 2392 696 628 S 0.0 0.0 0:00.00 │ └─ /bin/sh -c chronic sh -c '/usr/local/bin/refresh-savecodenow-statuses' XY12 root 20 0 17604 10156 4468 S 0.0 0.0 4:05.14 │ └─ /usr/bin/perl /usr/bin/chronic sh -c /usr/local/bin/refresh-savecodenow-statuses XYXY root 20 0 2392 760 688 S 0.0 0.0 0:00.00 │ └─ /bin/sh -c /usr/local/bin/refresh-savecodenow-statuses XY14 root 20 0 8336 3064 2840 S 0.0 0.0 0:00.00 │ └─ bash /usr/local/bin/refresh-savecodenow-statuses XY15 root 20 0 156M 88980 20180 S 0.0 0.3 0:02.54 │ └─ python3 /usr/bin/django-admin refresh_savecodenow_statuses XY62 root 20 0 2392 756 688 S 0.0 0.0 0:00.00 │ └─ /bin/sh -c chronic sh -c '/usr/local/bin/refresh-savecodenow-statuses' XY63 root 20 0 17604 10292 4600 S 0.0 0.0 4:21.16 │ └─ /usr/bin/perl /usr/bin/chronic sh -c /usr/local/bin/refresh-savecodenow-statuses XY65 root 20 0 2392 756 692 S 0.0 0.0 0:00.00 │ └─ /bin/sh -c /usr/local/bin/refresh-savecodenow-statuses XY66 root 20 0 8336 2884 2668 S 0.0 0.0 0:00.00 │ └─ bash /usr/local/bin/refresh-savecodenow-statuses XY67 root 20 0 156M 88548 20356 S 0.0 0.3 0:02.72 │ └─ python3 /usr/bin/django-admin refresh_savecodenow_statuses
I raised the priority for this as this seems to be a recurrent situation. Each time I
check this, it's right there, at least 3 refresh at the same time.
P1114 when ran sequentially does not seem to be exceeding 1 minute so the timing still happens every minute (in the diff).