Some process are oom killled.
The processing rate slowly until all the thread got oom killed
For example:
root@met:~# journalctl | grep 3550425 Jun 07 13:42:52 met kernel: [3550425] 1024 3550425 2981803 2916149 23543808 492 0 python Jun 07 13:51:42 met kernel: [3550425] 1024 3550425 6110159 6044894 48623616 492 0 python Jun 07 14:45:21 met kernel: [3550425] 1024 3550425 33115547 33050256 265089024 492 0 python Jun 07 14:45:21 met kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0-1,global_oom,task_memcg=/user.slice/user-1025.slice/session-27888.scope,task=python,pid=3550425,uid=1024 Jun 07 14:45:21 met kernel: Out of memory: Killed process 3550425 (python) total-vm:132462188kB, anon-rss:132199584kB, file-rss:1440kB, shmem-rss:0kB, UID:1024 pgtables:258876kB oom_score_adj:0 Jun 07 14:45:26 met kernel: oom_reaper: reaped process 3550425 (python), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
It seems the oom is always hapenning during the flush
/home/aeviso/origin-layer-50.log:2022-06-07 13:30:50,471-INFO - 3550425 - swh.provenance.origin.proceed_origin:89: Processed origin https://bitbucket.org/_wingers/flappy-bird-clone.git in 0:00:03.427797 /home/aeviso/origin-layer-50.log:2022-06-07 13:30:51,104-INFO - 3550425 - swh.provenance.origin.proceed_origin:77: Processing origin https://bitbucket.org/_yakov/gulp_template.git /home/aeviso/origin-layer-50.log:2022-06-07 13:30:54,630-INFO - 3550425 - swh.provenance.origin.proceed_origin:89: Processed origin https://bitbucket.org/_yakov/gulp_template.git in 0:00:03.525308 /home/aeviso/origin-layer-50.log:2022-06-07 13:30:55,156-INFO - 3550425 - swh.provenance.origin.proceed_origin:77: Processing origin https://bitbucket.org/_zzt/project-dfa.git /home/aeviso/origin-layer-50.log:2022-06-07 13:30:57,557-INFO - 3550425 - swh.provenance.origin.proceed_origin:89: Processed origin https://bitbucket.org/_zzt/project-dfa.git in 0:00:02.400495 /home/aeviso/origin-layer-50.log:2022-06-07 13:30:57,878-INFO - 3550425 - swh.provenance.origin.proceed_origin:77: Processing origin https://bitbucket.org/a-apshev/ngp.boost /home/aeviso/origin-layer-50.log:2022-06-07 13:30:58,935-INFO - 3550425 - swh.provenance.origin.proceed_origin:89: Processed origin https://bitbucket.org/a-apshev/ngp.boost in 0:00:01.056042 /home/aeviso/origin-layer-50.log:2022-06-07 13:31:00,822-INFO - 3550425 - swh.provenance.origin.proceed_origin:77: Processing origin https://bitbucket.org/a-boy/sage.git /home/aeviso/origin-layer-50.log:2022-06-07 14:42:37,626-INFO - 3550425 - swh.provenance.origin.proceed_origin:89: Processed origin https://bitbucket.org/a-boy/sage.git in 1:11:36.802968
aeviso@met:~$ ps -ef | grep defunct | awk '{print $2}' | xargs -i{} bash -c "grep {} ~/*.log | tail -n 1"
/home/aeviso/origin-layer-50.log:2022-06-07 13:36:42,007-INFO - 3550398 - swh.provenance.origin.proceed_origin:89: Processed origin https://bitbucket.org/VirtualReality/opensim.git in 0:51:08.826426
/home/aeviso/origin-layer-50.log:2022-06-07 14:42:37,626-INFO - 3550425 - swh.provenance.origin.proceed_origin:89: Processed origin https://bitbucket.org/a-boy/sage.git in 1:11:36.802968
/home/aeviso/origin-layer-50.log:2022-06-07 13:39:25,741-INFO - 3550429 - swh.provenance.origin.proceed_origin:89: Processed origin https://bitbucket.org/VirtualReality/opensim.git in 0:53:52.682789