In the mean time, logs can be reached in the dedicated dashboard
Oct 15 2021
Sep 29 2021
Sep 17 2021
Sep 8 2021
Sep 3 2021
Aug 5 2021
Jul 29 2021
Feb 18 2021
Feb 11 2021
Well my concern was about having different versions running at the same time, but Sentry is able to detect the version so that's not it.
I'm not sure to understand the real problem here.
As the indexer and indexer-storage are in same source repository, the versions should match or increase in //. Sentry should be able to deal with it as any other version upgrade.
Feb 8 2021
Feb 2 2021
Jan 6 2021
Dec 21 2020
- before :
root@riverside:~# pvscan PV /dev/sda1 VG riverside-vg lvm2 [<63.98 GiB / 0 free] Total: 1 [<63.98 GiB] / in use: 1 [<63.98 GiB] / in no VG: 0 [0 ] root@riverside:~# df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/riverside--vg-root 60G 56G 1.4G 98% /
(2% some cleanup seems to have occur since the creation of the task :) )
- disk extended on proxmox by 16Go on proxmox
(extract of dmesg of riverside) [350521.461023] sd 2:0:0:0: Capacity data has changed [350521.461339] sd 2:0:0:0: [sda] 167772160 512-byte logical blocks: (85.9 GB/80.0 GiB) [350521.461484] sda: detected capacity change from 68719476736 to 85899345920
- partition resized :
root@riverside:~# parted /dev/sda GNU Parted 3.2 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print free Model: QEMU QEMU HARDDISK (scsi) Disk /dev/sda: 85.9GB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags:
Dec 19 2020
I confirm that sentry is still happily processing events, more than 12 hours after the last upgrade :)
Dec 18 2020
After pushing through the updates up to 20.12.1, it seems the events are being processed correctly. I'm somewhat confident that the updated celery in the sentry image will not exhibit the same processing bug, but I'll keep an eye on the logs for a bit...
To look at the state of the celery queues, from https://docs.celeryproject.org/en/stable/userguide/monitoring.html#monitoring-redis-queues:
Looks like the events are/were getting stuck at the celery stage.
Thanks for the thorough investigation so far!
To eliminate another possible root cause, a test was done in a temporary project with the last version of the python library, it doesn't work either
Dec 17 2020
we have followed the event track on the consumer code without finding anything suspicious.
As a last try, we have fully rebooted the vm, but as expected, it changed nothing at all.
@olasd, if you have some detailed of the version upgrades you have performed yesterday, perhaps it could help to diagnose.