• before :

root@riverside:~# pvscan
  PV /dev/sda1   VG riverside-vg    lvm2 [<63.98 GiB / 0    free]
  Total: 1 [<63.98 GiB] / in use: 1 [<63.98 GiB] / in no VG: 0 [0   ]
root@riverside:~# df -h /
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/riverside--vg-root   60G   56G  1.4G  98% /

(2% some cleanup seems to have occur since the creation of the task :) )

• disk extended on proxmox by 16Go on proxmox

(extract of dmesg of riverside)
[350521.461023] sd 2:0:0:0: Capacity data has changed
[350521.461339] sd 2:0:0:0: [sda] 167772160 512-byte logical blocks: (85.9 GB/80.0 GiB)
[350521.461484] sda: detected capacity change from 68719476736 to 85899345920

• partition resized :

root@riverside:~# parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print free                                                       
Model: QEMU QEMU HARDDISK (scsi)
Disk /dev/sda: 85.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Remove out of scope changes

A user was correctly configured and a read test performed :

The network configuration is done. The server is now accessible from the internet at broker0.journal.staging.swh.network:9093

LGTM

lgtm, I have a doubt on the numa activation but as it's also activated for kelvingrove, I assume it's correct

Changing to high priority (@ardumont recommandation)

The request to expose the journal to internet was done this afternoon to the dsi.

To eliminate another possible root cause, a test was done in a temporary project with the last version of the python library, it doesn't work either

lgtm

we have followed the event track on the consumer code without finding anything suspicious.
As a last try, we have fully rebooted the vm, but as expected, it changed nothing at all.

@olasd, if you have some detailed of the version upgrades you have performed yesterday, perhaps it could help to diagnose.

LGTM

After one week, the disk used by kafka was around 85% of usage

root@journal0:/tmp# df -h /srv/kafka/logdir
Filesystem      Size  Used Avail Use% Mounted on
kafka-volume    481G  409G   73G  85% /srv/kafka/logdir

Compared to the production, the compression was not activated on the zfs pool:

root@kafka1:~#  zfs get all data/kafka  | grep compress
data/kafka  compressratio         1.55x                  -
data/kafka  compression           lz4                    inherited from data
data/kafka  refcompressratio      1.55x                  -

root@journal0:/tmp# zfs get all  | grep compress
kafka-volume  compressratio         1.00x                  -
kafka-volume  compression           off                    default
kafka-volume  refcompressratio      1.00x                  -

So the compression was activated :

root@journal0:/tmp# zfs set compression=lz4 kafka-volume
root@journal0:/tmp# zfs get all  | grep compress
kafka-volume  compressratio         1.00x                  -
kafka-volume  compression           lz4                    local
kafka-volume  refcompressratio      1.00x                  -

As this parameter is only used for the new written data, we have force a compact on the biggest topics : `directory, revision and content`

 % ./kafka-topics.sh --zookeeper $ZK  --alter --topic swh.journal.objects.revision --config min.cleanable.dirty.ratio=0.01
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
         Going forward, please use kafka-configs.sh for this functionality
Updated config for topic swh.journal.objects.revision.
vsellier@journal0 /opt/kafka/bin
 % ./kafka-topics.sh --zookeeper $ZK  --alter --topic swh.journal.objects_privileged.revision --config min.cleanable.dirty.ratio=0.01
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
         Going forward, please use kafka-configs.sh for this functionality
Updated config for topic swh.journal.objects_privileged.revision.

LGTM

changing the status to resolved as everything looks good \o/

After a new test, a vm deployed on pompidou can reach the network without any issue.
There were some glitches (kernel dump) after the migration, perhaps a reboot after the first migration test would have fixed to network problem.

We should also check with @olasd if the zookeeper[1-3] can be decommissioned if we remove this brokers

LGTM
as tested together, removing the deep option of lookup seems to result in a more predictable behavior :)

In T2888#55002, @vsellier wrote:

• smartctl extended test are running on all the esnode* disks to detect possible defects. The results will be availble in few hours

All the smartctl tests are done and no additional faulty disks were detected

Remark regarding the extension of the storage via the addition of a new data directory [1], so not sure it's the best way to do it:

• smartctl extended test are running on all the esnode* disks to detect possible defects. The results will be availble in few hours

We tried to temporarily restart esnode1 to reallocate the shards of the red indices for which esnode1 was the primary.
Actions:

• Mount/Remount the xfs partition to flush the xfs journal
• Perform a xfs_repair to ensure the fs is ok
• configure elasticsearch deallocate the shard managed by esnode1
• start esnode1
• wait for the shards redistribution (swh_workers-2020.09.03was quickly recovered, and the remaining systemlogs.2018 deleted)
• stop esnode1
• disable puppet to avoid a restart of elasticsearch on esnode1

LGTM

The free disk space is again around ~85% used on esnode3 (~79% on esnode2).
The systemlogs.*2020.01.* indices were removed.

LGTM

The shard allocation is reactivated, it should have enough free disk space to replicate all the shard on the 2 nodes.

1. cleanup of systemlogs index before 2020 (2018/2019)

Short term plan :

• Remove old systemlogs indexes older than 1year to start, but we can go to 3 months if necessary
• reactivate the shard allocation to have 1 replica for all the shards in case of a second node failure
• Launch a long smartcl test on all the disks of each esnode* server
• Contact DELL support to proceed to the replacement of the 2 failing disks (under warranty(?)) [1]
• Try to recover the 16 red indexes if possible, if not, delete them as they are not critical

xfs has shutdown the partition so ES is lost .

It seems there is a quite limited indices impacted by the corruption :

❯ curl -s  http://${ES_NODE}/_cat/indices | grep red                                                                                                        22:20:27
red    open  systemlogs-2020.08.30               o_gpFSjQRBuQBvWqaqA_dA 1 1                                   
red    open  systemlogs-2020.08.27               U4fKujQhTXmbGsx7zzLiPw 1 1                                   
red    open  systemlogs-2020.08.28               JVz-yhe4SeSow1TQPT61Jg 1 1                                   
red    open  systemlogs-2020.08.29               6avrSP3bRW2ZiwSlTpN0tA 1 1                                   
red    open  systemlogs-2020.08.22               jY7nPiXDS6a6aBnTDHNd1A 1 1                                   
red    open  systemlogs-2020.08.16               AK8wyDFQQ2KOgbzIdLvPqQ 1 1                                   
red    open  systemlogs-2020.08.13               o6OowHj-TMCBSglETaTj4w 1 1                                   
red    open  systemlogs-2020.08.10               NN0H_eaXQJW_20lsIMmg0Q 1 1                                   
red    open  systemlogs-2020.08.08               pkJVICAdSbqn3JgHU1h5Yw 1 1                                   
red    open  systemlogs-2020.09.07               naRyJEkZRCeOY5h_2avRyg 1 1                                   
red    open  systemlogs-2020.09.03               wb0DMaeqT2-Lh4nx8rafgQ 1 1                                   
red    open  systemlogs-2020.09.01               jelq1Ij5SGWQAKDqdbCYlQ 1 1                                   
red    open  swh_workers-2020.09.03              c1ZiRR8HS9W44T3nVd7f9Q 2 1  2733325        0   1.6gb    1.6gb
red    open  systemlogs-2020.07.24               743a1usWSw-whONPLhcKrA 1 1                                   
red    open  systemlogs-2020.07.25               zFkfn6l5SA-sby3A0SOAtw 1 1                                   
red    open  systemlogs-2020.07.17               PxL7sBrUQ8SXtbOEG5v_3A 1 1

sdb and sdc on esnode1 have serious issues.
(there is no other disks with errors on other servers)

root@esnode1:~# smartctl -a /dev/sdb
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-13-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

LGTM no more apt install dstat \o/ :P

With the "optimized" configuration, the import is quite faster :

root@search-esnode0:~# curl -XPOST -H "Content-Type: application/json" http://${ES_SERVER}/_reindex\?pretty\&refresh=true\&requests_per_second=-1\&\&wait_for_completion=true -d @/tmp/reindex-production.json    
{
  "took" : 10215280,
  "timed_out" : false,
  "total" : 91517657,
  "updated" : 0,
  "created" : 91517657,
  "deleted" : 0,
  "batches" : 91518,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [ ]
}

"took" : 10215280, => 2h45

• diff landed and applied on the server
• VIP 128.93.166.40 configured on the firewall
• NAT Port forward of port 9093 from public ip to internal journal0 declared on the firewall
• DNS declaration of broker0.journal.staging.swh.network in gandi
• Ask to DSI to apply the kafka firewall profile to 128.93.166.40
• Configure a user to test the pipeline

rebase

I agree for the default site but we have several legit requests from the monitoring not correctly routed so the configuration needs to be adapted.

The production index origin was correctly copied from the production cluster but it seems without the configuration to optimize the copy.
We keep this one and try a new optimized copy to check if the server still crash in an OOM with the new cpu and memory settings.

FI: The origin index was recreated with the "official" mapping and a backfill was performed (necessary after the test of the flattened mapping)

The deployment manifest are ok and deployed in staging so this task can be resolved.
We will work on reactivating search-journal-client for the metadata in another task when T2876 is resolved

We tried to change the mapping type of the field intrinsic_metadata from nested to flattened as you have suggested, we have now a new error related to the huge size of a description.
ES can be configured to accept bigger fields but I'm not sure it's relevant regarding the description field content.

The copy of the production index is restarted.
To improve the speed of the copy, the index was tuned to reduce the disk pressure (it's a temporary configuration and should not be used in a normal case as it's not safe) :

cat >/tmp/config.json <<EOF
{
  "index" : {
    "translog.sync_interval" : "60s",
	"translog.durability": "async",
	"refresh_interval": "60s"
  }
}
EOF

• Parition and memory extended with terraform.
• The disk resize needed some console actions to be extended :

The production index import failed because the limit of 90% of used disk spaces was reached at some time to fall back to around 60G after a compaction
The progression was 80M documents of 91M.