In order to test the seagate replacement procedure, only one disk will be replaced through the manufacturer warranty :
/dev/sda / SLOT 00
Model Family: Seagate Enterprise Capacity 3.5 HDD
Device Model: ST6000NM0115-1YZ110
Serial Number: ZAD0S2NR
The disk was excluded from the zfs pool so it can be replaced without side effects
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T2939 Replace out of order disks on db1.staging and storage1.staging | ||
| Migrated | gitlab-migration | T3033 Replace first disk on storage1.staging |
The disk has been sent to the manufacturer this morning, we now have to wait to hear from him.
The disk was put back in place on the server.
The new information are:
=== START OF INFORMATION SECTION === Model Family: Seagate Enterprise Capacity 3.5 HDD Device Model: ST6000NM0115-1YZ110 Serial Number: ZADA82DQ LU WWN Device Id: 5 000c50 0c4be3956 Firmware Version: SN05 User Capacity: 6,001,175,126,016 bytes [6.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 044 Pre-fail Always - 6384 3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 1 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 253 045 Pre-fail Always - 417113 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 1 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 1 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 253 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 067 067 040 Old_age Always - 33 (Min/Max 20/33) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 0 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 2 194 Temperature_Celsius 0x0022 033 040 000 Old_age Always - 33 (0 20 0 0 0) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 6384 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 253 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1h+03m+24.293s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 0 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 6384
The led was still red and needed to be manually force to normal via the `ledctl` tool
root@storage1:~# apt install ledmon root@storage1:~# ledctl normal=/dev/sda
It's confirmed via the person on site it's ok.
A smartctl test is in progress:
root@storage1:~# smartctl -t long /dev/sda
To be sure the disk is ok as it seems there is a high count of Raw_Read_Error_Rate, a complete read/write test was launched. It seems it will take some times to complete:
root@storage1:~# badblocks -v -w -B -s -b 4096 /dev/sda Checking for bad blocks in read-write mode From block 0 to 1465130645 Testing with pattern 0xaa: 0.74% done, 3:16 elapsed. (0/0/0 errors)
The disk was tested completely with read/write operations (interrupted on the 2d pass)
root@storage1:~# badblocks -v -w -B -s -b 4096 /dev/sda Checking for bad blocks in read-write mode From block 0 to 1465130645 Testing with pattern 0xaa: 0.74% done, 3:16 elapsed. (0/0/0 errors) badblocks -v -w -B -s -b 4096 /dev/sda done Reading and comparing: 100% done, 16:30:03 elapsed. (0/0/0 errors) done Testing with pattern 0x55: ^C4.04% done, 21:10:33 elapsed. (0/0/0 errors) Interrupted at block 59217472
No errors were detected by smartctl:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 083 064 044 Pre-fail Always - 184990704 3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 1 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 070 060 045 Pre-fail Always - 10656750 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 27 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 1 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 068 067 040 Old_age Always - 32 (Min/Max 20/33) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 698 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 0 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 2 194 Temperature_Celsius 0x0022 032 040 000 Old_age Always - 32 (0 20 0 0 0) 195 Hardware_ECC_Recovered 0x001a 083 064 000 Old_age Always - 184990704 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 27h+42m+08.630s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 12194785456 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 13449334312
The disk can be reintroduced on the zfs pool as a spare:
root@storage1:~# zpool add data spare /dev/disk/by-id/wwn-0x5000c500c4be3956 root@storage1:~# zpool status -v data pool: data state: ONLINE scan: scrub repaired 0B in 0 days 04:36:18 with 0 errors on Sun Feb 14 05:00:21 2021 config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 wwn-0x5000c500a22eef5e ONLINE 0 0 0 wwn-0x5000c500a23e85cf ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 wwn-0x5000c500a23e3868 ONLINE 0 0 0 wwn-0x5000c500a22eed6f ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 wwn-0x5000c500a22f48c9 ONLINE 0 0 0 wwn-0x5000c500a22ef2c4 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 wwn-0x5000c500a23e7af4 ONLINE 0 0 0 wwn-0x5000c500a23d253b ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 wwn-0x5000c500a23cf9ba ONLINE 0 0 0 wwn-0x5000c500a23e4511 ONLINE 0 0 0 cache nvme-INTEL_SSDPED1K375GAQ_FUKS70860038375AGN ONLINE 0 0 0 spares wwn-0x5000c500a23d19b6 AVAIL wwn-0x5000c500c4be3956 AVAIL errors: No known data errors
