{"id":656,"date":"2021-06-27T14:46:49","date_gmt":"2021-06-27T20:46:49","guid":{"rendered":"http:\/\/lucasmanual.com\/blog\/?p=656"},"modified":"2021-06-27T20:27:53","modified_gmt":"2021-06-28T02:27:53","slug":"gpt-mdadm-debian-stable-failed-drive-replacement-howto","status":"publish","type":"post","link":"https:\/\/lucasmanual.com\/blog\/gpt-mdadm-debian-stable-failed-drive-replacement-howto\/","title":{"rendered":"gpt\/mdadm\/debian stable\/failed drive replacement howto"},"content":{"rendered":"\n

It’s this time again. Last time in 2012<\/a>. Fast forward to 2021. Here we are again. This time it a little different. The drive has not failed yet but it shows signs of failures.<\/p>\n\n\n\n

smartctrl says it cannot read few sectors.<\/p>\n\n\n\n

Waring: One thing you learn over many years working with computers, servers, etc, is that you CANNOT ignore hardware failures. They will bite you back if you think you can leave it for few extra day(s). Mine, as well as your policy should be: if you get a warning that something is wrong, you need to act. If you work for a business that means you ship the new drive overnight. No excuses should be allowed in this regard.
<\/p>\n\n\n\n

With that hardware failure policy you and your business has better chances.
<\/p>\n\n\n\n

Debian Stable.<\/strong><\/p>\n\n\n\n

Install gdisk <\/strong>
aptitude install gdisk
<\/code>
Show details of partition md0
mdadm --detail \/dev\/md0<\/code><\/pre>\n\n\n\n
*2021 drive failing.
cat \/proc\/mdstat 
 Personalities : [raid6] [raid5] [raid4] 
 md0 : active raid5 sdb1[4] sda1[3] sdc1[1]
       3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3\/3] [UUU]<\/pre>\n\n\n\n
**Note my 2012 failure was this:<\/code>
cat \/proc\/mdstat
 Personalities : [raid6] [raid5] [raid4]
 md0 : active raid5 sdc1[1] sdd1[2]
       3907028864 blocks level 5, 64k chunk, algorithm 2 [3\/2] [_UU]<\/code><\/pre>\n\n\n\n
Since our drive did not fail yet but will soon, we will mark it as failed.<\/p>\n\n\n\n
mdadm \/dev\/md0 -f \/dev\/sdb1
\nmdadm: set \/dev\/sdb1 faulty in \/dev\/md0<\/pre>\n\n\n\n
If we didn’t fail it we would get this error if we tried to remove it in next step:<\/p>\n\n\n\n
mdadm \/dev\/md0 -r \/dev\/sdb1
\nmdadm: hot remove failed for \/dev\/sdb1: Device or resource busy<\/pre>\n\n\n\n

Lets remove the drive from mdadm: (not if you don’t know if ifs sdb1 you can run lsblk to confim)<\/p>\n\n\n\n
mdadm \/dev\/md0 -r \/dev\/sdb1
 mdadm: hot removed \/dev\/sdb1 from \/dev\/md0<\/pre>\n\n\n\n
#We can see our mdadm now shows the drive missing
cat \/proc\/mdstat 
 Personalities : [raid6] [raid5] [raid4] 
 md0 : active raid5 sda1[3] sdc1[1]
       3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3\/2] [_UU]<\/pre>\n\n\n\n
 <\/code>
 SHUT DOWN IF YOU NEED TO REPALCE DRIVE.  <\/strong><\/p>\n\n\n\n
If you are putting it in a same slot you should have it mark as same name, but we need to make sure. If the drive name changed you would not want to be making partitions changes on a wrong drive. MAKE SURE NEW DRIVE IS STILL sdb.<\/strong>
Look how disk is structured and what partition type it has <\/strong>
sgdisk -p  \/dev\/sdb
 Disk \/dev\/sdb: 3907029168 sectors, 1.8 TiB
 Logical sector size: 512 bytes
 Disk identifier (GUID): 0ED13F81-6EEA-4E12-9F27-DD806CF1F09C
 Partition table holds up to 128 entries
 First usable sector is 34, last usable sector is 3907029134
 Partitions will be aligned on 8-sector boundaries
 Total free space is 0 sectors (0 bytes)<\/code><\/p>\n\n\n\n
#sgdisk -R=\/dev\/TO_THIS_DISK \/dev\/FROM_THIS_DISK<\/strong>
sgdisk -R=\/dev\/sdb \/dev\/sda
\n#Give new GUID since above options clones the disk including GUID<\/strong>
\nsgdisk -G \/dev\/sdb<\/code><\/p>\n\n\n\n
Now readd the drive to md0 <\/strong>
 mdadm \/dev\/md0 -a \/dev\/sdb1
<\/code>

Check the status <\/strong><\/pre>\n\n\n\n
cat \/proc\/mdstat 
 Personalities : [raid6] [raid5] [raid4] 
 md0 : active raid5 sdd1[4] sda1[3] sdc1[1]
       3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3\/2] [_UU]
       [>\u2026\u2026\u2026\u2026\u2026\u2026..]  recovery =  0.0% (253788\/1953511936) finish=384.8min speed=84596K\/sec

#....few minutes later

cat \/proc\/mdstat 
 Personalities : [raid6] [raid5] [raid4] 
 md0 : active raid5 sdd1[4] sda1[3] sdc1[1]
       3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3\/2] [_UU]
       [=>\u2026\u2026\u2026\u2026\u2026\u2026.]  recovery =  7.4% (145761912\/1953511936) finish=391.5min speed=76950K\/sec<\/pre>\n\n\n\n

Done. Check back in few hours to see if it finished.<\/strong>
Keywords: fdisk,sdisk, sgdisk, gdisk,parted,gpt, mbr,raid5,mdadm,linux,debian,business, dell,hp,server,policy finish=786.3min speed=41401K\/sec<\/code>
Done. Check back in few hours to see if it finished.<\/strong>
Keywords: fdisk,sdisk, sgdisk, gdisk,parted,gpt, mbr,raid5,mdadm,linux,debian,<\/p>\n","protected":false},"excerpt":{"rendered":"
It’s this time again. Last time in 2012. Fast forward to 2021. Here we are again. This time it a little different. The drive has not failed yet but it shows signs of failures. smartctrl says it cannot read few sectors. Waring: One thing you learn over many years working with computers, servers, etc, is… Continue reading gpt\/mdadm\/debian stable\/failed drive replacement howto<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,3,34,27,4,33],"tags":[39,25,37,26,36,35,38],"class_list":["post-656","post","type-post","status-publish","format-standard","hentry","category-corporate","category-debian","category-hardware","category-it-department","category-linux","category-policy","tag-business","tag-debian","tag-mdadm","tag-nvme","tag-sda","tag-sdb","tag-stable","entry"],"_links":{"self":[{"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/posts\/656","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/comments?post=656"}],"version-history":[{"count":2,"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/posts\/656\/revisions"}],"predecessor-version":[{"id":659,"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/posts\/656\/revisions\/659"}],"wp:attachment":[{"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/media?parent=656"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/categories?post=656"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lucasmanual.com\/blog\/wp-json\/wp\/v2\/tags?post=656"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}