The recovery of the server

Submitted by User Name on Wed, 01/10/2024 - 19:58
Forums
Sub-Forum

(with initially enabled borgbackup)

 

On the top of raid1.

root@grml ~ # mdadm --assemble --scan
mdadm: /dev/md/boot has been started with 2 drives.
mdadm: /dev/md/pv00 has been started with 2 drives.
root@grml ~ # lvs -a -o +devices
LV   VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices           home centos -wi------- 472.00m                                                     /dev/md126(118)   root centos -wi-------  <4.00g                                                     /dev/md126(236)   swap centos -wi------- 512.00m                                                     /dev/md126(1259)  tmp  centos -wi------- 472.00m                                                     /dev/md126(0) 

 

In case of complete disk backup it actually doesn't matter as steps will be very similar.

 

I assume you booted into rescue system.

 

1.1. Make sure to stop the raid devices

root@grml ~ # cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md126 : active (auto-read-only) raid1 sda2[0] sdb2[1]      
5729280 blocks super 1.2 [2/2] [UU]      
bitmap: 0/1 pages [0KB], 65536KB chunk
md127 : active (auto-read-only) raid1 sda1[0] sdb1[1]      
524288 blocks super 1.2 [2/2] [UU]      
bitmap: 0/1 pages [0KB], 65536KB chunk

 

So in my case I have to stop md126 and md127 using

root@grml ~ # mdadm --stop /dev/md126mdadm: stopped /dev/md126
root@grml ~ # mdadm --stop /dev/md127mdadm: stopped /dev/md127

 

1.2. Mount your backup storage

You would need to mount your backup storage to put the

 

1.3 Choose drive(s) to backup

As this is RAID1 setup and both drives ideally should have the same data ( which might not be true, but we just assume it is for now )

I have to choose from what drive I want to take the partitioning and data for my backup.

Of couse you can repeat the steps to clone the other drive aswell.

 

I choose to backup /dev/sda in this case.

root@grml ~ # borg create --read-special ssh://u123456atu123456.your-storagebox.de:23/./backups/server1::hostname-disk1 /dev/sda

 

And after a while it's done.

We have a backup of whole partition / filesystem.

 

Keep in mind you can recover on disks with the same size or larger ( easily, you might be able to loop mount and shrink ... another topic ) .

 

Ok nice, now let us assume we need to restore.

My disks show no signs of partitions or data:

root@grml ~ # fdisk -l /dev/sda
Disk /dev/sda: 8 GiB, 8589934592 bytes, 16777216 sectors
Disk model: VBOX HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@grml ~ # fdisk -l /dev/sdb
Disk /dev/sdb: 8 GiB, 8589934592 bytes, 16777216 sectors
Disk model: VBOX HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

 

2.1. Recover a disk from backup

root@grml ~ # borg extract --stdout ssh://u123456atu123456.your-storagebox.de:23/./backups/server1::hostname-disk1 | dd of=/dev/sda bs=10M

root@grml ~ # fdisk -l /dev/sda
Disk /dev/sda: 8 GiB, 8589934592 bytes, 16777216 sectors
Disk model: VBOX HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000819af
Device     Boot   Start      End  Sectors  Size Id Type

/dev/sda1  *       2048  1052671  1050624  513M fd Linux raid autodetect
/dev/sda2       1052672 12521471 11468800  5.5G fd Linux raid autodetect

 

nice we got something.

Now if you have the second drive aswell, you might restore that one and it probably would reassemble again.

Also I would recommend to check if the data extracted is valid, again another topic - up to you.

 

I assume we have a backup of one disk only, therefore we need to do a few more things.

The next steps might fail your recovery if the 2nd disk contains data / md super block still. If you

are absolutely sure you do not need any data from it, remove it. It can be done as easy as dd if=/dev/null of=/dev/sdb bs=1024 count=1

or similar.

 

2.2 Assemble Raid

root@grml ~ # mdadm --assemble --scan
mdadm: /dev/md/localhost:pv00 has been started with 1 drive (out of 2).
mdadm: /dev/md/localhost:boot has been started with 1 drive (out of 2).
root@grml ~ # cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md126 : active (auto-read-only) raid1 sda1[0]      
524288 blocks super 1.2 [2/1] [U_]      
bitmap: 0/1 pages [0KB], 65536KB chunk
md127 : active (auto-read-only) raid1 sda2[0]      
5729280 blocks super 1.2 [2/1] [U_]      
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>

 

 

Ok we have a Raid1 again with a single drive.

We can follow the steps as explained here.

 

2.3 Clone Partition table

 

As I have MBR format, have a working /dev/sda I'll use the following commands to clone the partition layout

from /dev/sda to /dev/sdb

root@grml ~ # sfdisk --dump /dev/sda > sda_parttable_mbr.bak
root@grml ~ # sfdisk /dev/sdb < sda_parttable_mbr.bak
Checking that no-one is using this disk right now ... OK
Disk /dev/sdb: 8 GiB, 8589934592 bytes, 16777216 sectors
Disk model: VBOX HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Created a new DOS disklabel with disk identifier 0x000819af.
/dev/sdb1: Created a new partition 1 of type 'Linux raid autodetect' and of size 513 MiB.
/dev/sdb2: Created a new partition 2 of type 'Linux raid autodetect' and of size 5.5 GiB.
Partition #2 contains a linux_raid_member signature.
/dev/sdb3: Done.
New situation:
Disklabel type: dosDisk
identifier: 0x000819af
Device     Boot   Start      End  Sectors  Size Id Type
/dev/sdb1  *       2048  1052671  1050624  513M fd Linux raid autodetect
/dev/sdb2       1052672 12521471 11468800  5.5G fd Linux raid autodetect
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

 

For GPT use the sgdisk command as explained in the linked wiki + change the GUID

 

2.4 Attach the missing disk

 

Now we have /dev/sdb back, we can attach it to the existing RAID again.

As my situation looks like this:

md126 : active raid1 sda1[0]
md127 : active raid1 sda2[0]

 

I'm going to attach sdb1 to md126 and sdb2 to md127

root@grml ~ #  mdadm /dev/md126 -a /dev/sdb1
mdadm: added /dev/sdb1
root@grml ~ #  mdadm /dev/md127 -a /dev/sdb2
mdadm: re-added /dev/sdb2
root@grml ~ # cat /proc/mdstat              
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md126 : active raid1 sdb1[2] sda1[0]
      524288 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md127 : active raid1 sdb2[1] sda2[0]
      5729280 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

In case the array did not rebuild for some reason, you might need to force it

echo "repair" > /sys/block/md126/md/sync_action
echo "repair" > /sys/block/md127/md/sync_action

Be sure the data on boths disks are in sync.

 

2.5 Mount the system

 

Discover your volume groups

root@grml ~ # vgscan --mknodes -v  
Found volume group "centos" using metadata type lvm2
root@grml ~ # vgchange -a y centos  
4 logical volume(s) in volume group "centos" now active
root@grml ~ # lvscan
ACTIVE            '/dev/centos/tmp' [472.00 MiB] inherit
ACTIVE            '/dev/centos/home' [472.00 MiB] inherit
ACTIVE            '/dev/centos/root' [<4.00 GiB] inherit
ACTIVE            '/dev/centos/swap' [512.00 MiB] inherit

 

Mount accordingly.

root@grml ~ # mount /dev/centos/root /mnt
root@grml ~ # mount /dev/centos/home /mnt/home
root@grml ~ # mount /dev/centos/tmp /mnt/tmp
root@grml ~ # mount /dev/md126 /mnt/boot

 

I have my /boot outside the LVM thus I needed to use /dev/md126 in this case.

 

2.6 Prepare the chroot

 

Use the easy chroot-prepare script hetzner provides

chroot-prepare /mnt

 

OR

 

Do the following

root@grml ~ # mount --bind  /proc  /mnt/proc
root@grml ~ # mount --bind  /sys  /mnt/sys
root@grml ~ # mount --bind  /dev  /mnt/dev
root@grml ~ # mount --bind  /run  /mnt/run

 

2.7 Updating grub

 

If disk IDs have changed it might be needed to update grub. It won't hurt to do it anyways.

 

chroot into your system:

root@grml ~ # chroot /mnt

 

Update grub config:

grub2-mkconfig -o /boot/grub2/grub.cfg

 

If you see something like

 

warning: Couldn't find physical volume `(null)'.

 

you can get rid of this by issuing the following command for each partition

blockdev --flushbufs /dev/sda1
blockdev --flushbufs /dev/sda2
blockdev --flushbufs /dev/sdb1
blockdev --flushbufs /dev/sdb2

 

and repeat the mkconfig command.

 

WARNING: Failed to connect to lvmetad. Falling back to device scanning.

Can be ignored though.

 

2.8 Install grub

 

I like to have grub on both drives so I do

[root@grml /]# grub2-install /dev/sda
Installing for i386-pc platform.
Installation finished.
No error reported.
[root@grml /]# grub2-install /dev/sdb
Installing for i386-pc platform.
Installation finished.
No error reported.
[root@grml /]#

 

2.9 Check your fstab and mdadm.conf

 

Verify your /etc/fstab is on par with your blkids - as you might have UUID entries in your fstab.

[root@grml /]# cat /etc/fstab
## /etc/fstab
# Created by anaconda on Fri Oct 22 22:52:44 2021
## Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=77f5e2a2-c21d-4578-84d5-d4e4e83ce715 /boot                   xfs     defaults        0 0
/dev/mapper/centos-home /home                   xfs     defaults        0 0
/dev/mapper/centos-tmp  /tmp                    xfs     defaults        0 0
/dev/mapper/centos-swap swap                    swap    defaults        0 0

[root@grml /]# blkid
/dev/sr0: UUID="2021-07-25-15-59-45-00" LABEL="grml64-full 2021.07" TYPE="iso9660" PTTYPE="dos"
/dev/loop0: TYPE="squashfs"
/dev/sdb1: UUID="837c322d-6eca-4f9d-78cb-39a45dda4c18" UUID_SUB="1472e7e4-ddf3-4bb0-6e1c-847542be4588" LABEL="localhost:boot" TYPE="linux_raid_member"
/dev/sdb2: UUID="d20bf3bc-db7b-93c7-8c9a-945417b88b39" UUID_SUB="af7fb02e-0d66-e927-6159-0dd28cd5a9a4" LABEL="localhost:pv00" TYPE="linux_raid_member"
/dev/sda1: UUID="837c322d-6eca-4f9d-78cb-39a45dda4c18" UUID_SUB="e1a133b9-d0b1-d221-bf18-058136f19566" LABEL="localhost:boot" TYPE="linux_raid_member"
/dev/sda2: UUID="d20bf3bc-db7b-93c7-8c9a-945417b88b39" UUID_SUB="3bbf87ed-b242-043b-e298-491ff6a82174" LABEL="localhost:pv00" TYPE="linux_raid_member"
/dev/md127: UUID="BOwc0f-kD4o-ev3u-lSQ6-f3kW-BbG5-2SAR84" TYPE="LVM2_member"
/dev/md126: UUID="77f5e2a2-c21d-4578-84d5-d4e4e83ce715" TYPE="xfs"
/dev/mapper/centos-tmp: UUID="8ddd7de3-9383-4ead-aa10-76f66daa77b1" TYPE="xfs"
/dev/mapper/centos-home: UUID="948e590f-755e-4d35-9930-8757838b7591" TYPE="xfs"
/dev/mapper/centos-root: UUID="44502232-11cc-4f18-a87c-cb09ded426e2" TYPE="xfs"
/dev/mapper/centos-swap: UUID="530b345f-a26c-4b00-9351-b7dddfbd9b58" TYPE="swap"

 

[root@grml /]# cat /etc/mdadm.conf
# mdadm.conf written out by anacondaMAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md/boot level=raid1 num-devices=2 UUID=837c322d:6eca4f9d:78cb39a4:5dda4c18
ARRAY /dev/md/pv00 level=raid1 num-devices=2 UUID=d20bf3bc:db7b93c7:8c9a9454:17b88b39

 

Looks good for me.

Otherwise fix it and regenerate your initramfs.

 

dracut -f is your friend - not mine ;)

 

2.10 Unmount everything and reboot

 

Crucial steps are done now. Lets clean up and leave.

[root@grml /]# exit
exit
chroot /mnt /bin/bash  3.97s user 2.96s system 0% cpu 12:02.47 total
root@grml ~ # umount /mnt/dev
root@grml ~ # umount /mnt/run
root@grml ~ # umount /mnt/proc
root@grml ~ # umount /mnt/sys
root@grml ~ # umount /mnt/boot
root@grml ~ # umount /mnt/home
root@grml ~ # umount /mnt/tmp
root@grml ~ # umount /mnt
root@grml ~ # reboot   

 

 

  • 24 views