CISS-150 Project 7 – RAID

Posted on By William Jojo
CISS-150-Project

CISS-150 Project 7 (10 points)

(Updated March 3, 2021)

Overview

This project will involve creating a software-based RAID-1 in Ubuntu using mdadm.

Learning outcomes

  • Planning and design.
  • Enhancing existing virtualization and storage skills.
  • Modifying the virtual environment.
  • Installing updates and application software.
  • Creating a RAID array.
  • Perform a RAID recovery.
STOP!

Before you proceed, it is STRONGLY recommended that you gracefully shutdown the Ubuntu VM and take a snapshot.

Remember that the snapshot will allow you to revert to a known good state. These are complex and detailed steps. If you make a mistake, you will want a way to recover and start fresh.

So, if you happen to mess up the creation of the RAID, you can simply revert the snapshot and the new hard disks will simply be dropped and you can start again from the beginning.

TAKE A SNAPSHOT NOW!

Current Configuration

You can get a peek of the current disk hardware by using some commands from Project 2. These are noted below:

To get a list of disks and partitions as block devices in /dev use:

student@student-vm:~$ ll /dev/sd*
brw-rw---- 1 root disk 8, 0 Dec 28 11:59 /dev/sda
brw-rw---- 1 root disk 8, 1 Dec 28 11:59 /dev/sda1
brw-rw---- 1 root disk 8, 2 Dec 28 11:59 /dev/sda2
brw-rw---- 1 root disk 8, 5 Dec 28 11:59 /dev/sda5

and to note the SCSI host adapter (host32 in this example):

student@student-vm:~$ ll /sys/block/sd*
lrwxrwxrwx 1 root root 0 Dec 28 11:58 /sys/block/sda -> ../devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda/

and to see the hierarchy of block devices:

student@student-vm:~$ lsblk | grep -v loop
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                 8:0    0    30G  0 disk 
├─sda1              8:1    0   512M  0 part /boot/efi
├─sda2              8:2    0     1K  0 part 
└─sda5              8:5    0  29.5G  0 part 
  └─vgubuntu-root 253:0    0  29.5G  0 lvm  /
sr0                11:0    1  1024M  0 rom

and finally, look at the current md (multiple devices) status.

student@student-vm:~$ cat /proc/mdstat 
Personalities : 
unused devices: <none>

All of this is simply to establish what we already have as disk hardware and the state of RAID.

There is only the one disk, /dev/sda, and there is no RAID yet, so all of this is as it should be for the moment.

Adding Disks

Important Note!
These next steps can be done either live or with the VM shutdown. Doing it live is more exciting and shows the full power of VMware and being able to hot-add devices to a running system.

At this point we will add the disks. This is done through VMware since we need to create the disk and then present it to the guest.

  1. Right click on your VM and select Edit Settings.
  2. Select the pull-down ADD NEW DEVICE.
  3. Select Hard Disk.
  4. Set the size to 5GB. (The default is most likely 16GB)
  5. Select OK.
  6. Repeat steps 2-5.
  7. Make sure both New Hard disk sizes are set to 5GB.
  8. Select OK.

Both of the hard disks will be presented to the guest, but will not be recognized until either the bus is scanned or a reboot.

To scan the bus, we simply have to send a string to a specific file. Below is a series of commands that shows the disks prior, the scanning of the SCSI bus, and the subsequent list thereafter.

Important Note!
We are becoming root for the remainder of this work.
student@student-vm:~$ sudo -i
[sudo] password for student:
root@student-vm:~# ll /dev/sd*
brw-rw---- 1 root disk 8, 0 Dec 28 13:11 /dev/sda
brw-rw---- 1 root disk 8, 1 Dec 28 13:11 /dev/sda1
brw-rw---- 1 root disk 8, 2 Dec 28 13:11 /dev/sda2
brw-rw---- 1 root disk 8, 5 Dec 28 13:11 /dev/sda5
root@student-vm:~# echo "- - -" >>/sys/class/scsi_host/host32/scan
root@student-vm:~# ll /dev/sd*
brw-rw---- 1 root disk 8,  0 Dec 28 13:11 /dev/sda
brw-rw---- 1 root disk 8,  1 Dec 28 13:11 /dev/sda1
brw-rw---- 1 root disk 8,  2 Dec 28 13:11 /dev/sda2
brw-rw---- 1 root disk 8,  5 Dec 28 13:11 /dev/sda5
brw-rw---- 1 root disk 8, 16 Dec 28 13:58 /dev/sdb
brw-rw---- 1 root disk 8, 32 Dec 28 13:58 /dev/sdc

Therefore we will be creating our raid with /dev/sdb and /dev/sdc. The /dev/sda disk and its partitions will remain unchanged.

Creating the RAID

I order to create the RAID, we will need to install some software to assist.

root@student-vm:~# apt update
root@student-vm:~# apt install mdadm

This software will help with the administration of the /dev/md0 device. This is going to be the RAID.

root@student-vm:~# mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: size set to 5237760K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
root@student-vm:~#

Now we will recheck the state of md.

root@student-vm:~# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdc[1] sdb[0]
      5237760 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
root@student-vm:~#

Creating the Filesystem

Now that the RAID is created, we can create the filesystem.

root@student-vm:~# mkfs.ext4 -F /dev/md0
mke2fs 1.45.5 (07-Jan-2020)
Creating filesystem with 1309440 4k blocks and 327680 inodes
Filesystem UUID: 98f8c9cc-43d5-47e5-81e4-46ebf8eb0692
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

Now that the filesystem is created, we can mount it. Remember that a filesystem needs as access point which we call the mount point. It helps to maintain the overall directory structure.

root@student-vm:~# mkdir -p /mnt/data
root@student-vm:~# mount /dev/md0 /mnt/data/
root@student-vm:~# df -h /mnt/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        4.9G   20M  4.6G   1% /mnt/data
root@student-vm:~#

To make sure the RAID will be managed on each boot, we must update the config.

root@student-vm:~# mdadm --detail --scan | tee -a /etc/mdadm/mdadm.conf 
ARRAY /dev/md/student-vm:0 metadata=1.2 name=student-vm:0 UUID=8bf6a379:78138c94:1fca4309:9441ca57

Finally, we will update the initramfs to make the RAID available early so that it can be mounted.

root@student-vm:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.8.0-44-generic
root@student-vm:~#

Create A File

At this point you can use the storage.

root@student-vm:~# touch /mnt/data/newfile
root@student-vm:~# ll /mnt/data
total 24
drwxr-xr-x 3 root root  4096 Mar  3 20:36 ./
drwxr-xr-x 3 root root  4096 Mar  3 20:36 ../
drwx------ 2 root root 16384 Mar  3 20:32 lost+found/
-rw-r--r-- 1 root root     0 Mar  3 20:36 newfile
root@student-vm:~#

This is fully usable storage available at every boot.

Performing a Recovery

In this section we will simulate a failure and remove a disk. Then recreate the disk and heal the RAID. If we have done our job, we will still have access to the data during this time and can reconstruct while still using the filesystem.

Remove A Disk

  1. Right click on your VM and select Edit Settings.
  2. Hover over one of the 5GB hard disks and move the cursor to the far right and select the X to remove the drive.
  3. Select Delete files from datastore.
  4. Select OK.

At this point the RAID should be broken. We can check with:

root@student-vm:~# mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Wed Mar  3 20:32:19 2021
        Raid Level : raid1
        Array Size : 5237760 (5.00 GiB 5.36 GB)
     Used Dev Size : 5237760 (5.00 GiB 5.36 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Wed Mar  3 20:37:06 2021
             State : clean 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : student-vm:0  (local to host student-vm)
              UUID : db793172:6e0d081a:0feb2a0b:7815b199
            Events : 17

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
root@student-vm:~#

But, nothing has changed yet. We need to write a change to the disk to make sure some data is synced between the drives.

Creating another file will do nicely. Note the additional call to the sync command.

root@student-vm:~# touch /mnt/data/afterbreak; sync

Once the changed are written to the disk things should look more broken.

root@student-vm:~# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdc[1] sdb[0](F)
      5237760 blocks super 1.2 [2/1] [_U]
      
unused devices: 
root@student-vm:~# mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Wed Mar  3 20:32:19 2021
        Raid Level : raid1
        Array Size : 5237760 (5.00 GiB 5.36 GB)
     Used Dev Size : 5237760 (5.00 GiB 5.36 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Wed Mar  3 20:38:41 2021
             State : clean, degraded 
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 1
     Spare Devices : 0

Consistency Policy : resync

              Name : student-vm:0  (local to host student-vm)
              UUID : db793172:6e0d081a:0feb2a0b:7815b199
            Events : 21

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       1       8       32        1      active sync   /dev/sdc

       0       8       16        -      faulty   /dev/sdb
root@student-vm:~#

So we can now remove the disk from the RAID. Be sure to specify the one that has actually failed for you.

root@student-vm:~# mdadm --manage /dev/md0 --remove /dev/sdb
mdadm: hot removed /dev/sdb from /dev/md0
root@student-vm:~# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdc[1]
      5237760 blocks super 1.2 [2/1] [_U]
      
unused devices: 
root@student-vm:~#

Now we just need to create a new disk. Follow the procedure from earlier to add exactly one 5GB hard disk.

root@student-vm:~# echo "- - -" >>/sys/class/scsi_host/host32/scan
root@student-vm:~# ll /dev/sd*
brw-rw---- 1 root disk 8,  0 Mar  3 20:26 /dev/sda
brw-rw---- 1 root disk 8,  1 Mar  3 20:26 /dev/sda1
brw-rw---- 1 root disk 8,  2 Mar  3 20:26 /dev/sda2
brw-rw---- 1 root disk 8,  5 Mar  3 20:26 /dev/sda5
brw-rw---- 1 root disk 8, 16 Mar  3 20:32 /dev/sdb
brw-rw---- 1 root disk 8, 32 Mar  3 20:32 /dev/sdc
root@student-vm:~#

Now we will copy the partition table to the newly replaced disk.

root@student-vm:~# sfdisk -d /dev/sdc | sfdisk /dev/sdb
sfdisk: /dev/sdc: does not contain a recognized partition table
Checking that no-one is using this disk right now ... OK

Disk /dev/sdb: 5 GiB, 5368709120 bytes, 10485760 sectors
Disk model: Virtual disk    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Done.

New situation:
root@student-vm:~#

Finally, we will add the disk back into the RAID so the recovery process may begin.

root@student-vm:~# mdadm --manage /dev/md0 --add /dev/sdb
mdadm: added /dev/sdb
root@student-vm:~# mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Wed Mar  3 20:32:19 2021
        Raid Level : raid1
        Array Size : 5237760 (5.00 GiB 5.36 GB)
     Used Dev Size : 5237760 (5.00 GiB 5.36 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Wed Mar  3 21:07:59 2021
             State : clean, degraded, recovering 
    Active Devices : 1
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : resync

    Rebuild Status : 68% complete

              Name : student-vm:0  (local to host student-vm)
              UUID : db793172:6e0d081a:0feb2a0b:7815b199
            Events : 37

    Number   Major   Minor   RaidDevice State
       2       8       16        0      spare rebuilding   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
root@student-vm:~# cat /proc/mdstat
Personalities : [raid1] 
md0 : active raid1 sdb[2] sdc[1]
      5237760 blocks super 1.2 [2/2] [UU]
      
unused devices: 
root@student-vm:~#

You may be lucky enough to catch the state of recovering, as noted above in the sample output.

At this point the RAID is completely rebuilt and everything it healthy.