Linux Crumbs
Simbiotic - A Cyberpunk Thriller Novel

Setting Up RAID and Migrating MythTV Recordings

Summary

This tutorial details the process of setting up RAID on a Mythbuntu 12.04 installation and then migrating the existing MythTV recordings to the RAID.

This tutorial is the third in a mini series that started with Why Consider RAID for PVR Recordings and Solid State Drive for Mythbuntu Operating System, and proceeded to Migrating Existing MythTV 0.24 on HDD to MythTV 0.25 on SSD.

The original series of articles that outline the path of creating a custom-built PVR can be found at the following links:

Contents

Strategy for Migrating MythTV Recordings to RAID

At this point in this RAID mini series, Mythbuntu 12.04 is installed and running from a new separate drive (SSD), and the original hard drive contains the active MythTV recordings and a no-longer used Mythbuntu 11.04. The strategy is to install a new hard drive, configure Mythbuntu to work with Linux Software RAID, build half of the RAID, migrate the MythTV recordings to the new RAID (half), setup RAID monitoring, then finish up by adding the original drive to the RAID.

Note that when building a RAID, it is important to have identically sized storage areas, and drives with the same sector size. To place the storage area size under my control, I recommend partitioning the drive and then creating the RAID using a partition. Moreover I think it is a good idea to partition all devices to ensure that partition editors do not view the drive as unpartitioned and hence empty, which might result in accidentally overwriting the drive.

TIP:   Perform Steps When Not Recording Shows

The RAID setup and PVR recordings migration process can take several hours and during this time the disk drives will be very active. This high level of activity might adversely impact the recording of TV shows. Hence it is advisable to choose a block of time in which your PVR has no scheduled recordings.

Now onto the setting up the RAID migrating the PVR recordings.

Install New Hard Drive (HDD) in PVR

In my computer I plugged the drives into the SATA ports in the following order:

  • /dev/sda - new SSD for Mythbuntu 12.04 OS.
  • /dev/sdb - original HDD containing Mythbuntu (11.04) and recordings.
  • /dev/sdc - new HDD for mirrored RAID storage of recordings.
  • /dev/sr0 - original DVD R/W optical drive.

Following is a picture of my PVR with the SSD (middle left) and two HDDs (lower right) installed.

MythTV PVR with SSD and two HDDs installed

Install and Configure Packages for Linux Software RAID

  1. Install Linux Software RAID package.

    sudo apt-get install mdadm

    TIP:   Use TAB or ARROWS to navigate prompts, and ENTER to accept.

    1. At first Package Configuration screen:
      TAB to "Ok" and press ENTER.
    2. At second Package Configuration screen:
      ARROW UP to "No configuration", and TAB to "Ok" and press ENTER.

  2. Fix problem that Ubuntu 12.04 can't reliably boot with SW RAID.

    1. Edit /usr/share/initramfs-tools/scripts/mdadm-functions.

      sudo leafpad /usr/share/initramfs-tools/scripts/mdadm-functions
      Add the "udevadm settle" line as indicated below:

      {
        udevadm settle
        mdadm --misc --scan --detail --test >/dev/null 2>&1
        return $((! $?))
      }
    2. Rebuild the current initramfs.

      sudo update-initramfs -u
  3. Ensure boot from degraded RAID

    1. Edit /etc/initramfs-tools/conf.d/mdadm.

      sudo leafpad /etc/initramfs-tools/conf.d/mdadm
      Change:

      BOOT_DEGRADED=false
      To:

      BOOT_DEGRADED=true

      NOTE:   RAID Monitoring Suggested

      In the default configuration, Mythbuntu will not boot if the RAID is in a degraded state (for example, one drive has failed). Hence when a drive in the RAID fails, we will become aware of this by Mythbuntu failing to fully boot up.

      Since we change the configuration to continue with boot even in a degraded state, we need another way to know when there are problems with the RAID. In a later section we describe how to add basic RAID monitoring.


    2. Rebuild the current initramfs.

      sudo update-initramfs -u
  4. Install gparted graphical partitioning software.

    sudo apt-get install gparted

    NOTE:   Mythbuntu 12.04 uses older GParted 0.11.0

    This older version of GParted does not properly detect Linux Software RAID drives. As such to partition the RAID you will need to pass the RAID name. For example:

    sudo gparted /dev/md0

Setup New Hard Drive as Half of RAID Level 1 Mirror

The Linux Software RAID commands are very flexible. These commands support building a complete two drive RAID mirror directly. Further the commands also support building one half of the RAID mirror (a RAID in a degraded state), and then later adding the other half of the RAID mirror. Since I needed to migrate my existing recordings to the RAID I used the latter method. In these instructions I will also point out how you could build the RAID mirror directly with both drives in one command.

In this example I have the following three drives:

  • /dev/sda - new SSD for Mythbuntu 12.04 OS.
  • /dev/sdb - original 1 TB HDD containing recordings and inactive Mythbuntu 11.04.
  • /dev/sdc - new 1 TB HDD for mirrored RAID storage of recordings.
  1. Add the additional hard drive to your PVR.

    Ideally the hard drives should be the same size. This is not critical though as we will use partitions to define the size of the RAID. Hence a 2 TB drive could be configured with a smaller 1 TB partition so that it could match another 1 TB drive with a 1 TB partition.
  2. Partition the new drive (For example: /dev/sdc).

    Run GParted from a terminal window using the following command:

    sudo gparted
    1. Select the /dev/sdc device using the upper right-hand-side drop-down list.
    2. Use Device -> Create Partition Table to write an msdos (MBR) partition table.
    3. Select the unallocated space.
    4. Use Partition -> New to create one new unformatted primary partition spanning the drive.
      NOTE: On my 1 TB drive this is 931.51 GiB or 953,869 MiB.
    5. Use Edit -> Apply All Operations to apply this operation.
    6. Close the Applying pending operations window.
    7. Use Partition -> Manage Flags to set "raid" flag.
    8. Exit gparted.
  3. Create half of the RAID level 1 mirror (degraded RAID) on the new drive partition (For example: /dev/sdc1).

    sudo mdadm --create /dev/md0 --auto=p --level=1 \
      --raid-devices=2 missing /dev/sdc1

    NOTE:   How-to set up both drives in the mirrored RAID at once (For example: sdb1 and sdc1)

    To set up both drives at once, ensure that both drives contain the exact same size partitions and use a command such as:

    sudo mdadm --create /dev/md0 --auto=p --level=1 \
      --raid-devices=2 /dev/sdb1 /dev/sdc1
    In my experience creating only half of a RAID level 1 is quick because the step to mirror the RAID from one drive to another has been postponed until later.

    When adding the second 1 TB drive in the RAID level 1 mirror, it took 3 hours to mirror the RAID from one drive to another.

    You can use the watch command to frequently check the progress of building the RAID:

    watch -n 60 cat /proc/mdstat
    Use Ctrl+C to stop watching the progress.


  4. Partition the (degraded) RAID.

    sudo gparted /dev/md0
    1. Use Device -> Create Partition Table to write an msdos (MBR) partition table.
    2. Select the unallocated space.
    3. Use Partition -> New to create one new ext4 primary partition spanning the drive with label "data".
      NOTE: On my 1 TB partitioned drive this is 953,738 MiB.
    4. Use Edit -> Apply All Operations to apply this operation.
    5. Close the Applying pending operations window.
    6. Exit gparted.
  5. Update /etc/mdadm/mdadm.conf to re-assemble RAID with name md0 on boot.
    1. Add the RAID configuration to the end of the /etc/mdadm/mdadm.conf file with the following command:

      sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf
      This should add a line like the following to the end of mdadm.conf:

      ARRAY /dev/md/0 metadata=1.2 UUID=8d8a88cc:e239f398:8c202a01:d3750e45 name=mythpvr:0
      NOTE: If you run this command multiple times, then you will need to edit /etc/mdadm/mdadm.conf to remove the duplicate entries.
    2. Rebuild the current initramfs.

      sudo update-initramfs -u

Copy Recordings to RAID

  1. Copy recordings from source partition to new (degraded) RAID level 1 partition.

    In this example I assume that your recordings and such are stored in /data/mythtv/.... If not, feel free to substitute your own mount points where appropriate.

    1. Mount the RAID partition /dev/md0p1 at /mnt.

      sudo mount /dev/md0p1> /mnt
    2. Copy the mythtv data to the RAID.

      sudo cp -p -r -v /data/mythtv /mnt
      NOTE: Depending on the amount of recordings you have this might take a long time.
    3. Unmount the RAID partition from /mnt.

      sudo umount /mnt
  2. Change to mount new RAID partition at /data

    1. Unmount /data.

      sudo umount /data
    2. Edit /etc/fstab entry to mount /data from new RAID

      sudo leafpad /etc/fstab
      Change existing /data mount entry from something like the following:
      # /data on /dev/sda3
UUID=2f02b4a8-acfa-4a09-b809-73d1823d1d01 /data   ext4   defaults   0   2
      To an entry to mount the RAID partition like the following:
      # /data on /dev/md0p1
/dev/md0p1                                /data   ext4   defaults   0   2
    3. Test mounting the directory with:

      sudo mount -a

Setup RAID Monitoring [Optional]

At this point we have setup a degraded RAID which means that only half of the RAID mirror is operational. This is a great time to also setup and test RAID monitoring.

We desire RAID monitoring so that we will be notified when one of the RAID hard drives fails. Without RAID monitoring, a problem with RAID hard drive failure might go unnoticed until the last half of the RAID failed. At such a point all of the MythTV recordings would be lost.

In the following steps we setup a simple visual indication for when the RAID is running in a degraded state.

  1. Create script for mdadm to notify of RAID problems.

    1. Create new script file /usr/bin/mdadm-notify-mythtv.sh

      sudo leafpad /usr/bin/mdadm-notify-mythtv.sh
      and paste in the following contents:

      ---------- begin file /usr/bin/mdadm-notify-mythtv.sh ----------
      #!/bin/bash
#    Purpose:  To notify user of problems with Linux software RAID
#  Called by:  /etc/mdadm/mdadm.conf
# Parameters:  1 - Event, 2 - Device, 3 - Optional Component
#
#
# Write message to MythTV Frontend / MythWelcome
#
mythutil --message \
  --message_text="RAID Status :: $1 :: $2 :: $3" \
  --timeout=5 \
  --bcastaddr=127.0.0.1
#
# Write message to Desktop
#
notify-send 'RAID Status' "$1 :: $2 :: $3" \
  --expire-time=5000
      ---------- end file /usr/bin/mdadm-notify-mythtv.sh ----------
    2. Ensure script file is executable.

      sudo chmod a+x /usr/bin/mdadm-notify-mythtv.sh
  2. Add notification script to mdadm configuration.

    1. Edit /etc/mdadm/mdadm.conf:

      sudo leafpad /etc/mdadm/mdadm.conf
      add the following two lines:

      # notify if problems with RAID
      PROGRAM /usr/bin/mdadm-notify-mythtv.sh
      The resulting file should look something like the following:

      ---------- Begin example file mdadm.conf ----------
      # mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=8d8a88cc:e239f398:8c202a01:d3750e45 name=mythpvr:0

# notify if problems with RAID
PROGRAM /usr/bin/mdadm-notify-mythtv.sh

# This file was auto-generated on Thu, 11 Apr 2013 12:58:58 -0600
# by mkconf $Id$
      ---------- Begin example file mdadm.conf ----------
    2. Rebuild the current initramfs:

      sudo update-initramfs -u
    3. Test notification with:

      sudo mdadm --monitor --scan --oneshot --test 2>/dev/null
      When I ran my tests, I was only able to see a visual indication of any RAID problems viewing the XFCE desktop. While the MythTV Frontend was running, or when viewing a recording, I did not see a visual indication of RAID problems until after I exited the MythTV Frontend.
  3. Create a root crontab file to automatically have mdadm monitor the RAID at a time when you usually watch TV.
    I choose 8:35pm or 20:35 in 24 hour time format.

    1. Start up the crontab file editor using the leafpad editor.

      sudo sh -c "export VISUAL=leafpad && crontab -e"
    2. Add and save the following three lines at the end of the file:

      ---------- Begin three crontab lines ----------
      #
# Monitor mdadm RAID and notify if problems
35 20 * * * /sbin/mdadm --monitor --scan --oneshot 2>/dev/null
      ---------- End three crontab lines ----------
    3. Confirm the crontab file by listing the contents.

      sudo crontab -l

Add Remaining Half of RAID Level 1 Mirror

At this point the PVR is fully operational on a degraded RAID. The original drive is not used because we have migrated the Mythbuntu OS to an SSD, and migrated the recordings to the new HDD RAID. The final step is to repurpose the original hard disk drive for use as the other half of the RAID level 1 mirror.

CAUTION:   Original Hard Drive WILL BE OVERWRITTEN!

This final step will overwite all of the data that was on the original drive. If you have been following along with these PVR articles then at this time the original hard drive contains an inactive Mythbuntu OS, already migrated PVR recordings, and a linux-swap partition.

If you have any other important data on the drive, then be sure to backup this data prior to repurposing this drive for use as the other half of the RAID level 1 mirror.

  1. Repartition the original drive (For example: /dev/sdb).

    Run GParted from a terminal window using the following command:

    sudo gparted
    1. Select the /dev/sdb device using the upper right-hand-side drop-down list.
    2. Use Device -> Create Partition Table to write a new empty msdos (MBR) partition table.
    3. Select the unallocated space.
    4. Use Partition -> New to create one new unformatted primary partition spanning the drive.
      NOTE: On my 1 TB drive this is 931.51 GB or 953,869 MiB.
    5. Use Edit -> Apply All Operations to apply this operation.
    6. Close the Applying pending operations window.
    7. Use Partition -> Manage Flags to set "raid" flag.
    8. Exit gparted.
  2. Add this newly created partition to complete both halves of the RAID level 1 mirror.

    sudo mdadm /dev/md0 -a /dev/sdb1

    NOTE:   Monitoring RAID Build Progress

    When adding the second 1 TB drive in the RAID level 1 mirror, it took 3 hours to mirror the RAID from one drive to another.

    You can use the watch command to frequently check the progress of building the RAID:

    watch -n 60 cat /proc/mdstat
    Use Ctrl+C to stop watching the progress.

    Note that there are methods suggested on the Internet to increase speed if necessary. Unfortunately I did not find any that worked for me.

RAID Migration Complete

Congratulations, you have now configured your MythTV PVR to store recordings on RAID level 1 mirrored hard drives. Now you can benefit from an easier recovery from a failed RAID drive without losing any recordings. :-)

Since I recently experienced problems with one of my RAID hard drives (after 995 days of operation), I plan to document how I discovered the problem and subsequently replaced the failing RAID drive.

References

While setting up RAID for the Mythbuntu MythTV PVR recordings, I found the following references useful:

Future Articles

In future articles, I plan to cover the following steps of our path:

The Medusa Deception novel - free first in series
News   Documentation     Copyright (c) 2013-2024 Curtis Gedak

Get GParted at SourceForge.net. Fast, secure and Free Open Source software downloads Valid HTML 4.01! Correct CSS!