Monitoring MythTV Disks to Alert of Pending Failure

Summary

This tutorial describes how to set up a script to monitor disk drives with the intention of alerting the user of impending failure. This early warning is intended to help you replace disk drives before complete failure.

These steps were documented using Mythbuntu 14.04. The steps should also work other GNU/Linux distros with minor changes, such as substituting apt-get with other package management tools like yum, dnf, or zypper.

Update 2021-04-20: Add script check for new replacement attribute Reallocated_Event_Count seen on Kingston SSD.

Background
Script Setup Steps
Script Monitoring Complete
References

Background

Disk failures over the past few years have helped me to discover a sign of impending failure. The experience has been gained with spinning hard disk drives from Seagate, but might be applicable to other manufacturers as well.

The key indication of impending failure is an increase in the raw value of the Reallocated_Sector_Count in smart disk monitoring. A non-zero value for Reallocated_Sector_Count indicates the disk surface has started to degrade. The number represents the sectors on the disk that have gone bad and are now mapped to a spare reserve of good disk sectors. Note that there is a limited pool of reserved good sectors, and it takes extra seek time to apply the sector mapping thus slowing down reads and writes to the disk drive.

In practice I've had successful MythTV recordings with Reallocated_Sector_Count values up to 600, but your experiences might vary. If MythTV recordings are important to you then I recommend replacing the disk drive earlier rather than later.

Script Setup Steps

The following commands are entered in a terminal window.

Ensure smart monitoring tools are installed.

sudo apt-get update && sudo apt-get install smartmontools

Create /usr/bin/mythtv-disk-monitor.sh script file.

sudo nano /usr/bin/mythtv-disk-monitor.sh

Copy and paste the following lines:

----- begin mythtv-disk-monitor.sh -----

#!/bin/bash
#
#       Name:  mythtv-disk-monitor
#    Purpose:  To use smartmon tools to monitor mythtv disk drives
#              and notify user of problems
#  Called by:  root crontab
#
# Example crontab entry - daily at 8:40pm:  sudo crontab -e
#
# # Monitor disk drives for problems
# 40 20 * * * /usr/bin/mythtv-disk-monitor.sh > /dev/null 2>&1
#

#
#  List of disk drives to check (Modify as needed)
#
DRIVES="sda sdb sdc"

function notifyuser {
  # Notify user of issue with disk drive
  #   Parameters:
  #     1 - Event
  #     2 - Device
  #     3 - Optional Component

  #
  # Write message to MythTV Frontend / MythWelcome
  #
  mythutil --message \
    --message_text="Drive Status :: $1 :: $2 :: $3" \
    --timeout=5 \
    --bcastaddr=127.0.0.1
  #
  # Write message to Desktop
  #
  notify-send 'Drive Status' "$1 :: $2 :: $3" \
    --expire-time=5000
}

#
# Check all drives if reallocated sector count greater than X
#
for k in $DRIVES; do
  if [ -e /dev/$k ]; then
    # https://stackoverflow.com/questions/22727107/how-to-find-the-last-field-using-cut
    RSCT=`/usr/sbin/smartctl -A /dev/$k | grep -E 'Reallocated_Sector_Ct|Reallocated_Event_Count' | rev | tr -s ' ' | cut -d' ' -f1 | rev`
    if [ $RSCT -ge 10 ]; then
      notifyuser "Reallocated_Sector_Count" /dev/$k $RSCT
      #
      # Sleep to separate potential multiple notifications
      #
      sleep 6
    fi
  fi
done

----- end mythtv-disk-monitor.sh -----

Feel free to adjust the DRIVES= line to suit your hardware.

Save file and exit editor with Ctrl+X.

Make /usr/bin/mythtv-disk-monitor.sh file executable.
```
sudo chmod 755 /usr/bin/mythtv-disk-monitor.sh
```
Test script execution (should be no errors).
```
sudo /usr/bin/mythtv-disk-monitor.sh
```
Schedule the script to run under the root user.
```
sudo crontab -e
```
Copy and append the following lines:

----- begin root crontab lines -----
```
#
# Monitor disk drives for problems
40 20 * * * /usr/bin/mythtv-disk-monitor.sh > /dev/null 2>&1
```
----- end root crontab lines -----

In this example I chose 8:40pm as a likely time when I might be watching a MythTV recording. Feel free to use a different time when you might be watching MythTV.

Figure 1: Example notification of disk drive with bad sectors

Script Monitoring Complete

Congratulations. You now have an automated disk monitoring tool watching out for impending disk failure. :-)

References

While developing this tutorial I found the following reference useful:

Identifying and Replacing a Failing RAID Drive