Fully Automated Backups of QNAP-TS410

One of my clients has a couple of sites with a QNAP NAS device at each that they use as file servers.  These QNAPs have a built in "backup" facility that you can use to automate copying the data from one device to another.

Why did I put backup in quotes?  Because, this is not really backup, it is replication which is not the same thing at all.

To illustrate the difference, imagine you have a spreadsheet say, with important financial data in.  One day, one of your staff deletes a large chunk of the contents of this spreadsheet and saves the file.  A couple of days later you realise this and go to your backup to get an older version of the file from before the data was removed...

If all you have is a replica of the current files then you are properly stuffed.

Now this didn't actually happen to my client luckily but when I realised what the situation was (I had inherited the environment...) I knew I had to do something about it.

After much research I found there was no simple way to do this, but thanks to one article I found, and a lot of hard work on my part, I managed to piece together a full solution for automated daily backups (with reporting) using rsnapshot which I thought I would share in the hopes that it might help someone else out.

Many thanks go to http://www.sysnet.co.il/ where I found most of the ground work for this.  This article builds upon that one, detailing how to fix a few problems I encountered with these specific QNAPs, and also extends it to produce automated reporting on the success of the backups

http://www.sysnet.co.il/files/Daily%20incremental%20backups%20with%20rsnapshot.pdf

Installation of Packages

Log on to the web interface of the QNAP

• Go to Applications… QPKG Plugins
• Click GET QPKG
• If this fails – “Sorry, QPKG information is not available”
• Check DNS servers have been configured correctly under System Administration… Network
• If still not working, try downloading the package from somewhere else and copying to the NAS as this is very internet speed dependent and prone to timing out.
• Select the Optware IPKG and download the appropriate package
• For the QNAP-TS410 devices this is the ARM (x10/x12/x19 series) for TS-419P
• IPKG just allows to install a much wider range of packages onto the QNAP
• Extract the zip file to get  a .qpkg file then click on INSTALLATION
• Browse to the qpkg file and click INSTALL
• If this fails then use scp to copy the qpkg file to the NAS and then install from the command line using 
sh Optware0.99.163_arm-x19.qpkg
and this will show you more information about why it has failed
• If you get an error saying:
  “Optware 0.99.163 installation failed. /share/MD0_DATA/optware existed. Please remove it first.”
  Then type the following command and try the install again:
  rm –rf /share/MD0_DATA/.qpkg/Optware
• Once installation is successful, go to QPKG INSTALLED
• Click on Optware and click ENABLE
• Use putty (or whatever) to connect via SSH to the QNAP
• Run the following commands to install the necessary packages
/opt/bin/ipkg update            (there will be some errors here but ignore them)
/opt/bin/ipkg install rsnapshot
/opt/bin/ipkg install nano

Rsnapshot Configuration

Configuration of rsnapshot is managed by editing the file /opt/etc/rsnapshot.conf

Initially this needs to be set up with the location on the file system to store the snapshots and a list of folders to be backed up.

Note that for remote backups the data is pulled from the server to be backed up NOT pushed to the backup server as you might expect.

To set the snapshot folder location, edit /opt/etc/rsnapshot.conf

nano /opt/etc/rsnapshot.conf

and change:

snapshot_root  /opt/var/rsnapshot

to

snapshot_root  /share/MD0_DATA/rsnapshot

Exclude any rsnapshot backups and other data you do not want backed up by adding the following:

exclude        /share/MD0_DATA/rsnapshot

exclude        /share/MD0_DATA/Backup

exclude        /share/MD0_DATA/Qdownload

exclude        /share/MD0_DATA/Qmultimedia

exclude        /share/MD0_DATA/Qrecordings

exclude        /share/MD0_DATA/Qusb

exclude        /share/MD0_DATA/Qweb

Change the intervals to keep however many backups you want (rsnapshot is very efficient at not duplicating data so I configure it for 3650 daily backups, or ten years’ worth) by setting the BACKUP INTERVALS section as follows:

#interval      hourly     6

interval       daily      3650

#interval      weekly     4

#interval      monthly    12

Comment out the default backup sets by putting a hash in front of these lines in the BACKUP POINTS / SCRIPTS section

#backup        /etc/      localhost/

#backup        /opt/etc/  localhost/

Add the following line to carry out the actual backup:

backup admin@<site1_ip_address>:/share/MD0_DATA/    <site1>

NOTE:  The fields in this file must be separated by tabs, not spaces

After making any changes to the rsnapshot.conf, run the following command to make sure that everything is OK

rsnapshot configtest

Setting Up SSH Keys

In order for the scheduled backups to be able to access the remote NAS device we need to set up SSH keys – this will allow the backup server to present a certificate to the remote server for authentication.

First copy the key from the backup server to the source server:

ssh admin@<site1_ip_address> "echo `cat ~/.ssh/id_rsa.pub` >> ~/.ssh/authorized_keys"

NOTE: the above command is all one line

Accept the authenticity of the host if prompted and then enter the admin password when prompted

Next set the permissions on the configuration file and add this to the autorun to ensure it is preserved following a reboot.

Login to the source NAS (Site 1) and enter the following commands:

nano /tmp/config/autorun.sh

Add these three lines to the file and save it:

mount –t ext2 /dev/mtdblock5 /tmp/config
chown admin.administrators /mnt/HDA_ROOT/.config
umount /tmp/config


Scheduling Rsnapshot

Cron is used to schedule the actual snapshots.  To configure this edit /etc/config/crontab so that it has the following rsnapshot entries:

0 23 * * * /opt/bin/rsnapshot daily

After making any changes to this file, make sure to run the following command to ensure the changes are preserved if the NAS is rebooted

crontab /etc/config/crontab

 

Email Configuration

To set up the NAS for email, use the QNAP web administration page and go to System Administration… Notification… and enter the settings for your mail server: 

You can send a test message from the ALERT NOTIFICATION tab to check this is working.

Setting Up Monitoring

The rsnapshot log file is stored here: /opt/var/log/rsnapshot

For automated alerting, create the following script, and then schedule it through crontab (make sure this script is stored under /share/MD0_DATA/… as files stored in /root for example, get removed on system reboot.)

/share/MD0_DATA/rsnapshot/check_backup.sh

#!/bin/bash
recipient=recipient@example.local
sender=sender@example.local
subject="Site 1 Backup Report"
log=/opt/var/log/rsnapshot
tmpfile=/opt/var/log/rsnapshot.tmp

function log {
  /bin/echo "[`date +%d/%b/%Y:%H:%M:%S`] $*\n" >> $log
}

if grep -q "rm -f /opt/var/run/rsnapshot.pid" $log
then
  if grep -q ERROR: $log
    then
      status=FAILED
    else
      status=SUCCESSFUL
  fi
else
  status="FAILED, INCOMPLETE"
# The following lines safely kill the backup to stop it
# impacting performance during the day
# comment them out if you do not want this to happen
  log "Backup window exceeded, terminating processes"
  snapPID=`ps -ef | grep "/opt/bin/perl -w /opt/bin/rsnapshot daily" | awk '{print $1}'`

  rsyncPID=`ps -ef | grep "/opt/bin/rsync -a --delete --numeric-ids" | awk '{print $1}'`
  log "rsnapshot pid = $snapPID"
  log "rsync pid = $rsyncPID"
  log “Killing rsnapshot process”
  kill $snapPID
  log "Sleeping for 5 minutes to wait for rsnapshot to exit"
  sleep 300
  log “Killing rsync processes”
  kill $rsyncPID
fi

echo To: $recipient > $tmpfile
echo From: $sender >> $tmpfile
echo Subject: $subject - BACKUP $status >> $tmpfile
echo "Disk space used by backups: $(rsnapshot du | grep total)" >> $tmpfile
echo "Most recent backup sizes: " >> $tmpfile
echo "$(du -sh /share/MD0_DATA/rsnapshot/daily.0/*)" >> $tmpfile
echo "Backup Log:" >> $tmpfile

cat $tmpfile $log | /mnt/ext/usr/sbin/ssmtp -v $recipient

rm $log
rm $tmpfile


NOTE: lines in bold have been wrapped in this document – they must be all on one line in the actual script.

Make the script executable by typing the following command:

chmod +x /share/MD0_DATA/rsnapshot/check_backup.sh

Then schedule the script to run each morning by adding the following lines to the crontab:

0 7 * * * /share/MD0_DATA/rsnapshot/check_backup.sh

Restoring Data

Use the following command to show all the backups and when they completed:

[~] # ls -Ahlt  /share/MD0_DATA/rsnapshot/

drwxr-xr-x    3 admin    administ     4.0k Jul 10 23:21 daily.0/

drwxr-xr-x    3 admin    administ     4.0k Jul  9 23:40 daily.1/

drwxr-xr-x    3 admin    administ     4.0k Jul  8 23:41 daily.2/

drwxr-xr-x    3 admin    administ     4.0k Jul  7 23:37 daily.3/

drwxr-xr-x    3 admin    administ     4.0k Jul  6 23:29 daily.4/

drwxr-xr-x    3 admin    administ     4.0k Jul  5 23:27 daily.5/

drwxr-xr-x    3 admin    administ     4.0k Jul  5 04:59 daily.6/

In each of the daily folders there is a subdirectory for each server it has backed up and below that is a full replica of that server’s data.

Restoring files is simply done using the scp command on the backup NAS which has the following syntax:

scp <source_file> <site1_ip_address>:<destination>

If the source file/path has spaces in then enclose the full path in quotes.

So, to restore the file expenses.xls in the Contractors share from the 7^th July, use this command:

scp “/share/MD0_DATA/rsnapshot/daily.2/<site1>/share/MD0_DATA/Contractors/expenses.xls” <site1_ip_address>:/share/MD0_DATA/Contractors

NOTE:  The above command is all one line.

Dealing With Overrunning Backups
If backups over run into the day this could affect performance of the NAS and internet access.  This is particularly likely when the system is first setup as there may be a lot of data to transfer.  Once the initial transfer has taken place only changes are sent so the backup is much quicker.

The check_backup.sh script automatically detects this and stops the backup running, but if you need to do it manually at another time then perform the following steps:

To check if a backup is still running, SSH to the NAS and type the following commands:




[~] # ps aux | grep rs

31897 admin   3256 S   /opt/bin/perl -w /opt/bin/rsnapshot daily

32219 admin   2140 S   /opt/bin/rsync -a --delete --numeric-ids
32220 admin   4748 S   /opt/bin/ssh -l admin <site1_ip_address>
32301 admin   1932 S   /opt/bin/rsync -a --delete --numeric-ids

If this shows any rsync or rsnapshot processes then the backup is still running.  To stop the backup kill the processes in this order:

/opt/bin/perl –w /opt/bin/rsnapshot daily

/opt/bin/rsync ….

To do this, use the command “kill” followed by the PID (process identifier) – this is the first number in the output of the ps command.  So for the example above, to stop the backup, issue these commands.

kill 31897

ps aux | grep rs

Wait for the rsnapshot daily process to exit – this may take a while so periodically run the ps command again to check if it is still there.  Once it has finished, then kill the first rsync process (this will stop the others as well)

kill 32219

NOTE:  Use the correct PIDs, don’t just copy the ones here !!!

If you kill the rsync process first then rsnapshot notices this and rolls back the snapshot that was in progress deleting all the data.  Any previous days’ data is still there but if the backup was overrunning then you do not want to have to start copying the new data all over again – by doing it this way round the backup can pick up where it left off next time.

References
Incremental QNAP Backups with rsnapshot http://www.sysnet.co.il/files/Daily%20incremental%20backups%20with%20rsnapshot.pdf

Rsnapshot      http://www.rsnapshot.org/