6. Configure Automatic Backup¶
Start the virtual machine and log in:
ssh -p 8015 hjc@dynamicshjc.case.edu
Check for and install system updates on the virtual machine:
sudo apt-get update sudo apt-get dist-upgrade sudo apt-get autoremove
Download and install the backup script:
sudo wget -O /usr/local/sbin/backup-wiki https://neurowiki-docs.readthedocs.io/en/latest/_downloads/backup-wiki
Set the MySQL password inside the script:
read -s -r -p "MySQL password: " DBPASS && sudo sed -i '/^SQLPASS=/s|=.*|='$DBPASS'|' /usr/local/sbin/backup-wiki; DBPASS= ; echo
Protect the password:
sudo chown root:www-data /usr/local/sbin/backup-wiki sudo chmod ug=rwx,o= /usr/local/sbin/backup-wiki
If you are curious about the contents of the script, you can view it here:
backup-wiki
#!/bin/bash # Copyright (c) 2016, Kendrick Shaw, Jeffrey Gill, Hillel Chiel # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: # # * Redistributions of source code must retain the above copyright notice, # this list of conditions and the following disclaimer. # * Redistributions in binary form must reproduce the above copyright notice, # this list of conditions and the following disclaimer in the documentation # and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. DEFAULTNAME=$(hostname)-backup-$(date +'%Y%m%dT%H%M%S%z') # ISO-8601 timestamp SQLUSER=root SQLPASS=<MySQL password> WIKIDB=wikidb DJANGODB=djangodb WWWDIR=/var/www WIKIDIR=$WWWDIR/mediawiki DJANGODIR=$WWWDIR/django function usage { echo "Usage:" echo " backup-wiki backup [-q] [backup_filename]" echo " backup-wiki restore [-q] backup_filename" echo echo "Options:" echo " -q Quiet mode, print only errors" exit } # read in options QUIET=false while getopts ":q" opt; do case $opt in q) # the flag -q was set QUIET=true ;; \?) # an unrecognized flag was set echo "Invalid option: -$OPTARG" >&2 usage ;; esac done shift $((OPTIND-1)) # read in backup_file argument if [ -z "$2" ]; then # second argument not provided -- use default file name BACKUPNAME=$DEFAULTNAME else BACKUPNAME=$(dirname $2)/$(basename $2 .tar.bz2) fi ##### BACKUP MODE ##### if [ "backup" == "$1" ]; then if [ $QUIET = false ]; then echo "Starting backup to archive $BACKUPNAME.tar.bz2 ..." fi # create a temporary directory if [ $QUIET = false ]; then echo "Creating temporary directory ..." fi TEMPDIR=$(mktemp -d) # copy the MediaWiki, Django, and JSNeuroSim source code if [ $QUIET = false ]; then echo "Copying files ..." fi cp -R $WWWDIR $TEMPDIR # dump the wiki database in sql format if [ $QUIET = false ]; then echo "Exporting the wiki database in SQL format ..." fi mysqldump --user=$SQLUSER --password=$SQLPASS $WIKIDB --complete-insert --result-file $TEMPDIR/wikidb.sql 2>&1 | grep -v "\[Warning\] Using a password" # dump the django database in sql format if [ $QUIET = false ]; then echo "Exporting the Django database in SQL format ..." fi mysqldump --user=$SQLUSER --password=$SQLPASS $DJANGODB --complete-insert --result-file $TEMPDIR/djangodb.sql 2>&1 | grep -v "\[Warning\] Using a password" # dump the django database in json format (a more portable backup of the content) if [ $QUIET = false ]; then echo "Exporting the Django database in JSON format ..." fi python $DJANGODIR/manage.py dumpdata > $TEMPDIR/djangodb.json # compress everything into a single file if [ $QUIET = false ]; then echo "Compressing directory into an archive file ..." fi mkdir -p $(dirname $BACKUPNAME) tar -cjf $BACKUPNAME.tar.bz2 -C $TEMPDIR . chmod o= $BACKUPNAME.tar.bz2 # delete the temporary directory if [ $QUIET = false ]; then echo "Deleting temporary directory ..." fi rm -rf $TEMPDIR if [ $QUIET = false ]; then echo "Done!" echo echo "NOTE: The backup you just created contains sensitive student information, quiz" echo "answers, and passwords. Keep this file in a safe place!" echo fi ##### RESTORE MODE ##### elif [ "restore" == "$1" ]; then if [ -z "$2" ]; then echo "Missing argument: backup_filename" >&2 usage elif [ -e "$BACKUPNAME.tar.bz2" ]; then if [ "$(whoami)" != "root" ]; then echo "Aborting: superuser privileges needed" >&2 usage else if [ $QUIET = false ]; then echo "Starting restoration from archive $BACKUPNAME.tar.bz2 ..." fi # extract the files if [ $QUIET = false ]; then echo "Unpacking archive ..." fi TEMPDIR=$(mktemp -d) tar -xjf $BACKUPNAME.tar.bz2 -C $TEMPDIR # stop the server if [ $QUIET = false ]; then echo "Stopping the web server ..." fi apache2ctl stop # copy the files, fixing permissions and owners if [ $QUIET = false ]; then echo "Copying files ..." fi rm -rf $WWWDIR cp -R $TEMPDIR/$(basename $WWWDIR) $WWWDIR chown -R www-data:www-data $WWWDIR chmod -R ug+rw $WWWDIR find $WWWDIR -type d -exec chmod g+s {} \; # restore the wiki database if [ $QUIET = false ]; then echo "Restoring the wiki database ..." fi echo "DROP DATABASE IF EXISTS $WIKIDB;" | mysql --user=$SQLUSER --password=$SQLPASS 2>&1 | grep -v "\[Warning\] Using a password" echo "CREATE DATABASE $WIKIDB;" | mysql --user=$SQLUSER --password=$SQLPASS 2>&1 | grep -v "\[Warning\] Using a password" mysql --user=$SQLUSER --password=$SQLPASS $WIKIDB < $TEMPDIR/wikidb.sql 2>&1 | grep -v "\[Warning\] Using a password" # restore the django database if [ $QUIET = false ]; then echo "Restoring the Django database ..." fi echo "DROP DATABASE IF EXISTS $DJANGODB;" | mysql --user=$SQLUSER --password=$SQLPASS 2>&1 | grep -v "\[Warning\] Using a password" echo "CREATE DATABASE $DJANGODB;" | mysql --user=$SQLUSER --password=$SQLPASS 2>&1 | grep -v "\[Warning\] Using a password" mysql --user=$SQLUSER --password=$SQLPASS $DJANGODB < $TEMPDIR/djangodb.sql 2>&1 | grep -v "\[Warning\] Using a password" # restart the server and clean-up if [ $QUIET = false ]; then echo "Restarting the web server ..." fi apache2ctl start rm -rf $TEMPDIR if [ $QUIET = false ]; then echo "Done!" fi fi else echo "Bad argument: file $BACKUPNAME not found" >&2 usage fi else usage fi
Todo
Make the backup script that lives on DynamicsHJC and the cron job list downloadable.
In a different Terminal window, log into the VirtualBox Machines (vbox) account on DynamicsHJC:
ssh vbox@dynamicshjc.case.edu
Create a script that remotely executes the backup script and moves the archive to the backup drive. Create the file
mkdir -p /Volumes/CHIELWIKI/backups/neurowiki/2016 vim /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh
and fill it with the following:
#!/bin/bash #REMOTESSH="ssh hjc@neurowiki.case.edu" REMOTESSH="ssh -p 8015 hjc@dynamicshjc.case.edu" #REMOTESCP="scp -q hjc@neurowiki.case.edu" REMOTESCP="scp -q -P 8015 hjc@dynamicshjc.case.edu" REMOTEFILE=neurowiki-backup-`date +'%Y%m%dT%H%M%S%z'`.tar.bz2 if [ -z "$1" ]; then LOCALFILE=/Volumes/CHIELWIKI/backups/neurowiki/2016/$REMOTEFILE else LOCALFILE=/Volumes/CHIELWIKI/backups/neurowiki/2016/$1 fi $REMOTESSH backup-wiki -q backup $REMOTEFILE $REMOTESCP:$REMOTEFILE $LOCALFILE chmod go= $LOCALFILE $REMOTESSH rm $REMOTEFILE
Make the script executable:
chmod u+x /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh
Copy the public SSH key from the vbox account on DynamicsHJC to the virtual machine to allow automatic authentication (you will be asked to accept the unrecognized fingerprint of the virtual machine — this is expected — and you will be asked for your password to the virtual machine twice):
ssh-keygen -R [dynamicshjc.case.edu]:8015 scp -P 8015 ~/.ssh/id_rsa.pub hjc@dynamicshjc.case.edu: ssh -p 8015 hjc@dynamicshjc.case.edu "mkdir -p .ssh && cat id_rsa.pub >> .ssh/authorized_keys && rm id_rsa.pub"
Test whether automatic authentication is working by trying to log into the virtual machine from the vbox account on DynamicsHJC — you should NOT need to enter your password:
ssh -p 8015 hjc@dynamicshjc.case.edu
If this works without you needing to enter a password, automatic authentication is properly configured. You should
logout
to return to the vbox account on DynamicsHJC.In the vbox account on DynamicsHJC, create a backup schedule. Create the file
vim /Volumes/CHIELWIKI/backups/neurowiki/2016/crontab
and fill it with the following:
################################################################################ # NEUROWIKI # ################################################################################ # Make hourly backups on the odd-numbered hours (except 1 AM and 3 AM) 0 5,9,13,17,21 * * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh hourA.tar.bz2 0 7,11,15,19,23 * * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh hourB.tar.bz2 # Make daily backups at 1 AM 0 1 * * 0 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh sunday.tar.bz2 0 1 * * 1 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh monday.tar.bz2 0 1 * * 2 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh tuesday.tar.bz2 0 1 * * 3 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh wednesday.tar.bz2 0 1 * * 4 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh thursday.tar.bz2 0 1 * * 5 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh friday.tar.bz2 0 1 * * 6 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh saturday.tar.bz2 # Make weekly backups at 3 AM on the 1st, 8th, 15th, and 22nd of the month 0 3 1 * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh `date +'\%Y-\%m'`.tar.bz2 0 3 8 * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh 8th.tar.bz2 0 3 15 * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh 15th.tar.bz2 0 3 22 * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh 22nd.tar.bz2
Schedule the backups. In the vbox account on DynamicsHJC, dump the existing scheduled jobs to a temporary file:
crontab -l > /tmp/crontab.old
Edit the temporary file, and delete the backup jobs for last year’s NeuroWiki (leave the jobs for NeuroWikiDev for now). You can use
Shift+v
in Vim to enter Visual Line mode, the up and down arrow keys to select a block of lines, andd
to delete them all at once.vim /tmp/crontab.old
Now append the new jobs to the old and schedule them:
cat {/tmp/crontab.old,/Volumes/CHIELWIKI/backups/neurowiki/2016/crontab} | crontab
Verify that the backup jobs for this year’s NeuroWiki are properly scheduled, and that backup jobs for last year’s NeuroWiki are absent:
crontab -l
Log out of the vbox account:
logout
You can now return to your original Terminal window.
Shut down the virtual machine:
sudo shutdown -h now
Using VirtualBox, take a snapshot of the current state of the virtual machine. Name it “Automatic backups configured”.