Photography Backup Strategy - Photographic Memories

Photography Backup Strategy

12.28.2016

Background


I am Ann's husband, Brady, and I have a background in computer science/engineering, and create lots of backup solutions as part of the work that I do. When she began her business, I wanted to help my amazing wife create a redundant backup solution that was not completely reliant on a cloud provider. While she does use a cloud backup for a lot of her final JPEG images, many photo backup sites do not allow uploading of RAW images. Plus, it's a good idea to not be completely reliant on cloud backups, especially when your business depends on it. So, below, I created instructions to do a setup exactly as I did for her. The instructions are somewhat technical in detail, but I tried to make them manageable by being very detailed. If you have any questions about the setup, don't hesitate to leave a question in the comments!



Data Situation


First, I thought about how I wanted to do the backup. There are a lot of products available to store lots of data, but what I wanted to achieve was a store of all my photos, Lightroom data, etc, and also keep a backup of that. This is different than your average computer backup situation for the following reasons:

1.  A backup solution (Apple TimeMachine, for example), can easily backup the contents of your computer’s hard drive. However, the hard drive on my MacBook Pro is 256GB SSD, and with all my applications installed, this isn’t nearly enough space to house my raw and edited images.

2.  Since I’ll need to store all my images on a high density external hard drive (multiple TB), I will need to have everything stored on multiple drives in case of a drive failure.



Backup Details


So, I decided to keep it simple by purchasing two Western Digital MyPassport for Mac hard drives. I store all of the photos in one of these drives, and then configure a nightly process to backup the contents of one drive to the contents of another drive.


These instructions that follow are specifically for configuring this backup on an Apple system. Specifically, I am running El Capitan (10.11.6 at the time of this writing).


The first thing I did was got a USB Hub so I could connect all my USB drives and other devices and only take one of my precious two USB slots on my MacBook.


Then, I plugged in my hard drives and named them appropriately. You can rename them by right clicking on the drives in the Finder application. I called them “PhotoMemPrimary” and “PhotoMemBackup”, where the former will be where I store all my files, and the latter will be what gets automatically synchronized each night. Note that I never store anything directly on “PhotoMemBackup”. Its only purpose is to receive the nightly backup, effectively being a clone of the primary drive.



The Backup Script:


Now, I had to write a script to do my backing up. Some of these steps get a little technical, but I am going to walk through them in very specific detail, so I know you can do it, even if it is out of your comfort zone.


So, on your Mac, hold down the command key and then press the spacebar (this is how you bring up Spotlight Search), and type in “Terminal”, and press enter. You should be greeted with a window like this:

Mac Terminal Prompt

This is called a “shell” or a “command prompt” where you can do many powerful things if you know what you’re doing. But with great power comes great responsibility. The instructions I provide here aren’t going to do any damage to your computer, but there do exist some relatively simple commands that can be catastrophic. Follow the instructions and you will be OK, but it is important you follow them closely. I feel obligated to stress that you should only follow this procedure if you are comfortable with the risks.

I will give the commands to type, and I will precede them with a '$'. Do not type the '$' sign; it is just an indicator that it is the beginning of the new command. Also, you will always need to type the Enter/Return key after typing the command. We’re going to create a script that gets run every night. A script is just a set of commands that occur one after another, like an actor reciting her lines or a telemarketer trying to convince you to donate to their bogus charity. So, we need to create and edit a file that has these commands, and we need to tell the computer that it’s a file that can be run, or executed.

Let's try it with an example. Open a Terminal window, and type the 'whoami' and 'date' commands:

# This is the command prompt. The '$' indicates the start of the new command.
# You will not type the '$'. Lines that do not start with a '$' are the
# output of the commands.

$ whoami
annbickel
$ date
Wed Dec 28 17:10:30 EST 2016

OK, so we now are able to run commands in the Terminal. Now, we are going to edit a file in the terminal. The following command will open a new file for editing:

$ nano ~/backup.sh

'nano' is the name of a program - specifically it's a text editor that lets you edit a file within the terminal window. Now, copy and paste the following text into your editor:

#!/bin/bash

datenow=$(date "+%Y-%m-%d_%H%M%S")
outfile=${HOME}/backup-logs/${datenow}.txt
rsync -av /Volumes/<primaryDrive>/ /Volumes/<backupDrive>/ 1>${outfile} 2>${outfile}

/usr/local/bin/sendemail -f bradybickel@gmail.com -t <account>@gmail.com -u "Photographic Memories Backup ${datenow}" -m 'Backup log for today attached' -a ${outfile} -s smtp.gmail.com:587 -o tls=yes -xu <account> -xp <password>

Now, allow me to explain what the other lines are doing:

- The first line tells the terminal what kind of script we are running, and how it should interpret the commands. Just copy this one exactly as written

- The next line captures the current date and time, which allows us to store off a text file that details what is happening each time the script is run.

- Next, we define what this output text file is, and where it will be located. In my example, they will exist in our home directory (look for the little house icon in the left sidebar on Finder), in a folder called backup-logs. The files will be named based on the date the script is run (e.g. 2016-12-26_010000.txt will be the result of running the script on 12/26/2016 at 1am).

- The “rsync” command is a fancy copy command that only copies the differences between the source and destination folders. When you’re dealing with hundreds of gigabytes or terabytes, this is crucial, because doing a full nightly copy of the contents of one drive to another could take hours and hours. You want to avoid this by only copying what needs to be copied.

You will need to substitute some text in your own example. The <primaryDrive> and <backupDrive> in the rsync command will need to be replaced with whatever you called your hard drives. In my example these are PhotoMemPrimary and is PhotoMemBackup.

The last command is optional, and I use it so I get the status of the backups emailed to me so I know that they keep occurring, and in case I want to verify which files are getting transferred. If you do wish to do this, there are a few things you need to do:

- Use your own gmail account information. Depending on your Google account settings, you may need to generate an application specific password.

- You can install sendemail using brew: “brew install sendemail”.

- If you don’t want to do this part, just omit everything including “/usr/local/bin/sendemail” and after. You can always come back and add that line later if you wish.


Now that you’ve entered all the commands in the script file, press Ctrl-o, then Enter, then Ctrl-x. Now, we want to make the script runnable, and try to run it:

$ chmod a+x ~/backup.sh
$ mkdir ~/backup-logs
$ ~/backup.sh

If anything exists on your Primary external hard drive, the contents should get copied to your backup hard drive. You can verify that this happened by inspecting them. Now, we have a script, and have verified that it works (if not, leave a comment, and we can try to help!). We have not yet configured it to run automatically every night. To do this, you will require administrator rights. To verify you have administrative rights on your Mac, see this link (https://community.sophos.com/kb/hu-hu/122673). Now, we have to setup the job to run every night. First, create a configuration file that configures the job:

Type the following command to create the configuration file:

$ sudo nano /Library/LaunchDaemons/com.myDomainName.daily-backup.plist

and copy and paste the following text into your editor:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>com.myDomainName.daily-backup</string>
	<key>ProgramArguments</key>
	<array>
		<string>/Users/myUserName/backup.sh</string>
	</array>
	<key>LowPriorityIO</key>
	<true/>
	<key>Nice</key>
	<integer>1</integer>
	<key>StartCalendarInterval</key>
	<dict>
		<key>Hour</key>
		<integer>1</integer>
		<key>Minute</key>
		<integer>0</integer>
	</dict>
	<key>UserName</key>
	<string>myUserName</string>
	<key>GroupName</key>
	<string>staff</string>
</dict>
</plist>

You will have to replace with your own instances of myDomainName and myUserName. In my case, I used photographicmemoriespa for myDomainName, and annbickel for myUserName. If you’re not sure what your username is, type “whoami” at the terminal, and appropriately enough, it will tell you who you are.

This configuration is in what is called an XML file (eXtensible Markup Language), and it is often a way of defining a list of configurable settings. In this case, it is configuring how you want your nightly job to run. It’s specifying what it is, where it’s located, and when you want to run it. There are a series of keys that are defined. A key is the text that is between the “<key></key>” tags. Then, the information that follows the the value of that key. So, for key “Label”, our value is “com.myDomainName.daily-backup”. What is really important here is the value for “ProgramArguments”, which is a call to your script, “UserName”, which is your Mac user name, and “StartCalendarInterval”, which defines what time the script will run (ours will run at 1:00 AM everyday). There are advanced configurations that you can do with these configuration files, but this meets our needs.

Now that this has been created, it needs to be registered with the operating system. You do that by typing the following:

$ sudo launchctl load /Library/LaunchDaemons/com.myDomainName.daily-backup.plist

You can verify that this is setup correctly by typing:

$ sudo launchctl list | grep myDomainName

If you see something like:

- 0 com.myDomainName.daily-backup

then that means the service is registered and should execute daily. Don't forget to check the backup-logs folder in your home folder periodically to make sure your scripts are automatically running.


Feel free to leave any questions in the comments, and good luck!

Powered by SmugMug Log In