On Thursday, it was bring your daughters and sons to work day at Genentech. My little Marlowe is only 6 weeks old so I didn’t bring her, but my VP shot me a note Wednesday saying she was bringing her two boys, knew of some other kids who would be there, and wondered if I could give a 30 minute talk on systems. I responded, “sure.”

But why the heck do you tell kids ranging from 6-12 years old about your corporate ERP system? While fascinating, I doubt they’d care about the usual things that I work on and I wouldn’t dare show them PowerPoint slides. So what to do?

Thursday before lunch I slapped a Ruby script together (code below). It did 2 things: 1) let you decide how many plants were producing widgets, and 2) how fast they could produce. The first run looked like this:

How many plants are there? (max 6)
1
About how many seconds does it take a plant to make something?
5
Monitoring a network of 1 plants
STARTING P1
Plant 1: +3
Plant 1: +7
Plant 1: +3

Monitoring stopped
Plant 1 made 13

As the output slowly appeared I had them add up the widgets produced, which they did with ease. We clearly didn’t need systems … yet. Then we increased the number of plants and speed of production until we got to this scenario:

How many plants are there? (max 6)
6
About how many seconds does it take a plant to make something?
1
Monitoring a network of 6 plants
STARTING P1 |             |             |             |             |            
Plant 1: +8 |             |             |             |             |            
            | STARTING P2 |             |             |             |            
            | Plant 2: +9 |             |             |             |            
            |             | STARTING P3 |             |             |            
            |             | Plant 3: +3 |             |             |            
            |             |             | STARTING P4 |             |            
            |             |             | Plant 4: +5 |             |            
Plant 1: +8 |             |             |             |             |            
            |             |             |             | STARTING P5 |            
            |             |             |             | Plant 5: +4 |            
            | Plant 2: +5 |             |             |             |            
            |             |             |             |             | STARTING P6
            |             |             |             |             | Plant 6: +1
            |             | Plant 3: +7 |             |             |            
            |             |             | Plant 4: +3 |             |            
Plant 1: +9 |             |             |             |             |            
            |             |             |             | Plant 5: +1 |            
            |             |             |             |             | Plant 6: +8
            | Plant 2: +5 |             |             |             |            
            |             | Plant 3: +9 |             |             |            
            |             |             | Plant 4: +1 |             |            
            |             |             |             | Plant 5: +4 |            
Plant 1: +2 |             |             |             |             |            

Monitoring stopped
Plant 1 made 27
Plant 2 made 19
Plant 3 made 19
Plant 4 made 9
Plant 5 made 9
Plant 6 made 9

That’s a shortened example. In our exercise we let it run until each plant had made hundreds of widgets. Though they tried hard, they simply couldn’t keep up. At this point I stopped and asked them if it was now clear why we use systems and one kid answered beautifully: yeah, because you can’t count that fast. A week ago if you asked me, “why do you use SAP ERP,” I probably would have given you some long-winded, boring answer citing dollars, value, and other things that would put you to sleep. I like the simple version much better. Thanks kids.

If you’re curious, here’s the code. It’s not elegant, and could certainly be refactored, but I think it worked nicely to demonstrate why we use systems at work.

# Get some input about the plants in the network
puts "How many plants are there? (max 6)"
num_plants = gets.to_i
puts "About how many seconds does it take a plant to make something?"
seconds_to_make = gets.to_i
puts "Monitoring a network of #{num_plants} plants"

# Store plant inventory in an array
@@plants = Array.new(num_plants, 0)

# Wait for the user to hit enter and exit with inventory information
Thread.new do
  loop do
    s = gets.chomp
    puts "Monitoring stopped"
    count=0
    @@plants.each{|p| puts "Plant #{count+=1} made #{p}"}
    exit
  end
end

# Starts an independent plant thread and logs inventory
def start_plant(plant, sleep_for=seconds_to_make)
  Thread.new do
    loop do
      making = rand(9)+1
      front_pad = '            | '*plant
      back_pad = ' |            '*(5-plant-(6-@@plants.length))
      puts "#{front_pad}Plant #{plant+1}: +#{making}#{back_pad}"
      @@plants[plant] += making
      sleep sleep_for+(rand(0))
    end
  end
end

# Start N plant threads
1.upto(@@plants.length) do |p|
  plant = p-1
  front_pad = '            | '*plant
  back_pad = ' |            '*(5-plant-(6-@@plants.length))
  puts "#{front_pad}STARTING P#{plant+1}#{back_pad}"
  start_plant(plant, seconds_to_make)
  sleep 1
end

# Endless loop to allow other threads to run until interrupted
loop do; sleep 1; end

Marlowe was born in San Francisco, California at the University of California San Francisco (UCSF) hospital at 12:23pm on Sunday, 10 March 2013 (10 + 3 = 13 … yay! A math trivia birthdate!). She weighed in at 7 pounds 7 ounces, and measured 21 inches in length.

Both Alicia and Marlowe are doing great and getting a lot of needed rest. Marlowe’s middle name is borrowed from my paternal grandmother. Her first name was her mom’s top pick and after witnessing the process of giving birth, mom had final decision rights on the name. Luckily, when we both gave our final suggestion on what her name should be around noon on the 11th (24 hours after her birth), we both agreed Marlowe matched her personality.

For those of you looking for more pics, I will upload highlights by day to the link below for the first week or two, so check back each day for more:

Click for Marlowe Pics!

$ cat ./apt-update 
sudo apt-get update; sudo apt-get dist-upgrade; sudo apt-get autoremove

Nothing too crazy there. It updates the apt-get cache, performs the upgrade, and then removes all the residual junk that’s laying around. Well, almost all. If you do this enough, eventually you’ll see the following (assuming you’ve got the default motd Ubuntu script running and you’re logging in from a terminal):

=> /boot is using 86.3% of 227MB

This is because that script I mentioned doesn’t consider old kernel images to be junk. However, unless you’ve got an abnormal /boot partition, it doesn’t take too many old images to fill it up.

A quick Google search found Ubuntu Cleanup: How to Remove All Unused Linux Kernel Headers, Images and Modules. The solution on the page had exactly what I’m looking for, however, I couldn’t take it at face value. While the article offers an adequate solution, it doesn’t offer much explanation. The remainder of this article explains the details for this one-liner noted in the article above:

Note: Only run this if you’ve rebooted after installing a new kernel.

Ick. Let’s dig into what’s going on here. The pipe characters are chaining a bunch of commands together. Each command’s output becomes the input for the next. Given that, let’s walk through what’s going on in 3 steps.

Step 1 – List Linux Kernels: dpkg -l 'linux-*'

This one is fairly simple. Just pull up the man page for dpkg:

dpkg - package manager for Debian
...
dpkg-query actions
              -l, --list package-name-pattern...
                  List packages matching given pattern.

So, it should just list out the packages matching the pattern provided. Let’s try:

$ dpkg -l 'linux-*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
un  linux-doc-2.6.                     (no description available)
un  linux-doc-3.0.                     (no description available)
un  linux-doc-3.2.                     (no description available)
un  linux-doc-3.5.                     (no description available)
ii  linux-firmware 1.95         all          Firmware for Linux kernel drivers
un  linux-image                        (no description available)
un  linux-image-2.                     (no description available)
ii  linux-image-2. 2.6.38-11.50 amd64        Linux kernel image for version 2.
ii  linux-image-2. 2.6.38-8.42  amd64        Linux kernel image for version 2.
un  linux-image-3.                     (no description available)
rc  linux-image-3. 3.0.0-12.20  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.0.0-13.22  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.0.0-14.23  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.0.0-15.26  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.0.0-16.29  amd64        Linux kernel image for version 3.
ii  linux-image-3. 3.0.0-17.30  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.2.0-23.36  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.2.0-24.39  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.2.0-25.40  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.2.0-26.41  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.2.0-27.43  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.2.0-29.46  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.2.0-30.48  amd64        Linux kernel image for version 3.
rc  linux-image-3. 3.2.0-31.50  amd64        Linux kernel image for version 3.
ii  linux-image-3. 3.2.0-32.51  amd64        Linux kernel image for version 3.
ii  linux-image-3. 3.5.0-17.28  amd64        Linux kernel image for version 3.
ii  linux-image-3. 3.5.0-18.29  amd64        Linux kernel image for version 3.
ii  linux-image-3. 3.5.0-19.30  amd64        Linux kernel image for version 3.
ii  linux-image-3. 3.5.0-21.32  amd64        Linux kernel image for version 3.
ii  linux-image-3. 3.5.0-22.34  amd64        Linux kernel image for version 3.
ii  linux-image-3. 3.5.0-23.35  amd64        Linux kernel image for version 3.
ii  linux-image-ex 3.5.0-17.28  amd64        Linux kernel image for version 3.
ii  linux-image-ex 3.5.0-18.29  amd64        Linux kernel image for version 3.
ii  linux-image-ex 3.5.0-19.30  amd64        Linux kernel image for version 3.
ii  linux-image-ex 3.5.0-21.32  amd64        Linux kernel image for version 3.
ii  linux-image-ex 3.5.0-22.34  amd64        Linux kernel image for version 3.
ii  linux-image-ex 3.5.0-23.35  amd64        Linux kernel image for version 3.
ii  linux-image-ge 3.5.0.23.29  amd64        Generic Linux kernel image
ii  linux-image-se 3.5.0.23.29  amd64        Transitional package.
un  linux-initramf                     (no description available)
un  linux-kernel-h                     (no description available)
un  linux-kernel-l                     (no description available)
ii  linux-libc-dev 3.5.0-23.35  amd64        Linux Kernel Headers for developm
un  linux-restrict                     (no description available)
ii  linux-sound-ba 1.0.25+dfsg- all          base package for ALSA and OSS sou
un  linux-source-2                     (no description available)
un  linux-source-3                     (no description available)
un  linux-source-3                     (no description available)
un  linux-source-3                     (no description available)
un  linux-tools                        (no description available)

You can look at the man page for dpkg-query for more detail, but the items with an “i” for the 2nd character are currently installed. We’ll essentially want to dig through these installed old kernels and purge the ones we don’t want.

Step 2 – Show Unused Linux Kernels: sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d'

This one is a little more tricky. We need to dissect it from the inside out. In the middle of that mess, you’ll see:

"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"

That’s essentially a command inside a command (kind of like the pipe concept, but injected in middle instead of the start of the input stream). We can drop the encapsulating “$( )” to see what this does:

$ uname -r
3.5.0-23-generic

… and then …

$ uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/"
3.5.0-23

As we see, this is using the uname command with the -r option to print the kernel release that’s active on our system. That next piece calls a powerful program called sed. It uses a regular expression to trim down the output of uname. If read in English, it says, “for the input given, try to first find as many characters as you can until you get to the last hyphen and remember them; if after the hyphen you find only non-numeric characters then replace the matched string with what was remembered, otherwise do nothing.” The regular expression is much more eloquent. :)

To better understand the remaining logic, we can reduce that original monster to the following by manually inserting “3.5.0-23″ where that uname/sed combo was before:

sed '/^ii/!d;/'"3.5.0-23"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d'

This isn’t as bad as it first appears. It’s simply a chain of matches with /d or /!d, which mean delete the match or delete everything but the match respectively. Plus there’s one substitution. So …

/^ii/!d 

… means, delete all lines except those that start with ii

/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d

… means, delete all lines with the current kernel version in them (remember, this is equivalent to /'"3.5.0-23"'/d)

s/^[^ ]* [^ ]* \([^ ]*\).*/\1/

Consume all non-space characters, if any, followed by a space. Repeat. Remember the next set of non-space characters. Consume everything that follows. Replace everything with the remembered match.

/[0-9]/!d

… means, delete all lines except those with numbers.

My final output is the following:

linux-image-2.6.38-11-server
linux-image-2.6.38-8-server
linux-image-3.0.0-17-server
linux-image-3.2.0-32-generic
linux-image-3.5.0-17-generic
linux-image-3.5.0-18-generic
linux-image-3.5.0-19-generic
linux-image-3.5.0-21-generic
linux-image-3.5.0-22-generic
linux-image-extra-3.5.0-17-generic
linux-image-extra-3.5.0-18-generic
linux-image-extra-3.5.0-19-generic
linux-image-extra-3.5.0-21-generic
linux-image-extra-3.5.0-22-generic

Step 3 – Remove the Unused Kernels: xargs sudo apt-get -y purge

From the xargs man page, “xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial-arguments followed by items read from standard input. Blank lines on the standard input are ignored.” In other words, xargs is going to let us issue the same command to each of those items listed in the last section.

The rest is fairly self explanatory. For every package listed, we’re going to ask apt-get to purge it, which the man page states means, “packages are removed and purged (any configuration files are deleted too).” The -y option means apt-get won’t ask for a confirmation. If you’re uneasy about the process, you might remove this for a dry-run, but given the cyclic nature of how we’re calling the command, it will abort (which is fine for a dry run, but you’ll have to add -y for it to work).

For completeness, sudo runs the purge as the super user.

All Together

Let’s try without -y first:

$ dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get purge
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  linux-image-2.6.38-11-server* linux-image-2.6.38-8-server* linux-image-3.0.0-17-server* linux-image-3.2.0-32-generic* linux-image-3.5.0-17-generic*
  linux-image-3.5.0-18-generic* linux-image-3.5.0-19-generic* linux-image-3.5.0-21-generic* linux-image-3.5.0-22-generic* linux-image-extra-3.5.0-17-generic*
  linux-image-extra-3.5.0-18-generic* linux-image-extra-3.5.0-19-generic* linux-image-extra-3.5.0-21-generic* linux-image-extra-3.5.0-22-generic*
0 upgraded, 0 newly installed, 14 to remove and 0 not upgraded.
After this operation, 1,359 MB disk space will be freed.
Do you want to continue [Y/n]? Abort.

Note the abort. This occurs automatically. Everything looks good here. Our current kernel isn’t in the list. Let’s do it! This time a lot more happens. I’ve inserted [SIMILAR OUTPUT DELETED] where the output is iterating over versions:

$ dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  linux-image-2.6.38-11-server* linux-image-2.6.38-8-server* linux-image-3.0.0-17-server* linux-image-3.2.0-32-generic* linux-image-3.5.0-17-generic*
  linux-image-3.5.0-18-generic* linux-image-3.5.0-19-generic* linux-image-3.5.0-21-generic* linux-image-3.5.0-22-generic* linux-image-extra-3.5.0-17-generic*
  linux-image-extra-3.5.0-18-generic* linux-image-extra-3.5.0-19-generic* linux-image-extra-3.5.0-21-generic* linux-image-extra-3.5.0-22-generic*
0 upgraded, 0 newly installed, 14 to remove and 0 not upgraded.
After this operation, 1,359 MB disk space will be freed.
(Reading database ... 196310 files and directories currently installed.)
Removing linux-image-2.6.38-11-server ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 2.6.38-11-server /boot/vmlinuz-2.6.38-11-server
update-initramfs: Deleting /boot/initrd.img-2.6.38-11-server
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 2.6.38-11-server /boot/vmlinuz-2.6.38-11-server
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.5.0-23-generic
Found initrd image: /boot/initrd.img-3.5.0-23-generic
Found linux image: /boot/vmlinuz-3.5.0-22-generic
Found initrd image: /boot/initrd.img-3.5.0-22-generic
Found linux image: /boot/vmlinuz-3.5.0-21-generic
Found initrd image: /boot/initrd.img-3.5.0-21-generic
Found linux image: /boot/vmlinuz-3.5.0-19-generic
Found initrd image: /boot/initrd.img-3.5.0-19-generic
Found linux image: /boot/vmlinuz-3.5.0-18-generic
Found initrd image: /boot/initrd.img-3.5.0-18-generic
Found linux image: /boot/vmlinuz-3.5.0-17-generic
Found initrd image: /boot/initrd.img-3.5.0-17-generic
Found linux image: /boot/vmlinuz-3.2.0-32-generic
Found initrd image: /boot/initrd.img-3.2.0-32-generic
Found memtest86+ image: /memtest86+.bin
done
Purging configuration files for linux-image-2.6.38-11-server ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 2.6.38-11-server /boot/vmlinuz-2.6.38-11-server
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 2.6.38-11-server /boot/vmlinuz-2.6.38-11-server

[SIMILAR OUTPUT DELETED]

Note how the apt-get process even runs grub for you so your reboot menu is ready to go. Very nice!

Now let’s check our /boot partition, which was using 86% of 227 MB of storage space.

$ du -h /boot --max-depth=0
35M	/boot

Ah, only 35 M in use, i.e., 15%. Plenty of room for new kernel releases!

The issue I ran into with my photo is called chromatic aberration. It’s a term that simply means not all colors of light had the same focal alignment when captured. Chromatic aberration usually can be seen by colors bleeding out on the edges of subjects in your photos. Check out this shot of me as an example:

At first glance this photo seems normal. However, if you look at my face you’ll notice it has an oddly harsh contrast with the light behind it. Zooming in reveals the culprit.

Now the purple-pink glow on the edge of my face is obvious. That’s chromatic aberration, specifically purple fringing. In this case it’s probably being caused by that very bright window behind me sending bright light past my face, which is being lit by comparatively dim light from the room (no flash).

In Lightroom, the fix is simple. While in Develop mode, expand the Lens Corrections box, and choose the Color sub-menu. From there, we’re just two clicks away from a clean photo. Simply check the Remove Chromatic Aberration box and set the amount to 5 (or whatever makes the purple color go away).

Tada! The photo is much better now. You can play with the other sliders to get the exact desired correction you’re looking for, but I find that most of the time those two steps are all I need.

And when we zoom back out we’ll find that the harsh contrast between my face and the window is much less with the chromatic aberration removed.

So that’s it. A check box and a slider and we’re done. It’s just one of many ways Lightroom makes quick photo touchups easy.

What I Want to Accomplish

I want my setup to enable the following:

1. Mobile Photo Management: To be clear, when I say “mobile,” I don’t mean smartphone. I take my Canon 5D Mark III everywhere and like to transfer photos to my Apple MacBook Pro immediately. It’s important that I be able to do whatever I need to do no matter where I am, i.e., I don’t want to be tied to my office.
2. Large Photo Archive: Photos (especially RAW photos) consume a lot of disk space. I want a file server to store any photos that I’m not actively working on. It has to be huge and scalable.
3. Passive Transport: I want my files to get to my file server as quickly as possible, but I don’t want it to be an active part of my workflow, e.g., I don’t want to get stuck waiting on a FTP program to finish its batch before I can disconnect from a wifi spot or put my computer to sleep. I’d rather it just happen when the opportunity arises.
4. One-Time, Offsite Backup: Once my files are on the file server, I want them to back up from the server only (i.e., I don’t want my laptop doing a second backup). Also, as I’ve noted before, I like offsite backup of my photos. A local copy isn’t good enough.
5. Accessibility. When home, I want to easily access all of my photos, whether they’re on my laptop or in the archive.

Sound like a lot to accomplish? It turns out it’s pretty easy to do.

The Right Tools For The Job

I use 4 tools for the following purposes:

1. Mobile Photo Management: Lightroom 4.3
2. Large Photo Archive: Linux File Server (though any OS will do as long as it meets your storage needs … more later)
3. Passive Transport: Dropbox
4. One-Time, Offsite Backup: Crashplan
5. Accessibility: Lightroom 4.3

Now let’s walk through each.

Mobile Photo Management

I use an Apple MacBook Pro and Adobe Lightroom for just about all of my photo work. The MacBook Pro is really just personal preference. Any laptop with decent specs running OS X or Windows (needed for Lightroom) will work just fine.

As for Lightroom, there are endless articles online describing the strong points of what it lets you do with your photos. I won’t reiterate those points, but will simply say it’s fantastic both for viewing and processing your photos. It’s a perfect on-the-road application that lets you quickly import, tweak, and export your photos to wherever you intend to share them.

What I will focus on is how it integrates with the other needs I listed, so let’s move on to those areas.

Linux File Server for Large Photo Archive

I essentially did 2 things:

1. Built a 5 TB RAID Array – all the details you need are on that link
2. Configured an Apple File System service on the Linux server using netatalk and avahi

I won’t go into details of those applications as there are plenty of other articles online documenting what they are and how to set them up, but the gist is that netatalk enables the sharing of a folder on your server to OS X, and avahi advertises it on your local network so that you see it in Finder. The key configuration after installing each is:

For netatalk, in /etc/netatalk/AppleVolumes.default:

"Media" -tcp -noddp -uamlist uams_guest.so,uams_dhx.so,uams_dhx2.so -nosavepassword

And in /etc/netatalk/afpd.conf:

# Access The Storage Area On Your Server
/mnt/store/private/photos "Photos" allow:markmcb

For avahi, in /etc/avahi/services/media_group.service:

<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="no">Media</name>

<service>
<type>_afpovertcp._tcp</type>
<port>548</port>
</service>

<service>
<type>_device-info._tcp</type>
<port>0</port>
<txt-record>model=RackMac</txt-record>
</service>

</service-group>

If you get that all set up, you should see something like the following in Finder (using what I listed, you’d only see the “Photos” folder).

OS X accessing a Linux file server via Finder.

If that all made sense, the rest of this steps in this article are quite easy in comparison. :)

Passive Transport

First, if you’re not already using Dropbox, check it out. The concept is simple: for every device you install Dropbox on there is a Dropbox folder. If you put a file in the Dropbox folder on device A, it uploads to Dropbox and downloads to devices B, C, D, etc., automatically. In other words, it’s a folder that syncs across devices. The key here is that it does this automatically, i.e., you don’t tell Dropbox to start a transport, it just does it once a file is saved and an Internet connection is available.

All we need to do is:

1. Install Dropbox on the laptop
2. Install Dropbox on the Linux server
3. Create a folder somewhere in Dropbox for our active photos
4. Add that folder to the folders list in Lightroom

For me, #3 is ~/Dropbox/Photos/Master Workspace, and when added to Lightroom it looks like:

As you can see, I have 313 active photos in this workspace. As soon as I import new photos into Lightroom, Dropbox will start to upload them. Let’s say I take 100 photos and import them from my hotel room. As soon as I import them, the upload starts. Let’s say it’s a slow connection and 20 have finished when my friend calls and wants to meet up for lunch. With Dropbox, I just pack up my laptop and go. Whenever I open it next and get on the net, it will resume. In the mean time, those 20 photos that transferred are being downloaded to my Linux server that’s always online. Once the server gets the copy, it backs them up (more on that in the next section). It’s important to note that the backup process is a separate one. While Dropbox offers some backup characteristics, you really shouldn’t depend on it as a backup solution.

The benefit here is that, when I’m away for a few days or weeks, my photos automatically make their way to a safer place on my server. In general I like to have my laptop be a “throw-away laptop.” That’s not to say I’d be happy if I lost it, but I like knowing that data loss from such an event would be minimal or none. Dropbox helps a lot in making that true (and not just for photos).

One-Time, Offsite Backup

Dropbox has now sent the files to my Linux server’s Dropbox folder. This is one of two places my photos reside in this process. Once I am done actively working with them, they will move to the archive folder that we shared earlier.

I use Crashplan for backup. I’ve been using it almost two years now and it’s really a tremendous service for the price. One of the cool things it does is smartly determine if a file needs to be backed up, or if you’ve already backed it up before and a copy is on their servers. This is critical for the workflow I’m describing. Here’s what happens:

1. From my remote location, I import photos to Lightroom
2. Lightroom stores the photos in my laptop’s Dropbox folder, let’s call it D1
3. D1 is uploaded to Dropbox
4. The Dropbox folder on my server, D2, gets the files pushed down
5. Crashplan is watching D2 and immediately starts backing up the files to the offsite backup destinations I’ve designated.
6. Crashplan is also watching the archive folder, A1. In the next section we’ll use Lightroom to move files from D1/D2 to A1. The nice thing Crashplan does is realize that the files being moved have already been backed up and simply updates some file pointers.

That last step is HUGE if you work with RAW files like I do. When you need to transfer 500 RAW files at 30+ MB each, the last thing you want to be doing is chewing up more bandwidth than you have to. Granted, you’re already moving the files twice: once through Dropbox, and once through Crashplan. However, if you started the process remotely it’s not bandwidth you’re going to miss, i.e., all or some of this happens when you’re away and is generally finished when you sit down at your home computer to work. I really, really, really love this aspect of this workflow.

Accessibility

The last point I’ll touch on is accessibility. Once you’re home and ready to do some serious post-processing, you want to quickly get to all of your files regardless of where they’re physically stored. Lightroom 4 is a dream in this regard. In the same way we added the Master Workspace folder above, all we need to do is add the Master Archive folder on the Linux server that we made available via Finder. Once you’ve added the folders, your catalog acts as the brain and even if you forget to mount your shared drive, Lightroom give you intelligent feedback:

Forgot to mount? Lightroom shows question marks indicating it can’t find the folder:

Not a problem, simply open finder and click on the shared drive to “mount” it. Mounted:

And that’s it. Now you can click on any of those folders and work on them as if they were all stored on your computer.  If you drag a photo or folder from the Master Workspace to the Master Archive, then files move to their new home on the server and off your laptop and out of Dropbox (this is really good since even a “pro” Dropbox account only stores 100 GB of data).

Wrap-Up

As with anything this complex, there are a bazillion ways to do it. I have tinkered with quite a few setups and this one is where I ended up. I really like the flexibility, the saved bandwidth, the immediacy of backup, and the ease of access. I hope this helps someone, and if you’ve got variations of this, I’d love to hear about them.

Lossy capture using JPG:

Lossless capture using PNG:

You can probably see the difference quite clearly. The jpg looks blotchy and the color of red has been distorted. This is because the JPG format will degrade image quality to produce a smaller file. Given that we’re talking 9 vs. 13 KB in this case, the savings don’t really matter. What does matter is that the image looks terrible and makes you look oh-so-non-tech-savvy.

The solution is quite simple if you’re using OS X. Simply change the default format for screen captures. Launch the Terminal app and issue the following commands:

$ defaults write com.apple.screencapture type png
$ killall SystemUIServer

That’s it! All of your screen captures will now be in the much cleaner PNG format. If you decide you’d rather have JPG again, just replace “png” with “jpg” in that first command.

-- AppleScript to open and control movie files in QuickTime Player X

-- The script is meant to run at the command line, like so:
-- $ osascript /path/to/movie.mp4 movie.mp4 800 60
-- If you don't want to use a shell, just remove this and 
-- the last line and manually populate the first 4 variables
on run argv
	
	set movieFile to item 1 of argv -- the full path with movie filename
	set movieName to item 2 of argv -- the file name only of the movie
	set desiredWidth to item 3 of argv -- the width of the movie on the screen
	set startOffset to item 4 of argv -- how many seconds into the movie to start
	
	-- this finds the right side of the screen, then offsets to the left
	-- the desired width of the movie
	tell application "Finder"
		set desktopBounds to bounds of window of desktop
	end tell
	set rightX to item 3 of desktopBounds
	set leftX to (rightX - desiredWidth)
	
	tell application "QuickTime Player"
		
		-- open the movie and bring it to the forefront (if other movies are open)
		open movieFile
		set lastOpenedWindow to (first window whose name contains movieName)
		set visible of lastOpenedWindow to true
		
		-- resize the movie using its original aspect ratio
		set movieBounds to bounds of lastOpenedWindow
		set widthOrigin to leftX
		set heightOrigin to 1
		set originalWidth to ((item 3 of movieBounds) - (item 1 of movieBounds))
		set originalHeight to ((item 4 of movieBounds) - (item 2 of movieBounds))
		set calculatedHeight to (round ((desiredWidth / originalWidth) * originalHeight) rounding down)
		set desiredWidth to (desiredWidth + widthOrigin)
		set calculatedHeight to (calculatedHeight + heightOrigin)
		set the bounds of lastOpenedWindow to {widthOrigin, heightOrigin, desiredWidth, calculatedHeight}
		
		-- if opening multiple movies, ensure they don't overlap
		set slideDown to 1
		repeat with aWindow in (get every window)
			set boundsOfAWindow to bounds of aWindow
			set heightOfAWindow to item 4 of boundsOfAWindow
			if (heightOfAWindow > slideDown) then
				if (name of aWindow does not contain movieName) then
					set slideDown to (heightOfAWindow + 1)
				end if
			end if
		end repeat
		set the bounds of lastOpenedWindow to {widthOrigin, (heightOrigin + slideDown), desiredWidth, (calculatedHeight + slideDown)}
		
		-- note, here we select a document, not a window
		set myMovie to document 1
		tell myMovie
			set audio volume to 0
			set current time to startOffset
			play
			activate -- makes QuickTime Player the frontmost application
		end tell
		
	end tell
end run

… a while back … you wrote about cloud back ups and I believe you recommended Crash Plan. I tried the free trial but was disappointed because it was SO slow. I like drop box, but couldn’t figure out a way to get it to automatically back up my whole computer like time machine did. I tried hooking up my old time capsule that we used to use as a router directly to my computer via ethernet, but I couldn’t get that to work. SO, I thought maybe it was just a random thing that Crash Plan was so slow the first time, so I signed up again in desperation. For 150+ gb, it says over 57 days?!? Do you think that seems right?

Do you have any other recommendations? I want something that worked like time machine did, preferably cloud based. I’m curious what you’re using.

The following is what I responded with via email, but I’ve actually gotten this question from a few people lately, so I thought I’d post it here.

The first fundamental question to ask is: what do you want when you say “backup”?

I’d say there are a few key criteria:

1. A true backup – nothing¹ you do can destroy your data
2. A fast, but at-risk backup – you’ve got a copy of everything, but you could destroy it
3. Sync – you can get to your files on any device with minor accident tolerance

CrashPlan is all #1 and #2, and a partial #3.  Time Capsule is #2.  Dropbox is #3.

CrashPlan is as fast as your upload bandwidth to the Internet, which in general is slow.  The key with CrashPlan is that it’s:

1. Unlimited storage for a super cheap rate.  I have almost 2 TB with them and it’s still less than $5 a month.  For Google Drive the same storage is $100/mo.  Dropbox is usually 4x Google, so while they don’t list the price for this, I assume it’d be $400/mo.  Big deal breaker.
2. It’s true backup.  First it’s off-site, so if your house burns down, someone steals your Time Capsule, or if any other catastrophe occurs, your data is still backed up.
3. If you have another computer with a large hard drive, you can have CrashPlan do super fast local backups.  For example, I back everything up to their cloud service, but I also back it up to my Linux server with a 5 TB drive.  If my laptop dies, I can quickly get my data from my local server.  If my house burns down, I can get my data from the cloud at a speed that corresponds to the speed of my Internet connection.

A key thing to note with Dropbox is that it is not backup.  If I borrow your computer, delete your photos folder, and you don’t realize for a few weeks it’s just gone.  No recovery.  Dropbox does keep track of recently deleted files and allows you to restore them, but it’s unpredictable in terms of how long they’ll be available.  That feature is more for a “whoops, I didn’t mean to hit delete” situation.  Not really a true backup.  To contrast, you can tell CrashPlan to never delete backup copies of a folder like “Photos,” even if you delete it on your computer.

If you want true backup, I haven’t seen anything better than CrashPlan for the price.  The initial backup will take some time, but I wouldn’t worry about it too much.  If it’s a real issue, I’d check on your Internet plan and see if there’s an affordable faster plan.

Personally, I use all 3.  :-)

1. Dropbox is my sync solution.  I see it as a quick portal between computers.  I only have 8 GB on my plan though, so file size is limited.

2. My Linux box acts as an Apple Time Capsule.  If I want to go “back in time” in some folder, I use this.  (If disk space ever became an issue, I’d remove this.  It’s really overkill with the local half of #3 below.)

3. CrashPlan also runs on my Linux box and I pay for the “Pro” service.  It’s my ultimate fallback.  It is slow, but it’s the only true backup I have.

Ultimately it comes down to choosing what you want/need.  I’ve got tens of thousands of photos at this point that I’d hate to lose.  Moreover, I’m shooting RAW now and my photos come in around 30 MB each.  I’d hate to pay Dropbox-like fees to backup.  CrashPlan backs those (and everything else) up for a great price.

Notes

1. When I say “nothing” I mean that it’s unlikely that a non-deliberate scenario would play out that would destroy data. Clearly, if you log into your backup account and say “delete my account” and then smash your computer, your data is likely gone forever. I’m not considering these unlikely and deliberate scenarios. ↑ back.

One of my “things to do before I’m 50” is

This should read:

One of my “things to do before I’m 50” is

It turns out these are just special characters that were improperly encoded for utf-8. The fix is simple enough: loop through your content and replace where needed.

If your database is big, this could take a long time unless you disable callbacks. The script below highlights both how to replace the characters using Ruby and how to disable your Rails callbacks to make this script run in seconds instead of hours (depending on the complexity of your callbacks).

replacements = []
replacements << ['…', '…']           # elipsis
replacements << ['–', '–']           # long hyphen
replacements << ['’', '’']           # curly apostrophe
replacements << ['“', '“']           # curly open quote
replacements << [/â€[[:cntrl:]]/, '”'] # curly close quote
klasses = [Comment, Article]           # replace with relevant classes
 
klasses.each do |klass|
  klass.all.each do |obj|
    original = obj.body
    replacements.each{ |set| obj.body = obj.body.gsub(set[0], set[1]) }
    unless (original == obj.body)
      #### Remove or Customize ####
      # This should reflect your models' callbacks.  It should be safe
      # since we're just doing a simple find/replace.
      Comment.skip_callback(:save, :after,  :do_after_save_tasks ) if obj.is_a?(Comment)
      Article.skip_callback(:save, :before, :do_before_save_tasks) if obj.is_a?(Article)
      #### End Remove or Customize ####
      obj.save! 
    end
  end
end

If you noticed, I used a regular expression for the curly close quote. This is because there is an invisible control character that is not easily copy/pasted into your code. Using [[:cntrl:]] is just an easier way to catch it.

class IntegerCompressor
  
  CompressionCharacterSet = %w(0 1 2 3 4 5 6 7 8 9
  A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
  a b c d e f g h i j k l m n o p q r s t u v w x y z)
  
  def self.to_base
    CompressionCharacterSet.length
  end

  def self.compress(number_to_convert)
    digits_needed = Math.log(number_to_convert, IntegerCompressor.to_base).floor + 1
    compressed_number_string = ''
    previous_remainder = number_to_convert
    (digits_needed-1).downto(0) do |power|
      r=previous_remainder.divmod(IntegerCompressor.to_base**power)
      compressed_number_string << CompressionCharacterSet[r[0]]
      previous_remainder = r[1]
    end
    compressed_number_string
  end
  
  def self.decompress(compressed_number)
    power = 0
    base_10_integer = 0
    compressed_number.to_s.reverse.each_char do |digit|
      base_10_integer += ((IntegerCompressor.to_base**power)*CompressionCharacterSet.index(digit))
      power+=1
    end
    base_10_integer
  end
  
end

It seems to work:

$ irb
>> require '/path/to/file/integer_compressor.rb'
=> true
>> IntegerCompressor.compress 1234567890
=> "1LY7VK"
>> IntegerCompressor.decompress "1LY7VK"
=> 1234567890

If anyone has a more elegant solution, I'd be curious to see it.

(Thanks to Tamara for the log refresher.)