Managing A Large Photo Library With Lightroom, Dropbox, and Crashplan

Late last year I made the switch to Adobe’s Lightroom 4 from Aperture and Picasa for a variety of reasons, but a big one was photo/file management. As my photo collection got larger I found that I’d need more than my laptop’s hard drive to store them all. Despite all the things I liked about Aperture and Picasa, both had clunky file management and the burden was enough to make me consider alternatives. I’m now very happy with the workflow described below using Lightroom, Dropbox, Crashplan, and a Linux file server.

What I Want to Accomplish

I want my setup to enable the following:

  1. Mobile Photo Management: To be clear, when I say “mobile,” I don’t mean smartphone. I take my Canon 5D Mark III everywhere and like to transfer photos to my Apple MacBook Pro immediately. It’s important that I be able to do whatever I need to do no matter where I am, i.e., I don’t want to be tied to my office.
  2. Large Photo Archive: Photos (especially RAW photos) consume a lot of disk space. I want a file server to store any photos that I’m not actively working on. It has to be huge and scalable.
  3. Passive Transport: I want my files to get to my file server as quickly as possible, but I don’t want it to be an active part of my workflow, e.g., I don’t want to get stuck waiting on a FTP program to finish its batch before I can disconnect from a wifi spot or put my computer to sleep. I’d rather it just happen when the opportunity arises.
  4. One-Time, Offsite Backup: Once my files are on the file server, I want them to back up from the server only (i.e., I don’t want my laptop doing a second backup). Also, as I’ve noted before, I like offsite backup of my photos. A local copy isn’t good enough.
  5. Accessibility. When home, I want to easily access all of my photos, whether they’re on my laptop or in the archive.

Sound like a lot to accomplish? It turns out it’s pretty easy to do.

The Right Tools For The Job

I use 4 tools for the following purposes:

  1. Mobile Photo Management: Lightroom 4.3
  2. Large Photo Archive: Linux File Server (though any OS will do as long as it meets your storage needs … more later)
  3. Passive Transport: Dropbox
  4. One-Time, Offsite Backup: Crashplan
  5. Accessibility: Lightroom 4.3

Now let’s walk through each.

Mobile Photo Management

I use an Apple MacBook Pro and Adobe Lightroom for just about all of my photo work. The MacBook Pro is really just personal preference. Any laptop with decent specs running OS X or Windows (needed for Lightroom) will work just fine.

As for Lightroom, there are endless articles online describing the strong points of what it lets you do with your photos. I won’t reiterate those points, but will simply say it’s fantastic both for viewing and processing your photos. It’s a perfect on-the-road application that lets you quickly import, tweak, and export your photos to wherever you intend to share them.

What I will focus on is how it integrates with the other needs I listed, so let’s move on to those areas.

Linux File Server for Large Photo Archive

I essentially did 2 things:

  1. Built a 5 TB RAID Array – all the details you need are on that link
  2. Configured an Apple File System service on the Linux server using netatalk and avahi

I won’t go into details of those applications as there are plenty of other articles online documenting what they are and how to set them up, but the gist is that netatalk enables the sharing of a folder on your server to OS X, and avahi advertises it on your local network so that you see it in Finder. The key configuration after installing each is:

For netatalk, in /etc/netatalk/AppleVolumes.default:

"Media" -tcp -noddp -uamlist uams_guest.so,uams_dhx.so,uams_dhx2.so -nosavepassword

And in /etc/netatalk/afpd.conf:

# Access The Storage Area On Your Server
/mnt/store/private/photos "Photos" allow:markmcb

For avahi, in /etc/avahi/services/media_group.service:

<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="no">Media</name>

<service>
<type>_afpovertcp._tcp</type>
<port>548</port>
</service>

<service>
<type>_device-info._tcp</type>
<port>0</port>
<txt-record>model=RackMac</txt-record>
</service>

</service-group>

If you get that all set up, you should see something like the following in Finder (using what I listed, you’d only see the “Photos” folder).

Finder

OS X accessing a Linux file server via Finder.

If that all made sense, the rest of this steps in this article are quite easy in comparison. :)

Passive Transport

First, if you’re not already using Dropbox, check it out. The concept is simple: for every device you install Dropbox on there is a Dropbox folder. If you put a file in the Dropbox folder on device A, it uploads to Dropbox and downloads to devices B, C, D, etc., automatically. In other words, it’s a folder that syncs across devices. The key here is that it does this automatically, i.e., you don’t tell Dropbox to start a transport, it just does it once a file is saved and an Internet connection is available.

All we need to do is:

  1. Install Dropbox on the laptop
  2. Install Dropbox on the Linux server
  3. Create a folder somewhere in Dropbox for our active photos
  4. Add that folder to the folders list in Lightroom

For me, #3 is ~/Dropbox/Photos/Master Workspace, and when added to Lightroom it looks like:

As you can see, I have 313 active photos in this workspace. As soon as I import new photos into Lightroom, Dropbox will start to upload them. Let’s say I take 100 photos and import them from my hotel room. As soon as I import them, the upload starts. Let’s say it’s a slow connection and 20 have finished when my friend calls and wants to meet up for lunch. With Dropbox, I just pack up my laptop and go. Whenever I open it next and get on the net, it will resume. In the mean time, those 20 photos that transferred are being downloaded to my Linux server that’s always online. Once the server gets the copy, it backs them up (more on that in the next section). It’s important to note that the backup process is a separate one. While Dropbox offers some backup characteristics, you really shouldn’t depend on it as a backup solution.

The benefit here is that, when I’m away for a few days or weeks, my photos automatically make their way to a safer place on my server. In general I like to have my laptop be a “throw-away laptop.” That’s not to say I’d be happy if I lost it, but I like knowing that data loss from such an event would be minimal or none. Dropbox helps a lot in making that true (and not just for photos).

One-Time, Offsite Backup

Dropbox has now sent the files to my Linux server’s Dropbox folder. This is one of two places my photos reside in this process. Once I am done actively working with them, they will move to the archive folder that we shared earlier.

I use Crashplan for backup. I’ve been using it almost two years now and it’s really a tremendous service for the price. One of the cool things it does is smartly determine if a file needs to be backed up, or if you’ve already backed it up before and a copy is on their servers. This is critical for the workflow I’m describing. Here’s what happens:

  1. From my remote location, I import photos to Lightroom
  2. Lightroom stores the photos in my laptop’s Dropbox folder, let’s call it D1
  3. D1 is uploaded to Dropbox
  4. The Dropbox folder on my server, D2, gets the files pushed down
  5. Crashplan is watching D2 and immediately starts backing up the files to the offsite backup destinations I’ve designated.
  6. Crashplan is also watching the archive folder, A1. In the next section we’ll use Lightroom to move files from D1/D2 to A1. The nice thing Crashplan does is realize that the files being moved have already been backed up and simply updates some file pointers.

That last step is HUGE if you work with RAW files like I do. When you need to transfer 500 RAW files at 30+ MB each, the last thing you want to be doing is chewing up more bandwidth than you have to. Granted, you’re already moving the files twice: once through Dropbox, and once through Crashplan. However, if you started the process remotely it’s not bandwidth you’re going to miss, i.e., all or some of this happens when you’re away and is generally finished when you sit down at your home computer to work. I really, really, really love this aspect of this workflow.

Accessibility

The last point I’ll touch on is accessibility. Once you’re home and ready to do some serious post-processing, you want to quickly get to all of your files regardless of where they’re physically stored. Lightroom 4 is a dream in this regard. In the same way we added the Master Workspace folder above, all we need to do is add the Master Archive folder on the Linux server that we made available via Finder. Once you’ve added the folders, your catalog acts as the brain and even if you forget to mount your shared drive, Lightroom give you intelligent feedback:

Forgot to mount? Lightroom shows question marks indicating it can’t find the folder:

Not a problem, simply open finder and click on the shared drive to “mount” it. Mounted:

And that’s it. Now you can click on any of those folders and work on them as if they were all stored on your computer.  If you drag a photo or folder from the Master Workspace to the Master Archive, then files move to their new home on the server and off your laptop and out of Dropbox (this is really good since even a “pro” Dropbox account only stores 100 GB of data).

Wrap-Up

As with anything this complex, there are a bazillion ways to do it. I have tinkered with quite a few setups and this one is where I ended up. I really like the flexibility, the saved bandwidth, the immediacy of backup, and the ease of access. I hope this helps someone, and if you’ve got variations of this, I’d love to hear about them.

22 thoughts on “Managing A Large Photo Library With Lightroom, Dropbox, and Crashplan

  1. Nice article, but i have a few questions. Is there a reason thatyou use drop box and crash plan? Isn’t drop box alone sufficient?

    • Good question. For a longer answer, see this post: http://markmcb.com/2012/06/10/understanding-offsite-backup-vs-local-backup-vs-syncing/

      In general, there are 2 big reasons Dropbox isn’t a good backup solution. 1) It’s got a max capacity and after that it starts selectively storing things. Let’s say you have the 100 GB Pro account like I do. With nothing but my most recent photos, I’m using 70% of my storage space. My photo archive right now consumes 220 GB. In the event of I lost all my devices, something would get lost as my nearly 300 GB of data simply won’t fit in my Dropbox account. 2) Let’s say you accidentally delete your photos and don’t realize it for 2 months. Unless you’ve paid for the special packrat feature, they’re gone as Dropbox only keeps deleted files for 30 days. (More info: https://www.dropbox.com/help/11/en).

      With Crashplan, there’s no limit to storage and if you delete a folder and realize it a year later your files will be there (assuming you’ve told it to never delete files). To be fair, if you have very few files and also pay for the Dropbox packrat feature, then it could be used as backup, but I still would argue that it’s not your best option as Crashplan has far more backup specific features (like multiple backup destinations for various recovery options).

      I hope that helps.

  2. I currently use Dropbox to backup my whole family picture collection, plus I do local backups to HDD in fireproof safe.

    I do not have packrat feature (I have 100GB pro + 120GB bonus space), but its not 100% needed. You can use the RSS feed of dropbox to keep an eye for random file modifications. I also like how automatic it is for backup, when I take a picture using my Android phone, it automatically backs up to Dropbox as well. LAN sync then grabs the file over wifi and transfers to my HTPC.

    • I agree, it’s quite handy. However, I would think of backup like this: if you can easily do something stupid and destroy your files, then it’s not backup. If you rely on Dropbox without packrat and delete a folder without realizing it and don’t catch it in your RSS feed, after 30 days it’s gone. A true backup doesn’t have that level of risk.

      Regarding the safe, it may not burn but it will get up to about 1,100ºF (~600ºC) in a structure fire. The non-operating temperature range for common HDDs is -40° F to 158° F (-40° C to 70° C). In a fire, those backups are toast.

    • Not really. Lightroom has publishing features (and plugins) and even syncs some meta data with some services (e.g., comments via flickr can be pulled in). I used to like the quickness of publishing with Picasa, but I think the G+ interface has evolved enough that dragging/dropping to the web page directly is just as easy. I suppose it just depends on the specific features you value most.

  3. Hello,

    Thanks to share you method. Photographic workflow is a bit a headache.

    Good idea to separate you daily process, from the archive folder.
    Indeed if you use your dropbox folder for other things, it allow you not to overload it, and get you full archives downloaded in every computer. Also dropbox get a bit expansive if the size of the files being managed raise.
    I can add to your post another DAM book technics which is to separate the archive in three buckets :
    - the archive (jpeg from the camera, NEF)
    - the original (ideally one unique DNG file per file)
    -the finalized JPEG exports for prints, PSD.

    It allow lightroom not to scan a same file twice, and immediately identify years after the working copy

    Have you experience with crashplan, and specificallt, what does it do when we need to rename a folder full of files. I think now Dropbox make as if it was a delete and brand new files to be synchronised.
    As my full image collection takes 125 Days to upload to crashplan, I hope it’s not like that ?

    Sébastien Lucas
    Photographer and Funder of http://www.archiref.com
    The visual source of architecture

    • Sébastien, Crashplan is really smart. If you move/rename a folder, it is aware and reacts in an efficient manner. Let’s say you move a folder with 1,500 files. You’ll see the status bar on the screen note that 1,500 files are waiting to be backed up. Once the backup starts, you’ll see it rapidly (i.e., seconds) go through them (I assume it’s just validating the structure hasn’t changed). Nothing but meta data gets uploaded, so no worries if you move your stuff.

  4. Hi, thanks for this article ! Just trying to be sure I understand….

    Conceptually you have 2 folders on your file server called “workspace” and “archive”. On the file server, both are backed up with Crashplan while “workspace” folder is also synced with Dropbox.

    On your laptop, you mount the “archive” folder but not the “workspace” folder. For the latter, you create a “workspace” folder on your laptop’s local HDD and you sync that with Dropbox.

    Is that right ?

    Also, is it possible to use a network attached storage solution instead of doing the the full server route ? I suppose that would really depend on what Crashplan supports?

    At the end of the day, then, you have access to all of your photos in the “workspace” folder from all of your mobile devices while the archived ones are only accessible at home. Or does Crashplan let you look at files via the web or on other devices?

    Thanks !

    • Yes, it sounds like you’ve got the gist of it. You could use NAS, but you’ll probably need another system to run CrashPlan (I haven’t seen it on the list of plugins for any out-of-the-box NAS systems).

      Yes on the mobile access too. CrashPlan does let you dig through your backup via mobile. Good luck!

  5. So i am new to the mac world.
    I have always used Picasa for quick view of my photos and for sharing albums with my family. and photoshop or photoshop elements to edit major edits. Picasa just picks up on any changes you make on your photo or photo name or folder, the changes are made automatically and show up when you open Picasa (I hope it will do the same on my iMac as it did on my PC).
    now moving to iMac (and loving how smooth everything is) but very disappointed with iPhoto and really thinking of downloading Picasa. I basically want my photos and edited photos to be in the same spot and have a place to quickly view my photos. when i tired using iPhoto thinking it would be like Picasa a place to view my photos and keep them in their original folder and when editing them in iPhoto or Photoshop it would show up in the viewing program, but it does not. (someone told me that Aperture will do that, but i don’t want to spend the money just to find out it does not.)
    So basically i want my photos displayed from their original folder location (as they do in Picasa) and if i make any kind of editing for it to show up in my “viewing program”
    Lightroom is more expensive and seemed harder to learn the Aperture from the little research that i did.
    Thanks in advance!

    • I got Aperture and i am in no better place then i was with iPhoto, I just wanted the organization of Picasa and a strong editor all in one….. A program that will work with my photos from their original folder. I guess you either do not have an answer for me or do not wish to deal with my questions. which ever one it is, could you please remove my questions from your site.
      Sorry for the bother
      Thank you, Orly

      • Sorry for the delay. I sometimes get behind on replying to comments. If sync is your absolute top priority, then I haven’t seen anything better than Picasa. That said, I think if you experiment with the export/publish features in Lightroom, you’ll find they offer close to the same features. It really comes down to what’s more important. While I agree it’d be nice if I could better keep my Lightroom library in sync with social media, it’s a tricky problem to solve as I tend to publish to more than one place. The big assumption with Picasa is that you want everything in Picasaweb/Google+. I’m not sure that’s a fair assumption for most folks, i.e., I publish to flickr, Google+, 500px, etc. It’s easy to push to those 3 sites, but having all 3 sync back to 1 source would be tricky. That’s where the publishing comes in handy. Good luck!

        • Thanks for responding. I actually thought that you only respond to people you know and that I stumbled on your page by mistake which is why I asked to be removed. So, no I don’t share my photos from Picasa to anything but an album I created for my dad to view since he lives overseas. I don’t even used Google+.
          I have always used Picasa for quick view of my photos and to see what needs to be edited with seeing all the photos in one spot. The reason I like it sooo much it’s because of the sync capabilities within the hard drive. Whatever changes I make in Photoshop Picasa just picks (Syncs my folders).
          So basically i want my photos displayed from their original folder location (as they do in Picasa) and if i make any kind of editing for it to show up in my “viewing program”. I got Aperture and i am in no better place then i was with iPhoto, I just wanted the organization of Picasa and a strong editor all in one….. A program that will work with my photos from their original folder.
          I tried Lightroom and it did not do that. Did I miss something?
          Lightroom did not seem to be the “two in one” that i am looking for. Picasa basically just mirrors my hard drive. Can you tell me if there is a program out there that is two in one? editor and an organizer that mirrors the original files and their changes?

          • You can do this to some degree using XMP, plugins, or publishing, but it’s not quite as simple as what you’re used to in Picasa.

            If you really want to use Lightroom and don’t want to get into all the details, a simple alternative might be to set up a saved export setting. Let Lightroom manage your originals (i.e., you never touch the files directly) and after you edit them, you export them to some folder. If you do more edits, just export and overwrite.

            Keep in mind, there are good reasons Lightroom and Aperture don’t behave like Picasa by default. Lightroom/Aperture are more for a professional photographer’s workflow.

  6. So i tried Lightroom, and it’s not for me. I guess i am used to Photoshop for the editing (but do not like it as an organizer. Lightroom did not seem to be the “two in one” that i am looking for. So i guess i have to continue to used Picasa as my viewer and organizer. It basically just mirrors my hard drive. Can you tell me if there is a program out there that is two in one? Great editor and an organizer that mirrors the original files and their changes?

    • Hello Mark,

      I used to be a professional photographer working for a local paper, I would edit photos in Photoshop, then PS4 and all changes could be saved in my original folder on the hard drive. I since then changed professions and now do it as a hobby. When I worked with PC or Mac back then, I would save all photos on my hard drive in folders, in order to view all my photos at one time I used to use ACDsee and when ever I would edit anything in Photoshop PS4 then PS5, I would save it either by replacing the original image or making a new version, it would reflect back for me to view. Jump a few years and then I used the combo of Picasa to view and Photoshop to edit, again any changes I would make in Photoshop would then reflect and show up in my viewer, which is Picasa.

      Fast forward a few years, LOL and I now own an iMac (three months now) and wanted to have a two in one. Everyone said get a mac and Aperture will do what you want, it does not. I would like Editor and viewer, but to still see all changes reflect in my original folders on my hard drive. No export import.

      I first tried Photoshop Elements to organize and that didn’t work. So I tried Lightroom and the same. Everyone said that Aperture will be two in one, edit and organize. And I would even be able to choose Photoshop as an editor. That is true BUT when working in Referenced Library then the pictures do not reflect back in the project. I would then have to go back to the original folder and import that file into my project. (way too many steps)

      WORK FLOW I would love to have is a program that mirrors my folders before and after I edit them in Photoshop, that is all. Weather I “Replace” the image with the changes or choose to make a new version with the changes. I want to see everything in one folder! All changes, and I can just view it. All the other fun stuff like books, webshare, slide show all the fun stuff can be done with all of them. But I want one that is “two in one” View, Edit and then syncs it back to my folders without me playing a cat and mouse game of import, export and so on……………

      Again, thanks for your time!

  7. Mark-
    You’re my hero. Thank you for posting your work flow. Wise man and congrats on your baby.
    I am about to sail around the globe with family and friends. Ideally we will document photo and video with DSLR and IphonesIpads(just the way it is). We will likely shoot footage and then edit on the computer and then upload into blogs and Dropbox/backup (as you have described) from internet cafes once we come into port.
    Might there be a way to remove the laptop from the equation with the use of Adobe Creative Cloud?

    • Jesse, it sounds like you have quite the trip ahead of you. Good luck! As for Creative Cloud, it might help but I really can’t say as I’ve never tried it personally. If you try it and it works, I’d love to know.

  8. Mark – I’ve been searching for an efficient method of moving a substantial iPhoto library into Lightroom. I found your very informative explanations during my search. Asking specifically, I am ready to make the move to Lightroom, as I believe it will offer the flow and organization I need. Question is, do you have recommendations on how I can migrate my existing iPhoto library (actually two), into Lightroom? One library is located on an external hard drive and the second is currently on my Macbook Pro.

  9. Have you considered using Connected Data’s Transporter Sync instead of Dropbox? The upfront cost of a hard drive and the $99 Transporter Sync would let you have up to a 4TB drive attached (for only another $150). Then there are not other recurring charges. I don’t know if Crashplan will back up the drive connected to the Transporter or not.

  10. Thank you. I am switching to lightrom right now. My patience of waiting for aperture 4 is all used up. I was using various aperure libraries before and manualy backing them up to drives. But with my switch I am looking for better ways to do it. This helped.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>