Shutterfly’s Kodak moment in 5 billion frames
We snap photos these days without a thought and accumulate several hundred images in a flash, so to speak. And then we face the constant challenge of moving them from one device to another and making sure there’s enough storage and stability to keep them safe for years to come.
Well, Shutterfly faces that challenge Monday -- but on an exponentially more massive scale.
Massive, as in 5 billion images -- or 10 years’ worth of Kodak customers’ memories. Shutterfly is transferring the photo files from Kodak Gallery, which is closing down as of July 2.
In place of the single-terabyte hard drives we might use for our personal photo collections, “Think of a room filled with 10,000 hard drives,” said Geoffrey Weber, Shutterfly’s chief information officer. A room packed tightly from floor to ceiling, with heavy-duty 90-terabyte hard drives, that is.
Redwood City, Calif.-based Shutterfly acquired online gallery assets of bankrupt Eastman Kodak at the end of April for $23.8 million, including the lists of Kodak Gallery’s 68 million U.S. and Canadian customers and all of the photos they’ve stored and projects they’ve created over the years.
“We’ve done several of these migrations in the past, but nothing of this scale,” said Shutterfly General Manager Karl Wiley, who noted that the Kodak move will be 16 times larger than its migration of American Greetings was last year.
Taking advantage of its free unlimited storage, Shutterfly’s own customers have amassed a sizable database of 10 billion photos since the company’s founding in 1999.
But with the Kodak Gallery merging, the company will begin to ingest half of what it took more than a decade to build up on its own site in just a matter of months.
How much data do 5 billion photos equal? Well, Shutterfly executives are tight-lipped about that, but they did say it’s on the scale of multiple petabytes, or thousands of terabytes.
That’s comparable to data in the Library of Congress’ Web archiving project, which contains about 3 petabytes, according to a blog post by Leslie Johnston, who manages the technical architecture for the library’s digitizing project.
The trip of just a mile or so that these billions of photos will take started with numerous blind spots on a road map that a core team of 25 spent more than a month planning, Weber said. The biggest challenge was that they had to start with unconfirmed assumptions and weren’t able to coordinate with Kodak initially.
Shutterfly made what ended up being an uncontested bid for Kodak’s gallery assets in March, but the team responsible for moving the contents didn’t have access to the facilities, the proprietary storage system or even Kodak’s staff until after the deal closed nearly two months later.
They didn’t even know how many images the move would involve, what they’d find in the Kodak data center or what kind of support -- if any -- they’d get from bankrupt company, Weber said.
But the six-week silence did offer them a lot of time to plan, exploring all of the possible and extreme scenarios -- could someone spoof accounts, for example -- to be prepared for anything they might encounter, he said.
With the responsibility of maintaining and safeguarding the precious and poignant memories of a few million customers, moving data is something that Shutterfly’s tech team thinks about -- a lot. They test and challenge their system for uploading through the “front door,” the way consumers get their photos on the site.
Going through the front door wasn’t going to work this time, though, Weber said. “We penciled that out as something that would take two years -- or we could spend a lot of money to do it differently.”
Instead, they decided to do file-by-file copying -- 5 billion images. The process will take months, not hours, and some serious storage.
On Monday, Kodak’s facility will be aglow with stack after stack after stack of Shutterfly’s signature bright orange, 7-inch, rack-mounted hard drives heating up and humming as the files shoot over into the servers.
These servers, which come in empty except for some proprietary security software to encrypt all the data they gather, will leave brimming with years’ worth of priceless snapshots and irreplaceable moments.
How does this affect you if you’re one of 68 million Kodak Gallery users moving to Shutterfly? It actually shouldn’t.
All of this carefully orchestrated digital gymnastics, technically, should have absolutely no effect on customers or their access to the photos. Several weeks ago, Shutterfly began shifting the metadata for the billions of photos on Kodak’s servers. This information provides the architecture for a bridge between systems, pointing former Kodak gallery users to the right place on the Kodak site, but through Shutterfly.
As the storage bricks fill up with the photos, they’ll be loaded and secured on a truck to make the short trek down the road from one Santa Clara, Calif., facility to another.
And don’t worry about any of the bright orange storage bricks, um, falling off the truck.
“If one of these bricks got loose in the wild somehow, there’s nothing that can be reconstructed into a useful image,” Weber said, noting they will be encrypted on the servers. Another security measure Shutterfly said it takes is keeping three copies of every image, each in a different location. That replication happens almost instantaneously, he said.
When the photo files get to their new home, Shutterfly will immediately integrate them into their own storage infrastructure. And, assuming all goes as planned, users will go from using an invisible bridge to access their shots on Kodak’s servers to finding them directly on Shutterfly, with no noticeable difference to them.
Shutterfly expects to have the last files moved over by November, though they will remain on Kodak’s servers for some time to come, Weber said.
When all is said and done, the big picture of 2012 for Shutterfly will have been dominated by its record transfer of the billions of snapshot memories. Talk about a serious Kodak moment.
“It’s an exciting thing to be part of,” Weber said. “Just pulling this off is a neat trick.”
[For the Record, 11:42 a.m. July 2: An earlier version of this post referred to Shutterfly’s chief information officer as Jeffrey Weber. His first name is spelled Geoffrey.]
RELATED
Picture this: Shutterfly snaps up Kodak Gallery
Hallmark cards to get more personal on Shutterfly’s site TreatInstagram site adds likes and comments, is no longer a dead end
Follow Michelle Maltais on Google+, Facebook or Twitter
