Reigning in Big Data: Enterprises Move to the Cloud to Manage Storage Needs, Improve Accessibility

Documents, e-mails, contracts, media and graphics – these are just a sample of the electronic content that businesses around the world generate each day. And while they’re not filling file cabinets and taking up valuable square feet of real estate, they are consuming valuable storage resources.

IDC (News - Alert) estimates that this year the volume of digital content will grow to 2.7 zettabytes (ZB), almost doubling over 2011. Making it more complicated is that more than 90 percent of this information will be unstructured, packed with rich information, but challenging to understand and analyze.

So what happens to this data? In essence, for most companies, it’s the enterprise equivalent of the TV show “Hoarders: Buried Alive.” By some accounts, as much as 70 percent of the data an enterprise stores goes untouched, either because companies are unsure of what can and cannot be deleted, or because employees cannot find what they need when they need it.

It’s no surprise that big data just keeps getting bigger as more and more enterprises move headfirst into the digital era. A recent LinkedIn survey asked businesses what technology trend will have the biggest impact on their companies, and big data is clearly on the radar screen, ranking third, just behind cloud computing and mobile.

But adding storage is simply not enough to keep up with big data demands and is an expensive proposition. CIOs need strategies to ensure that the data being saved can be accessed when needed – whether it’s stored onsite or in the cloud.

While adding more storage addresses the space issue, it does have the negative effect of making the haystack larger and the needles harder to find in an already challenging scenario.

The Intervention: Facing Down Storage Sprawl

Most businesses know they have a hoarding problem, and that, in turn, creates challenges for the IT department as they try to identify what to save and what can be eliminated in an effort to maximize storage resources and make data easier to access.

In the days when paper ruled and offices were lined with file cabinets, employees were more likely to keep only the documents they were using, after doing a routinely scheduled purge. But as we move from the physical document world to one where electronic files of varying types predominate, employees are saving everything, in case they ever need it.

And as experience shows, in most cases, they won’t.

This hoarder mentality results in growing demands on the IT staff and storage infrastructure. IT departments are tasked with finding ways to manage the vast amounts of unstructured data in a way that it can be useful to employees and the company, while minimizing the expenses associated with their growing storage needs. While many IT departments try to solve their storage dilemma by using expensive, high-performance SAN solutions, the key is knowing that all data is not created equal.

How data is used, combined with the enterprise’s unique needs, should dictate the type of storage, indexing and analysis software that is employed.

Taking Action: Organization 101

Just like on “Hoarders,” the first step is to get organized, deciding what you need to keep and where it will be stored. For the enterprise, this means that CIOs and IT departments must develop – or update – retention policies to deal with electronic files. When creating these policies, IT, legal and compliance personnel need to take into account any regulatory requirements they must adhere to for their specific industries, as well as what they need to do to meet the basic guidelines for the Federal Rules for Civil Procedures, in the event they have to produce historical documents for court cases or other purposes.

Enterprises also must evaluate the types of documents being stored, how employees use the documents and how often they are being accessed. With this information, CIOs and IT staff can determine what method of storage might be best, whether indexing and archiving solutions should be employed to give access to the data when it’s needed, and if they can extract additional value from the stored data to improve efficiency or gain a competitive business advantage.

Storage: A Hybrid Approach

When cleaning house, everyone knows it may not be possible to keep all of your possessions in that small apartment. Those big family heirlooms, the ones you hope to pass down to your children, sometimes have to moved into an off-site storage facility because you simply do not have the space at home or can’t afford to put on an addition. 

The same holds true for enterprises, which can benefit from a hybrid approach to their storage needs.  This hybrid approach entails using a combination of onsite and cloud storage to improve the usability of commonly accessed data, while maintaining historical archives.

There are several providers in the market now that are offering tools to assist with the hybrid approach, including Gladinet, Mezeo Software, Nasuni and Panzura, as well as Amazon’s AWS Storage Gateway (News - Alert). While their approaches may vary slightly, the general concept is a combination of on-premises hardware and cloud storage. 

These solutions all employ either an onsite appliance or software and can replace traditional file servers. These hybrid technologies enable IT to set policies that would automatically move infrequently used files to cloud storage. 

Going with a hybrid model is ideal for those enterprises that have a great deal of data – the untouched 70 percent – which needs to be retained for regulatory compliance or back-up. Commodity cloud storage solutions offer an inexpensive way to archive data, particularly the files that are rarely accessed. Then, the most commonly used data can be maintained on-premise to offer easy, fast access and convenience directly from the appliance.

By taking this hybrid approach to data storage, IT departments can support strong retention policies, maintain compliance, better control the cost and maintenance related to onsite storage solutions and reign in their storage requirements for maximum usability.

Shahin Pirooz, chief security officer and chief technology officer at CenterBeam, Inc., has a wealth of experience in operations management, account leadership, project management and customer relationship management. Shahin has deep technology expertise covering areas such as IT architecture (development, design, planning and implementation), as well as core tools, operating systems and programming languages. He’s an active blogger for IBM (News - Alert) and Computer Technology Review, and has contributed to Forbes, Virtual Strategy Magazine, Baseline and Enterprise Systems Journal, among others.