infoTECH Feature

October 03, 2016

The Evolution of Big Data over the Last Five Years

Vast amounts of data are being captured into massive data sets for analysis. Organizations are tapping this big data to obtain accurate models on trends, demographics, consumer patterns, and other insights regarding consumer behavior. Once something of a buzzword, big data has become a desirable advantage over the past five years. It's estimated that individual enterprise spending on big data now costs an average of $13.8 million yearly.

Big Data on the Cloud

Technology from only ten years ago seems increasingly inadequate to meeting greater demand from ever-growing datasets. Companies are now looking at cloud-based solutions because less technology is required in accessing and using it. The trend over the last few years has not been managing a tidal wave of raw data, but services that can package it neatly for end users.

Google (News - Alert) Cloud

Google recently released its own cloud-computing platform that provides client hosting on the same proven infrastructure that supports its highly successful products such as YouTube and Google Search. Google Cloud Platform offers a range of developer products that help users build projects from basic websites to complex analytical solutions. It's actually part of a suite of enterprise solutions called Google for Work with an impressive set of modular services and tools. Cloud Platform is likely to continue growing as an alternative for cloud services, data storage and translation, and tools to promote big data development. Google's Cloud is seen as an attempt to compete with the well-received Lambda service from giant online retailer Amazon and its AWS (Amazon Web Services (News - Alert)) cloud platform that only appeared in September of 2015. Lambda allows developers to upload and run their own code.


Business intelligence (BI) platforms from companies like Oracle (News - Alert) and Microsoft are being replaced with open-source platforms like Hadoop. From Apache, it's been around since 2003 as a search engine alternative but really came into its own after it was adopted by Yahoo in 2008 and saw the spinoff of Hortonworks, a separate company producing analytical software. Over the next several years, Hadoop gained a reputation for solving data problems, and became increasingly flexible as an open source platform. A favored approach to big data is Hadoop on OLAP (Online Analytical Processing). OLAP is the technology behind many business intelligence tools. It has powerful capabilities for data discovery, report viewing, complex calculations and forecast planning.


But Apache has come up with a new big data technology which it calls Spark. This is an open source framework engineered for a cloud-friendly clustered computing environment. Built from the ground-up as a fast engine for big data processing, it also has taken a modular approach for handling SQL, data streaming, machine learning, and even processing graphs and charts. It was originally released in May of 2014 and has been gaining in popularity. Even Microsoft (News - Alert) is incorporating Spark into its latest BI solutions.


IDC (International Data Corporation) estimates that 70 percent of major companies are now purchasing their analytical data from other sources. This mega-exchange of consumer data bears the increased risk of exposing sensitive information that must also have mandatory levels of protection according to government regulations. The challenge for cloud analytics lies in developing procedures and protocols to ensure that this massive data exchange is secure. Cybercrime consistently makes headlines, and with some very large and well-established companies such as Sony and Target (News - Alert). All over the globe, consumers are demanding that government and industry enact even more legislation to see that data security and controls are strictly enforced. However, as data is amassed on cybercrime, big data may hold the answers to staying ahead of the hackers.

These changed aspects of modern computing, including cybercrime, social media and cloud technologies, are now driving big data. Organizations must remain agile and informed to compete in a world where huge volumes of data are a valuable commodity.

Edited by Alicia Young

Subscribe to InfoTECH Spotlight eNews

InfoTECH Spotlight eNews delivers the latest news impacting technology in the IT industry each week. Sign up to receive FREE breaking news today!
FREE eNewsletter

infoTECH Whitepapers