Big data is not just for software companies, but increasingly permeates almost every industry
On 24 June, private equity (PE) firm TA Associates Inc. said it invested $25 million in lieu of a minority stake in Mumbai-based Fractal Analytics Inc., which provides advanced analytics to Fortune 500 companies. On 7 May, Wipro Ltd signed an agreement to invest $30 million for a minority position in Opera Solutions Llc, a big data science company headquartered in Jersey City, NJ.
Both these companies are one among the hundreds that have begun to understand, some as users and others as investors, the value of big data—a term used to describe the value in analyzing the mountains of data produced by companies and individuals with the help of algorithms rather than some human input the way business intelligence (BI) solutions do.
Intel Corp., for instance, estimates that the world generates 1 petabyte (1,000 terabytes) of data every 11 seconds or the equivalent of 13 years of HD (high-definition) video. Research firm International Data Corp. (IDC) estimated that in 2011, all of the data created in the world amounted to 1.6 trillion gigabytes. By 2020, 50 billion devices will be connected to networks and the Internet.
The proliferation of devices such as PCs and smartphones worldwide increased Internet access within emerging markets and the boost in data from machines such as surveillance cameras or smart meters has contributed to the doubling of the digital universe within the past two years alone—to a mammoth 2.8 ZB (zettabytes), according to a December report titled “IDC Digital Universe”, which was sponsored by EMC Corp.
IDC projects that the digital universe will reach 40 ZB by 2020, an amount that exceeds previous forecasts by 14%. There are 700,500,000,000,000,000,000 grains of sand on all the beaches on earth, which means 40 ZB is equal to 57 times the amount of all the grains of sand on all the beaches on earth. In 2020, 40 ZB will be 5,247 GB per person worldwide, the report said, adding that by 2020, emerging markets will supplant the developed world as the main producer of the world’s data.
It’s no wonder that in 2006, market researcher Clive Humby declared data “the new oil”, which underscores big data’s potential.
In a May 2011 paper, the McKinsey Global Institute (MGI) forecast the use of big data would become a key basis of competition and growth for individual firms, even as it acknowledged that policies related to privacy, security, intellectual property, and even liability will need to be addressed in a big data world.
MGI studied big data in five domains-—healthcare and retail in the US, the public sector in Europe, and manufacturing and personal-location data globally. The report said a retailer using big data to the full could increase its operating margin by more than 60%. It added that if US healthcare was to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year with two-thirds of that in the form of reducing US healthcare expenditure by about 8%.
In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data, not including using big data to reduce fraud and errors and boost the collection of tax revenues. And users of services enabled by personal-location data could capture $600 billion in consumer surplus.
Wipro, for instance, invested in Opera Solutions for its specialization in machine learning, which Opera applies to the world’s big data flows to extract predictive patterns, or signals based on which it offers a range of solutions, delivered as a service to improve front-line productivity and bottom line growth.
Infosys Ltd, according to PWC’s Global 100 Software Leaders Report released in May, used analytics to make sure it was deploying sales efforts most profitably. It analyzed lifetime revenues of clients and discovered that sales had drifted from a focus on the largest enterprises to some mid-sized businesses. The analysis also highlighted how many resources these smaller accounts typically consumed. One Infosys product for retailers, for example, takes data on purchasing patterns and uses an algorithm to make recommendations to consumers, just like Amazon and Netflix. Infosys is paid by value that its creates for the online store and it’s completely in the cloud, the report said.
On 9 May, IBM Corp. said the Central Bank of India is leveraging its analytics solution to transform its financial management processes which includes activities ranging from budgeting to forecasting to liquidity management, due to which the company is able to gain better insight into branch and regional office performance, allowing for further flexibility and quicker shifts in strategy to drive improved results while also maintaining regulatory compliance.
With the use of analytics, Central Bank of India has moved away from spreadsheet based planning to a smarter process that analyzes daily financial data based on actual performance and potential for growth.
Another key part of the bank’s transformation was the implementation of state-of-the-art static asset liability management solution which enables the company to reduce liquidity risk and also minimize impact of interest rate and foreign exchange rate movements. As such, the bank has achieved total automation of this tedious and time consuming process by consolidating approximately 7.5 million deals and trade positions across the company.
Why is big data so important?
By its very nature, network traffic is big data. The over 6 billion mobile subscriptions around the world, for instance, generate billions of text messages.
According to a Cisco report in June 2012, big data solutions could help reduce traffic jams or even eliminate them with predictive, realtime analysis on traffic flows, feeding immediate changes to traffic signals, digital signs, and routing, before backups begin. Paper receipts from retailers and banks that clutter one’s wallet could be replaced by electronic records. Businesses could enrich these records through contextual and comparative information. Individuals could manage, share, monetize, and utilize the data through, for example, budget management and health advice applications, noted the report.
Medical advances enabled by big data could include treatments personalized to a patient’s unique health issues. Epidemics of contagious diseases or food contamination could be predicted, tracked, and curtailed. Already, Google Flu Trends has aggregated data on searches for health information to predict outbreaks across the globe, the report noted.
On a planetary scale (or beyond), it said, big data can monitor and analyze vast amounts of information on anything from climate change, astrophysics, and energy consumption to geopolitics and socioeconomics.
India, too, is seeing a lot of interest in big data.
Consider this. On 11 June, Nasscom said it received 4,000 applications for angel funding and acceleration under its “10,000 start-ups” programme. The last day for applying was 30 May. Out of the total applications received, close to 23% were from Bangalore, followed by 20% from Delhi NCR (National Capital Region). Over 70% applications came from applicants under the age of 30 and about 15% applicants were women.
From a technology point of view, 66% of the applications came for Web/Internet start-ups, followed by mobile start-ups occupying a share of 24%. Applications from cloud and big data were at 16% and 11%, respectively.
Big data is not just for software companies, of course, but increasingly permeates almost every industry. Data analytics potentially makes every company a software company when it embeds the necessary technology into its products to make them part of the Internet of things, according to the PWC report cited above.
Nike puts sensors in its shoes to collect data to help you improve your performance. General Electric (GE) and Airbus equip their jet engines with sensors and software that allows them to gather and transmit data to increase safety and efficiency. Depending on how they choose to charge for these services, they could find themselves in the subscription software business.
“We won’t be surprised to see a Nike or GE or some other non-traditional “software” company join the Global 100 list in the future,” the PWC report said.
Not to be left behind, software and hardware vendors are putting in their mite behind big data to garner more revenue.
As of early 2012, the big data market stood at just over $5 billion based on related software, hardware, and services revenue, according to market research firm Wikibon. The total big data market reached $11.4 billion in 2012, ahead of Wikibon’s 2011 forecast. The big data market is projected to reach $18.1 billion in 2013, an annual growth of 61%. This puts it on pace to exceed $47 billion by 2017, the report said.
According to a 19 February Wikibon report, market leader IBM offers by far the largest product and services portfolio by both breadth and depth. Hewlett-Packard Co. or HP achieved second-place status in the overall big data market by revenue in 2012. Amazon continued and Google kicked off increasingly aggressive moves into the big data market, the report said.
In 2012, the top 10 vendors by revenue in the big data segment comprised IBM, HP, Teradata Corp., Dell Inc., Oracle Corp., SAP AG, EMC Corp., Cisco Systems Inc., Microsoft Corp. and Accenture Plc.
On 22 May, IBM unveiled the IBM Watson Engagement Advisor, a technology breakthrough that allows brands to crunch big data in record time to transform the way they engage clients in key functions such as customer service, marketing and sales. With the latest IBM Watson debut, IBM is enabling clients to better respond to market shifts in realtime, automate marketing, and transform the way they service their clients.
The IBM Watson Engagement Advisor “Ask Watson” feature greets, and offers help to, customers via any channel, be it through a website chat window or a mobile push alert, saving consumers the hassle of performing searches, combing through websites and forums, or waiting endlessly for a response about the information they need.
Since its television debut, IBM Watson (the supercomputer that even beat Jeopardy quiz gamers) “is smarter, faster and smaller—having gained a 240% improvement in system performance and a reduction in physical requirements by 75%”, says IBM.
This February, Intel said it’s is delivering an innovative open platform built on Apache Hadoop* that can keep pace with the rapid evolution of big data analytics. Analyzing one terabyte of data, which would previously take more than four hours to fully process, can now be done in seven minutes with its hardware, Intel claimed.
For example, in a hospital setting, the intelligence derived from this data could help improve patient care by helping caregivers make quicker and more accurate diagnoses, determine effectiveness of drugs, drug interactions, dosage recommendations and potential side effects through the analysis of millions of electronic medical records, public health data and claims records. Strict guidelines also exist globally for protecting health and payment information, making it imperative to maintain security and privacy while performing analytics.
Future of big data
An important, but largely untapped, type of data is the real-time actionable data generated by sources such as devices, sensors and video, which often provide the most value while interacting in real time—Cisco calls this data in motion. The network can provide useful contextual information to data in motion such as a person or device’s location, identity and presence (whether they are “available” or not). This data can be used by applications to make decisions or take actions that are immediately relevant, or even to predict future events. Machine-to-machine communication in factory automation is an example where data in motion could be extremely valuable in optimizing a production process.
According to the Cisco Visual Networking Index Global Mobile Data Traffic Forecast for 2012 to 2017, there will be more than 1.7 billion machine-to-machine connections by 2017.
However, there are a few issues that need to be ironed out before big data becomes mainstream.
On 11 June, Gartner Inc. said less than 10% of today’s enterprises have a true information strategy. And while technology is important to big data solutions, people are needed with the special skill sets and creativity—for instance, “data scientists” who transform raw data into information leading to discovery and insight, communicate what they’ve learned in creative and visual ways, and suggest business impact.
On 5 June, Gartner said big data will grow past its hype towards 2016 to become “just data” once the technologies mature, and organizations learn how to deal with it. “The bottom line is that not all information requires a big data approach,” said Frank Buytendijk, research vice-president at Gartner. “The new “big data way’ is not going to replace all other forms of information management. There is more room—and need—for experimentation in the area of “information of innovation”, for instance, with social media data, or by making processes more information-centric.”