2013 BIG DATA OPPORTUNITIES SURVEY
By Joseph McKendrick, Research Analyst Produced by Unisphere Research, a Division of Information Today, Inc. May 2013
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
3 EXECUTIVE SUMMARY
Big data isn’t just the exclusive property of big web properties such as Google, Facebook, eBay and Yahoo. While these companies blazed the path in harnessing the analytic power of big data, organizations of all different sizes and industry groups are now leveraging big data in many ways as well. That’s the finding of a survey of 304 data managers and professionals, conducted by Unisphere Research, a division of Information Today Inc. The survey, fielded in April 2013, revealed a range of practical approaches that organizations of all types and sizes are adopting to manage and capitalize on the big data flowing through their enterprises. It’s important to separate the big data hype from the onthe-ground reality for enterprises in their day-to-day businesses. For example, there’s a perception that big data only surrenders its wealth when crunched on the same large-scale clustering technologies that are employed by the big web companies. However, many big data problems and opportunities are being addressed on existing, conventional technologies such as relational databases. Most companies are not concerned with indexing the world’s web information or delivering real-time recommendations to millions of users. Rather, they are focused on better understanding how a particular product or service is being received, or who is more likely to buy a new offering. The bottom line is that big data is more than simply amassing large quantities of bits and bytes and attempting to slice and dice it enough to glean some insights. Rather, it’s a process of understanding the value of information to the business, and working with the business to achieve growth. The ability to manage and harvest important business data from datasets large and small is a concern for executives across the board, whether they are running manufacturing companies or schools. Key findings from the survey include the following: ■ For many organizations today, handling data is not business as usual. More than two-fifths report they have formal “big data” initiatives in progress, most of which focus on greater customer analysis. ■ Between one-third and one-half of the organizations surveyed report high levels of volume, variety, velocity, and value in data—the “four Vs” that define big data. Two-fifths now maintain data stores in the hundreds of terabytes and greater, and three-fourths face restive user bases that want more data to do their jobs. ■ Most executives intuitively understand the advantages big data can bring to their operations, especially with predictive analytics and customer analytics. However, organizational and skills issues may slow down their progress. Respondents represent a wide range of job titles and, organizations. Leading titles mentioned include DBAs (17%), data architects (13%) and IT consultants (11%). Another 15% of respondents are IT managers and executives, while 8% are business unit executives. More than one-fourth come from organizations with more than 10,000 employees, and the same number represent small firms with 100 or fewer employees. Industries represented include IT and tech, government, financial services, business and consumer services, and education. (See Figures 34–36.) Many respondents may be enthusiastic about the potential big data offers, but understand that it needs to be represented to corporate leaders as providing clear business benefits. As one respondent put it: “Management sees the opportunity associated with closer understanding of customers and micro segmentation. But we need to avoid the words ‘big data,’ as people do not really understand what it means—but do associate big data with big costs.” Still, there are a number of survey respondents whose companies are already seeing direct benefits from their big data efforts. “Our organization strives best when it maximizes the success of its customers and direct salesforce,” says a respondent. “Big data properly analyzed not only provides the information to make better and timely business decisions, it also helps to determine behaviors and success factors in our direct sellers.”
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
4 BIG DATA INITIATIVES
For many organizations today, handling data is not business as usual. More than two-fifths report they have formal “big data” initiatives in progress, most of which focus on greater customer analysis. More than two-fifths of respondents indicate that their organizations have some type of formal “big data” initiative underway. (See Figure 1.) This may include the formation of special cross-organizational teams or committees to examine ways to better exploit this resource, or programs within departments—such as marketing—designed to mine and generate insights on available data. “Management is starting to recognize the importance of big data, but is just unsure how we can integrate it into our company,” says one respondent. It’s not surprising that many of the largest organizations in this survey—those with 1,000 employees and up—are engaged in big data initiatives. But there are many small firms also engaging big data as well. More than a third of the smallest companies or agencies in the survey, 37%, report big data efforts, along with 43% of those organizations with employees numbering in the hundreds. (See Figure 2.) Industry groups in the survey leading the way with big data initiatives include services and retail (61%), financial services and insurance (58%), and IT and tech (45%). (See Figure 3.) What is driving these big data initiatives? There are a myriad of business reasons, but most respondents indicate that it is a way to improve internal processes. For a majority of the respondents with such efforts underway, 59%, there is a need to improve existing processes. In addition, another 41% are concerned with the need to create new business processes/models. Another 39% are concerned with the need to reduce time to access data. (See Figure 4.) Big data initiatives also represent a range of business activity. A majority of respondents with big data initiatives, 55%, indicate that they are focusing on customer analysis and segmentation. Close to half, 46%, are concerned with historical/archived data analysis, while 39% are applying big data analytics to production systems’ log monitoring and analysis. (See Figure 5.) The data being employed in many big data efforts isn’t necessarily exotic or hard to capture. Rather, most organizations with formal big data efforts underway are employing information that is already sitting on machines or in storage systems within their enterprises. The survey finds a majority of those respondents with big data initiatives, 51%, say they are working with existing production or transactional data. Another 49% are working with real-time data feeds, suggesting that this is an important early step for many organizations seeking to capitalize on big data. Another traditional source of data that may already be captured and managed within the walls of enterprises—ERP and CRM data—also are the basis of many big data efforts. (See Figure 6.) Two-fifths of organizations also rely on archived or historical data—for initiatives such as data warehousing—as the foundation of their big data efforts. “The further we can see back in historical trends in relation to today, both in-house and across the sector, the better able we are to make product and marketing decisions that decrease operation cost and increase revenue,” says one respondent.
Figure 1: Specific Enterprise “Big Data” Initiatives Underway?
Yes 43% No, it's business as usual within our data environment 42%
Don’t know/unsure 15%
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
5
Figure 2: Specific Enterprise “Big Data” Initiatives Underway —By Company Size
1 to 100 employees 101 to 1,000 employees 1,001 to 10,000 employees 10,000+ employees
(Multiple responses permitted.)
37% 43% 47% 53%
0 20 40 60 80 100
Figure 3: Specific Enterprise “Big Data” Initiatives Underway —By Industry
Retail/services Financial services/insurance IT services/products Manufacturing Government/education
(Multiple responses permitted.)
61% 58% 45% 29% 27%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
6
Figure 4: Drivers of Enterprise “Big Data” Initiatives
Need to improve existing processes Need to create new business processes/ models Need to reduce time to access data 59% 41% 39%
Need more visibility into our customer base 37% Management needs to develop/open new lines of business We need more visibility into our internal operations Need to realign more with customers Competitors are doing it Looking to package and sell information Other
(Multiple responses permitted.)
32% 32% 27% 25% 13% 6%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
7
Figure 5: Types of Big Data Initiatives
Customer analysis/segmentation Historical/archived data analysis 55% 46%
Production systems log monitoring/analysis 39% Website monitoring/log analysis IT systems log monitoring/analysis Competitive/market analysis Content management (photos, images, documents) Social media analysis Testing/analyzing new products/R&D Other
(Multiple responses permitted.)
36% 37% 36% 32% 25% 25% 5%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
8
Figure 6: Types of Data Used in Big Data Initiatives
Production or transactional data Real-time data feeds Textual data (documents) ERP data CRM data Archived/historical data Web logs Social media data Multimedia (imagery, audio) Machine to machine data Sensor data Spatial data Other
(Multiple responses permitted.)
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
9 THE ENTERPRISE BIG DATA LANDSCAPE
Between one-third and one-half of the organizations surveyed report high levels of volume, variety, velocity, and value in data – the “four Vs” that define big data. Two-fifths now maintain data stores in the hundreds of terabytes and greater, and threefourths face restive user bases that want more data to do their jobs. Big data is often defined by the “four Vs”—volume, variety, velocity and value. On all four counts, the survey shows, big data is everywhere, within a range of organizations. Volume: There are considerable volumes of data moving through enterprises. One-sixth of the organizations surveyed, in fact (17%), report having more than a petabyte of data they are currently managing, taking into account all clones, snapshots, replicas and backups. Another 22% report maintaining their total data across their organizations ranges in the hundreds of terabytes. (See Figure 7.) Not surprisingly, most of the highvolume data is seen within larger organizations. Sixty-eight percent of companies or agencies with more than 10,000 employees have data across their organizations exceeding 100 TBs, versus only 8% of the smallest firms with less than 100 employees. (See Figure 8.) Industry groups leading the way in terms of data volume include the financial services and insurance sector, where twothirds report enterprise data exceeding 100 TBs. Manufacturers are also big consumers of big data, with 39% reporting data scaling into the hundreds of terabytes. Retail and service firms follow at 36%. (See Figure 9.) The volume shows no signs of letting up. How will the overall volume of data in respondents’ organizations change over the next three years? Just about every respondent, 96%, expects their data to increase in volume. More than one-fourth of respondents, 26%, say it will double. (See Figure 10.) Variety: The ability to extract insights from unstructured data —which is the essence of big data—represents opportunities for real business returns. What percentage of data in respondents’ enterprises is unstructured data (web logs, social media data, sensor data, documents, imagery, audio)? A large segment, 42%, estimate that a significant portion of their data stores—25% or more—is already in the form of unstructured data. (See Figure 11.) Interestingly, unstructured data is as much a part of the picture for small firms as it is large, global organizations. (See Figure 12.) In addition, the prevalence of unstructured data cuts across most major industry groups in the survey. (See Figure 13.) An overwhelming majority, 87%, say the amount of unstructured data in respondents’ organizations will increase over the next three years. Close to a third, 31%, expect their unstructured data stores to increase by more than 50% in that time period. (See Figure 14.) Velocity: The ability to move data from its sources to decision makers and applications at real or right-time speeds (delivered within one hour of creation) is another demand of big data. This is a capability that still has not been put in place at a majority of enterprises. Only one-fourth of respondents say that a significant amount of the data they maintain (more than 25%) is delivered real time. (See Figure 15.) There is significantly more progress among organizations that have formal big data programs in place, the survey also reveals. Close to one-third of respondents with big data programs report that a significant portion of their data is delivered real-time, versus 21% of respondents with no formal efforts underway. (See Figure 16.) Small to mid-sized businesses tend to have larger proportions of real-time data, the survey finds. For example, one-third of organizations with between 100 and 1,000 employees have a significant share real-time data available to their enterprises, compared to only 15% of the largest companies. (See Figure 17.) By industry, the financial services sector leads the way, with one-third of organizations in this category offering a significant segment of its data in real-time mode. The tech sector follows at 23%, and government agencies with 22%. (See Figure 18.) Over the next three years, close to a third of respondents, 30%, expect significant growth in the amount of real-time data being made available, exceeding 50% of their current data stores. (See Figure 19.) A similar percentage expect to see end-user demand for real-time increase as well. One-third say there will be a significant increase in demand, exceeding 50% over current levels. (See Figure 20.) As one respondent put it: “The current systems we have do not provide instant-access—dashboarding— to data to make critical changes on each business day before that business day is over.” Value: Finally, what makes big data meaningful is its ability to deliver insights to decision makers that move organizations forward. Three-fourths of respondents say they see users demanding access to more data to better drive their decisions and insights. (See Figure 21.) “Everyone wants access to the data now and they also need queries to be fast, given the mountains of data we are facing,” says one respondent. In addition, respondents were asked to indicate the importance of big data to their businesses. A majority, 58%, report that big data is “extremely” to “very” important. (See Figure 22.) There is a strong business case to be made for harnessing the power of big
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
▼ ▼
10
data. One respondent outlined his company's commitment: “We believe that additional information, derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allows correlations to be found to operational issues,” the respondent says. “Quick resolution would benefit our organization to serve our customers better and increase our subscriber base.” Respondents whose organizations are actively addressing big data opportunities through formal programs are almost three times as likely to recognize the value of the resource. Close to nine-tenths of respondents with big data efforts indicate big data is “extremely” or “very” important to their businesses. (See Figure 23.) Small to mid-size businesses place the greatest importance on big data, the survey finds. Sixty-five percent of organizations with between 100 and 1,000 employees say big data is “extremely”or “very” important to their business, versus 35% of the largest organizations in the survey. (See Figure 24.) Financial services and tech firms are the most likely to see the business value in big data—close to two-thirds of respondents within each category say this information is important. (See Figure 25.)
Figure 7: Total Amount of Data Managed
< 5 Terabytes (TBs) 5 to 50 TBs 50 to 100 TBs 100 to 500 TBs 500 TBs to 1 Petabyte (PB) 1 to 10 PBs >10 PBs
(Multiple responses permitted.)
27% 24% 11% 11% 11% 8% 9%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
11
Figure 8: Big Data Sites Managing More Than 100 Terabytes —By Company Size
1 to 100 employees 101 to 1,000 employees 1,001 to 10,000 employees 10,000+ employees
(Multiple responses permitted.)
8% 28% 47% 68%
0 20 40 60 80 100
Figure 9: Big Data Sites Managing More Than 100 Terabytes —By Industry
Financial services/insurance Manufacturing Retail/services IT services/products Government/education
(Multiple responses permitted.)
66% 39% 36% 33% 31%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
12
Figure 10: Changes in Enterprise Data Volume Over Next Three Years
Increase by more than 100% Increase 51% to 100% Increase 26% to 50% Increase 10% to 25% Increase 1% to 10% No change Decrease
(Multiple responses permitted.)
26% 22% 21% 22% 6% 3% 0.35%
0 20 40 60 80 100
Figure 11: Percentage of Enterprise Data That Is Unstructured
None at this time <10% Between 10% to 25% Between 25% to 50% >50%
(Multiple responses permitted.)
5% 25% 29% 27% 15%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
13
Figure 12: Unstructured Big Data Sites With More Than 25% of Data Unstructured—By Company Size
1 to 100 employees 101 to 1,000 employees 1,001 to 10,000 employees 10,000+ employees
(Multiple responses permitted.)
44% 34% 46% 44%
0 20 40 60 80 100
Figure 13: Unstructured Big Data Sites With More Than 25% of Data Unstructured—By Industry
Financial services/insurance Manufacturing IT services/products Government/education Retail/services
(Multiple responses permitted.)
48% 48% 45% 45% 43%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
14
Figure 14: Changes in Enterprise Unstructured Data Volume Over Next Three Years
Increase by more than 100% Increase 51% to 100% Increase 26% to 50% Increase 10% to 25% Increase 1% to 10% No change Decrease Don’t know/unsure
(Multiple responses permitted.)
15% 16% 20% 19% 11% 5% 2% 11%
0 20 40 60 80 100
Figure 15: Percentage of Enterprise Data Delivered Real-Time
(Defined as within one hour of creation)
None at this time <10% Between 10% to 25% Between 25% to 50% >50%
(Multiple responses permitted.)
10% 35% 30% 9% 16%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
15
Figure 16: Big Data Programs’ Impact on Percentage of Enterprise Data Delivered Real-Time
(Respondents reporting more than 25% of data delivered real-time)
Formal “big data” program No big data efforts
31% 21%
0 20 40 60 80 100
Figure 17: Real-Time Big Data Sites With More Than 25% of Data Real-Time—By Company Size
1 to 100 employees 101 to 1,000 employees 1,001 to 10,000 employees 10,000+ employees
(Multiple responses permitted.)
29% 32% 18% 45%
0 20 40 60 80 100
Figure 18: Real-Time Big Data Sites With More Than 25% of Data Real-Time—By Industry
Financial services/insurance IT services/products Government/education Retail/services Manufacturing
(Multiple responses permitted.)
33% 23% 22% 18% 14%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
16
Figure 19: Changes in Enterprise Unstructured Data Volume Over Next Three Years
(Defined as within one hour of creation)
Increase by more than 100% Increase 51% to 100% Increase 26% to 50% Increase 10% to 25% Increase 1% to 10% No change Decrease Don’t know/unsure
(Multiple responses permitted.)
13% 17% 20% 26% 10% 4% 0% 10%
0 20 40 60 80 100
Figure 20: Anticipated Changes in End-User Demand for Enterprise Real-Time Data Over Next Three Years
(Defined as within one hour of creation)
Increase by more than 100% Increase 51% to 100% Increase 26% to 50% Increase 10% to 25% Increase 1% to 10% No change Decrease Don’t know/unsure
(Multiple responses permitted.)
16% 17% 23% 20% 10% 6% 0% 8%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
17
Figure 21: Users Demanding Access to More Data?
Don’t know/unsure 14% Yes 75%
No 12%
(Total does not equal 100% due to rounding.)
Figure 22: Importance of Big Data to Business
Not important today 14% Extremely important 23%
Very important 35% Somewhat important 27%
(Total does not equal 100% due to rounding.)
Figure 23: Big Data Programs’ Impact on Big Data Importance to Business
(Respondents indicating big data is “extremely” or “very” important)
Formal “big data” program No big data efforts
87% 32%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
18
Figure 24: Importance of Big Data to Business—By Company Size
(Respondents indicating big data is “extremely” or “very” important)
1 to 100 employees 101 to 1,000 employees 1,001 to 10,000 employees 10,000+ employees
(Multiple responses permitted.)
50% 65% 43% 35%
0 20 40 60 80 100
Figure 25: Importance of Big Data to Business—By Industry
(Respondents indicating big data is “extremely” or “very” important)
Financial services/insurance IT services/products Retail/services Government/education Manufacturing
(Multiple responses permitted.)
65% 65% 57% 54% 43%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
19 BIG DATA OPPORTUNITIES AND CHALLENGES
Most executives intuitively understand the advantages big data can bring to their operations, especially with predictive analytics and customer analytics. However, organizational and skills issues may slow down their progress. For many organizations, big data offers a range of opportunities to improve and increase business. The most popular application cited is the ability to perform predictive analytics, cited by 62% of respondents. Another 53% say big data offers ways to better understand their customers. More than twofifths are linking their big data efforts directly to business growth, employing big data tools and analysis to grow business revenues streams and compete more effectively in their markets. (See Figure 26.) Big data is playing a role in many industries. “We are looking at big data to help us provide better, safer patient care by learning from past episodes of care and outcomes,” says the manager of a healthcare facility. “As more EMR/EHR systems come online, we believe the medical community will demand access to big data.” While most respondents believe their big data efforts will deliver significant benefits, there are barriers for many as well. A large segment, 43%, say the lack of budget holds them back, while 35% are also concerned with a lack of skills. About a third cite both data governance issues as well as lack of urgency from business management, two highly inter-related issues. The ability to help business users connect with the data that is available to them is an important emerging role for data professionals. (See Figure 27.) Top management support is essential, states a respondent whose department has “the backing of the CEO and COO, to move forward in business intelligence and big data initiatives, in order to streamline and more accurately monitor and maintain our data flow.” This helps ensure that the agency’s “federal, state, and local regulatory requirements are met—as well protecting our grants, and funding streams,” the respondent explains. Interestingly, when the organization addresses big data opportunities through formal programs, they are far more likely to be concerned with issues such as data governance, skills, and technology requirements. The presence of such programs points to management's sense that big data is a priority. (See Figure 28.) While many respondents believe that their current technology is capable of helping them manage and capitalize on big data, current solutions in place also are often in need of an overhaul. More than a third of respondents, 36%, say faster querying of the data is a challenge they are attempting to get their arms around. Another 43% are concerned about providing faster access to the larger datasets that may be proliferating around them. (See Figure 29.) What strategies are respondents’ organizations pursuing to manage and capitalize on big data on an enterprise level? Onethird indicate they are working to extend and optimize their current infrastructure, while about one-fourth are adopting new frameworks such as Hadoop and in-memory databases. (See Figure 30.) When it comes to leadership in big data initiatives, IT departments are taking the lead among a large segment of respondents, as cited by 42%. One-third say senior business executives such as the CEO or CFO are taking charge of the effort. (See Figure 31.) What data management technologies do respondents’ organizations currently use or plan to adopt within the next 24 months to manage the big data challenge? Two-thirds are employing relational databases to meet the challenge. Close to half, 47%, are employing business intelligence tools. (See Figure 32.) As discussed in the previous section, skills availability is a critical challenge for organizations seeking to better manage or capitalize on their big data assets. There are three ways to address this challenge—either automate more, train more, or recruit more. For many respondents, automation and technology are considered the best paths. Close to four-fifths, 59%, say they plan to equip staff with easy-to-use analytic tools. Close to half, 47%, intend to train business users with self-service business intelligence tools. (See Figure 33.)
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
20
Figure 26: Big Data Opportunities
Perform predictive analytics Better understand our customers Grow business revenue streams Compete more effectively Lower business operational costs 62% 53% 42% 41% 37%
Expose new economic value & opportunities 35% Faster time to market new offerings Other
(Multiple responses permitted.)
27% 2%
0 20 40 60 80 100
Figure 27: Big Data Business Barriers
Lack of budget Lack of skills Data governance issues Not a business management priority Unsure of technology requirements Unsure of the value Other
(Multiple responses permitted.)
43% 35% 31% 31% 29% 32% 3%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
21
Figure 28: Big Data Programs’ Impact on Big Data Business Barriers
Formal “big data” program Lack of budget Lack of skills Data governance issues Not a business management priority Unsure of technology requirements Unsure of the value 41% 42% 45% 17% 33% 26% No big data efforts 44% 27% 20% 47% 22% 39%
Figure 29: Current Solutions’ Technology Barriers
Faster querying of the data Faster access to large data volumes/sets Faster acquisition and storage of data Capacity to store more data Faster access to non-traditional data types 51% 43% 36% 35% 33%
Capacity to store non-traditional data types 33% Cost of storage for larger or more varied datasets
(Multiple responses permitted.)
29%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
22
Figure 30: Enterprise Big Data Strategies
Extend and optimize current infrastructure We don’t have an enterprise strategy for big data at this time Adopt new frameworks such as Hadoop and in-memory databases Establish a head of big data/analytics strategy Implement training and education in big data and analytics 33% 26% 24% 22% 22%
Purchase applications that address specific 15% big data opportunities Establish a cross-enterprise committee/ center of excellence Don't know/unsure Other
(Multiple responses permitted.)
13% 22% 1%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
23
Figure 31: Big Data Corporate Leaders
IT department Senior business management (CEO, CFO) Line-of-business No one is in charge at this time Cross-enterprise committee/center of excellence Don't know/unsure
(Multiple responses permitted.)
42% 33% 22% 21% 11% 13%
0 20 40 60 80 100
Figure 32: Big Data Technologies Adopted or To Be Adopted in Next 24 Months
Relational databases (e.g., Oracle, DB2, etc.) 64% Business intelligence tools Hadoop/MapReduce In-memory platforms Other Open Source technologies NoSQL databases R Don’t know/unsure Other
(Multiple responses permitted.)
47% 28% 21% 20% 20% 12% 17% 4%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
24
Figure 33: How Skills Base Will Be Expanded Over Next 12 Months
Equip staff with easy-to-use analytic tools Train business users with self-service BI tools Train BI analysts in data mining Hire data scientists
(Multiple responses permitted.)
59% 47% 43% 19%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
25 DEMOGRAPHICS
Figure 34: Primary Job Titles
Database administrator (DBA) Data architect IT consultant Programmer/developer Director/manager of IS/IT or computer-related function Analyst/systems analyst Project manager Chief information officer/chief technology officer/VP of IT Systems administrator 17% 13% 11% 9% 9% 8% 7% 6% 5%
Executive management level for the business 5% Manager of a business unit Other
(Multiple responses permitted.)
3% 7%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
26
Figure 35: Number of Employees
(Include all locations, branches, and subsidiaries)
1 to 100 employees 101 to 500 employees 501 to 1,000 employees 1,001 to 5,000 employees 5,001 to 10,000 employees 10,000+ employees Don’t know/unsure
27% 14% 8% 12% 10% 26% 2%
0 20 40 60 80 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.
27
Figure 36: Primary Industries
IT services/consulting/system integration Government (all levels) Financial services Software/application development Business or consumer services Education (all levels) Manufacturing Retail/distribution Healthcare/medical Utility/telecommunications/transportation Insurance High-tech manufacturing Other
(Multiple responses permitted.)
17% 12% 10% 10% 6% 6% 6% 6% 6% 5% 3% 2% 10%
0 0
20 20
40 40
60 60
80 80
100 100
2013 Big Data Opportunities Survey was produced by Unisphere Research and sponsored by SAP. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters. To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701, Web: www.dbta.com.