Short Biography of Torrance Mayberry

Saturday, November 17, 2012

Big Data: A Playbook to pivot for success.

The idea that Big Data is about people was one of the salient themes highlighted during the panel discussion at the 2012 Big Data summit in Sydney, Australia. The reality of this Big Data phenomenon is being influenced by a trend in which the worlds ability to generate data (e.g. from humans and machines) is far exceeding our ability to analyse it. How organisations adapt to this trend in the near to long term is a critical factor enabling them to pivot for success . 

The Big Data phenomenon is a condition in which data comes in all scales and shapes encompassing millions of everyday new human interactions that are recorded and millions of old human interactions that are digitised. The phenomenon that follows has been creating massive new sources of data about human contention, collaboration and overall social behaviour. Most organisations recognise they need a resilient playbook that will enable them to pivot to find success. 

The pivot points for success in organisations can be found in how they formulate answers to questions such as:
· What data do we have?
· What data do we need or what data is valuable to the organisation?
· Is data viewed as an asset?
· What tells us that data is an asset?
· How can we extract its true value?
· How can we effectively use data as an asset to differentiate?
· How do we foster a climate for data driven discovery to flourish?

The answers to such questions need to be examined to help organisations shape the right conversations that will unleash the 4V of Big Data - ‘Value’. As shown in Figure A, digitisation is all around us and an organisations ability to pivot successfully depends on how they use data. The window of opportunity is rapidly narrowing and will soon leave many organisations with very little time to adapt to this Big Data phenomenon for competitive advantage. Value is much less about technology advancements and is more about how fast data experts enable their entire organisation to infer meaning from data and take action based on that meaning.

This thesis was recently played out in society on the backdrop of the 2012 American presidential election. There is no better case study than what was witnessed. The outcome from this high profile event provides more evidence that Big Data and the science of data are real. In addition, it provides a case study on the successful use and misuse of data at scale. Whilst both candidates attempted to use Big Data as a competitive advantage the playbook that was executed produced different outcomes. In the aftermath of the election lessons learnt from the winning candidate (or organisation) has outlined a playbook that enabled the candidate’s team to pivot for success to unleash the 5V of Big Data – ‘Victory’.

Figure A: This data ecosystem is on the verge of changing the way we live and work.


Signal validation for Big Data:

The presidential election affirmed that in this era of Big Data an organisation’s ability to harness data at scale as an asset is more than looking at data simply as something that just feeds into a report or dashboard that you would show to an executive once every month or quarter. The winning candidate fostered a mindset that not only looked at
data as an asset in itself, but infused in its climate the science of data. As these factors shaped the operating rhythm it ultimately enabled the winning candidates team to continuously pivot for success. In addition, the winning candidate amassed data experts who had in-depth understanding of how to effectively use accessible technologies, efficiently collect data from a variety of sources and establish a collaborative data ecosystem or whole system with people and machines working in harmony. The winning candidates team of data experts understood that for systems to succeed, especially systems for data analysis, they must be consciously designed to enhance the capabilities and thinking of people, not to replace them. [2] [3] [4]

For instance, the Time article, "Inside the Secret World of the Data Crunchers Who Helped Obama Win" suggests from day one the winning candidate had a priority to attract and retain data experts (e.g. data scientist, researchers, behavioural scientist and so on) that understood the science of data and knew how to use it as a competitive advantage.

The article goes on to suggest the mindset or tone was set at the top to use data as an asset and dissolve data silos that had been stored in many different systems. The team of data experts were empowered to innovate and create as they applied the science of data to establish a collaborative data ecosystem enabling the overall team to compete at its highest potential.

The collaborative data ecosystem enabled computer simulations to run the election 66,000 times every night to predict the likelihood of the candidates team winning each swing state. In addition, the systems ability to scale fostered a climate of data driven decision making that was so accurate it enabled the winning candidates team to not only uncover demand potential, but also to proactively pivot for success as they adapted to changes allocating resources to effectively interact with citizens across all channels. [5][6]

The opposing candidates team provides lessons learnt on how the misuse of data can quickly degrade its value. Although the opposing candidates team had a similar strategic intent, their lack of ability to apply science to data and infer meaning for execution prevented the overall team from competing at its highest potential. Their system was unable to deal with data at scale another critical factor inhibiting the opposing candidates teams ability to put Big Data and Analytics to work. [7] [8]

Among other lessons learnt (and there are many more), this disadvantage highlighted that in future the candidate with the best data and data experts who understand the science of data will have the competitive advantage.

Is this Big Data playbook to pivot for success new?

No, not really! The winning candidate
s use of data is not unlike what most Born Digital organisations have already embraced and embedded within their business models. Born digital organisations such as Google, Facebook, Amazon, Cloudera, FourSquare and many others have well established cultures that understand the value of data and how to apply the science of data to harness data as an asset. In addition, their data experts understand that behavioural and social science provides time-tested methodologies for discerning meaning in data. [9] [10] [11]

As Big Data innovations continue to introduce breakthroughs which establish a more resilient human link to the data being analysed, improvements are evident in reduced time to insight and an organisations ability to infer meaning from data at scale and take action based on that meaning. For instance, Cloudera Impalas distributed query execution engine, Facebook's Corona (Under the Hood: Scheduling MapReduce jobs more efficiently with Corona), or Amazons recent announcement to expand its AWS footprint in a new region in Sydney, Australia are just a few of the innovations that will enable data experts to help transform their organisations into data driven powerhouses. [12] [13] [14]

Is Big Data a possible macro growth story?

The Big Data innovations are among the green shoots that have the possibility to influence the macro economic growth story. As governments look for a balance through austerity measures, interventions that focus on policy to accelerate advancements in Big Data may enable the world to emerge faster from this current economic rough patch.

There have been many recent studies that have outlined the economic opportunities that Big Data offers on the backdrop of organisations, entrepreneurs, researchers, and practitioners producing new sustainable products or services. For instance, IDC has estimated Big Data growth to be at US$258.5 million in 2011 to $1.76 billion in 2016. This is on the back of a 46.8% five-year compound annual growth rate. Gartner estimates Big Data will create 4.4 million IT jobs globally to support Big Data by 2015 and in the Asia-Pacific region Gartner suggests 960,000 new IT jobs will be created in the next three years from Big Data. [15] [16] [17]

MarketsandMarkets has estimated Hadoop will grow into a market that will rise at a combined annual growth rate (CAGR) of 54.9% over the next five years from $1.56bn to reach $13.95bn by 2017. [18]  This is not to suggest that meeting these estimates alone will spill over to stabilise global growth. However, a focus on Big Data innovations may be one of the levers that enable
s the world to tip into a much better equilibrium. This has the possibility to accelerate through public and private partnerships with the right focus.


The science of data, the thinking, and the methods when brought together in a humanistic way are beautiful and breathtaking at the same time. The strategic intent to use Big Data and Analytics as a competitive advantage is not easy and in the aftermath of the high profile presidential election we learn that the Big Data phenomenon is real and its use can enable all organisations to pivot for success. In addition, managing data at scale to influence disruptions in markets or political systems is not isolated to America; it is within reach of every organisation, industry or sector that truly believes that data is an asset.

As organisations continue to evolve away from just looking at data as numbers on a report towards being a true asset in itself that needs to be leveraged in new ways, they too will pivot for success.


[1]. Martinez, K. (2011). The NSF Perspective on HEC-FSIO (as seen from CNS). (slidefinder). NSF Directorate for Computer and Information Science and Engineering.
[2]. Harris, D. (2011). Obama seeks data scientists for election edge. GigaOM.
[3]. Harris, D. (2012). How Obamas tech team helped deliver the 2012 election. GigaOM.
[4]. Cramer, R. (2012). Messina: Obama Won On The Small Stuff ( Romney aide admits: “We thought the game would be one thing, and it ended up being another.”). BuzzFeed.
[5]. Scherer, M. (2012). Inside the Secret World of the Data Crunchers Who Helped Obama Win. Time Magazine.
[6]. Martinez, J. (2012). Data drove Obamas ground game. The Hill.
[7]. Gallagher, S. (2012). Inside Team Romney's whale of an IT meltdown. Ars Technica. Technology Lab / Information Technology.
[8]. Burns, A., & Haberman, M. (2012). Romney's fail whale: ORCA the vote-tracker left team 'flying blind'. Politico.
[9]. Freda, A. (2012). The Promise of Big Data. Harvard School of Public Health
[10]. Mayberry, T. (2012). Big Data: The Hidden Asset in Born Digital Organisations. Torrance Mayberrys Blog.
[11]. Shiffman, G. (2012). Big Data Analytics and the Economics of Organized Violence. The Diplomatic Courier.
[12].  Robinson, H. (2012). Cloudera Impala. Paper Trail.  
[13]. Facebook Engineering. (2012). Under the Hood: Scheduling MapReduce jobs more efficiently with Corona. Facebook Engineering Notes. 
Amazon. (2012). New Asia Pacific (Sydney) Region in Australia - EC2, DynamoDB, S3, and Much More. Amazon Web Services Blog.
[15]. Stires, C., Jimenez, D., Chung, D., Tan, D., Sehgal, V., & Rajnish, A. (2012). APEJ Big Data Technology and Services 20122016 Forecast and Analysis. IDC. 
[16]. Pettey, C. (2012). Big Data Creates Big Jobs: 4.4 Million IT Jobs Globally to Support Big Data By 2015. The Gartner Group.
[17]. Barwick, H. (2012). Big data to create 960K new IT jobs in APAC by 2015: Gartner. ComputerWorld Techworld. 
[18]. Marketsandmarkets (2012). Hadoop & Big Data Analytics Market - Trends, Geographical Analysis & Worldwide Market Forecasts (2012 2017). Top Market Reports.

No comments: