On the heels of big data grabbing headlines the world over for its role in President Obama's re-election, could 2013 be the year big data makes the big leap into the mainstream, especially business?
Every second of every day technology captures and stores an unimaginable amount of data. Ninety per cent of the world's data was created or captured in just the last two years. And 80 per cent of that data is unstructured, coming from traditional systems, images, videos, tweets, posts and e-mails. The challenge is clear: how do we use all this data to make better decisions faster?
The Obama campaign sure knew. It crunched big data from social media, voters' lists, vehicle registrations and much more to ascertain things like the type of car people drive and whether they were likely to vote for the President. The Obama team then zeroed in on those likely to vote Democrat trying to get them to the polls, and didn't waste time and money on people who were likely voting Republican.
But there is also a downside; and the old saying "lies, damned lies and statistics" comes to mind. Big data ain't easy and assumptions made when interpreting the data are critical. For example, one of the world's best-known polling firms, Gallup, has laid eggs in the last three U.S. elections and may have contributed to Mitt Romney losing the White House in November's election by feeding his campaign inaccurate information from misinterpreted data.
Maybe it's this fear of misinterpretation, or maybe it's simple ignorance about the issue, but most businesses are not taking advantage of big data, either. They're using traditional business intelligence (BI) tools without combining them with data from social media posts and tweets and data from machines and sensors, the so-called "Internet of things."
Unlike finance and production areas of companies, departments like marketing and sales have been largely left out of the data-driven game. This isn't a secret: advertising's informal godfather, John Wanamaker, famously said, "Half the money I spend on advertising is wasted; the trouble is I don't know which half."
But that's no longer acceptable. Big data offers a solution that many cutting-edge firms like Google and Amazon are applying to traditionally "soft" disciplines like advertising and customer experience. The task ahead of us is to take the promise of big data and realize it in every department and in every industry. Some surprising pioneers are already leading the way:
• General Electric estimates that a big data driven industrial Internet could boost energy efficiency and save trillions globally in economic activity by 2025.
• It's estimated that if the U.S. healthcare system were to use big data to drive efficiency and quality, the sector could create more than $300 billion in value every year. Proportionally, the potential savings in Canada may be even higher. The potential in this space extends far beyond cost-savings: it seems likely that the researcher who discovers a life-changing cancer treatment breakthrough is in some capacity a data scientist.
• Since 2011 the Santa Cruz police department has been examining their crime incidence records for trends and patterns and assigning police officers to patrol where crimes are likely to occur. The project has been very successful, allowing the department to stop more crimes with fewer officers.
To make the big data future a reality, we'll need a new generation of data scientists, data-savvy senior executives, and flexible and responsive organizational structures. Managers in particular need to admit they don't always know the answer and that they must look to the data for guidance. Gathering a lot of data is not enough.
And it is one of the reasons for the upcoming Big Data Congress January 24 in Saint John. Co-hosted by the New Brunswick Information Technology Council (NBITC) and my firm, T4G Limited, the Big Data Congress has attracted some of the brightest minds in North America to share ideas on how best to bring together the public and private sectors to ensure well-trained data scientists are creating big data solutions for the 21st century.
We're excited about big data but that doesn't mean it's without challenges. We'd be remiss if we didn't identify the significant challenges of privacy and security concerns. Citizens, governments and firms will have to balance the real potential of big data solutions against legitimate rights to privacy. There is a way forward; it's just going to take a lot of thoughtful collaboration.