Big Data is still a hot topic although the term itself causes various levels of cynicism and sarcastic comment these days. I think there are a number of reasons for this – it’s certainly been talked about for a long time and it’s so overused that it’s almost meaningless.

The fact is that some businesses find hundreds of GBs is a lot of data while others it’s hundreds of TB. Certainly from a technology perspective, there’s no end of tools that can help you with data and much of it is Open Source – HDFS, Hadoop, Hive, Hue, Cassandra, Spark, R. The greater challenge technologically now, certainly in Australia, is finding the people who can make this work for your organisation.

But while the real imperative for Big Data is the need to tap the digital assets all our companies are amassing for insights and innovation. It seems like every Board is asking for a data and innovation strategy. It’s not Big Data that matters – it’s Leveraged Data that we are all hunting for.

This can be very challenging for a number of reasons. Finding people who can do the analysis is hard and real insights will not come from a stranger entering your business and combing through your data. Many commentators have concluded that the best way to achieve this is to bring business people, data scientists and technologists together within business units and empower them to innovate using data. Certainly, a recent Venture Beat article about AirBnBwould support this.

Another challenge is taking innovation through to making a sustained change in the business. Whilst we can easily demand innovation, making real and effective business change based on it can often be stymied. Sometimes this can come down to lack of political will but it very often boils down to a matter of money. I’ve had one commentator lament that whilst their business has made a tidy profit by helping companies to discover new insights and innovations through data, less than half have resulted in actual changes in the business (warning: anecdote alarm!).

But while the real imperative for Big Data is the need to tap the digital assets all our companies are amassing for insights and innovation. It seems like every Board is asking for a data and innovation strategy. It’s not Big Data that matters – it’s Leveraged Data that we are all hunting for.

I can understand this. “Productionising” innovation requires incorporation of new skills, new data sets, new technologies and new processes into an environment that needs to be well controlled and ordered. Why? Because companies need to be able to justify their actions and be auditable. This requires documentation, change management etc. Basically, order. All that new stuff needs to move from the laboratory to the factory. Yes, production engineering is expensive and that’s what we are doing when we try to shove an invention into our IT departments’ hands.

Businesses need a way to innovate and to subsequently make that innovation the new normal. Yes, it’s expensive, so it needs to be enshrined in our approach to keeping our staff trained, our budgeting, our corporate behaviours and in our KPIs (and other assorted corporate horrors). Oh yes, and how about the culture of our Boards? Otherwise, why bother making people contemplate change based on innovative use of data in the first place?

Another issue (in the reverse direction) is that we often have challenges making our production system data available to innovators. It’s tightly controlled and getting access to it requires the navigation of slow and Byzantine IT change control. Also, tapping production systems can slow them down.

This often seems to result in strange skunk-works innovation projects with data sources being covertly downloaded and hacked into MS Access databases running on Fred’s PC. Fred’s training as a high school math teacher enables his IT savvy, so we are told. Suddenly we find a corporate risk report based on Fred’s database that runs on Windows XP and MS Access. Fred left quite a few years ago, now. In fact, he died in 2006. We haven’t turned his PC off since then. What do you mean “the auditors will do their nut”?

Different companies have solved these issues in different ways and I’d love to hear your stories, especially if you can share what has enabled innovation based on data in your company.

This article originally appeared on Linkedin 24 July, 2015. It has been edited for clarity.

Chris Mendes

Author Chris Mendes

Executive Consultant, Big Data and Analytics

More posts by Chris Mendes

Leave a Reply

All rights reserved IndustrieIT