Prepare your business for 'big data' revolution

Likewise, any software product that collects, stores or analyses data is now invariably described as a big data solution.

Big data in many respects results from the convergence of cloud, mobile and social technologies. These technologies, together with new techniques for data storage, such as Hadoop, and new techniques for squeezing value out of data (data science and machine learning) offer amazing opportunities for competitive advantage. Big data not only provides opportunities for new solutions and new products, but it stands poised to stimulate the economy and enrich our lives in many ways. But it’s not all an upside. Businesses that continue using traditional models may be threatened with extinction as their competitors establish a big data advantage.

Though collection, processing and storage of unparalleled volumes of data creates new challenges for IT, these are not dissimilar to the challenges created by the long-established trend of increasing data volumes and processing power. We’re used to exponential growth. 

But it’s not enough simply to collect and store more data. We need to establish algorithms and business processes around the data that can transform business process. Establishing these smarter algorithms is going to severely challenge many enterprises.

The biggest retailers and online businesses have shown that it is possible to create smarter algorithms for campaign optimisation, customer-relationship management, dynamic pricing, risk/fraud assessment and so on. But to succeed, these algorithms need to be automated and integrated into the business process. 

It’s not enough to draw a pie-chart from the data and present it to the CMO or CEO. The data needs to be analysed and used to dynamically generate business decisions on both a micro- (individual customer) and macro- (product line, geographical) level.

Many enterprises are only just starting to realise that big data represents either an unparalleled opportunity or maybe even an existential threat. Amid this backdrop, the acquisition of data is critical. Enterprises that wait for a solid business plan before capturing all possible data will be at a disadvantage compared to those who start collecting everything immediately.

So the best advice for most enterprises is to make sure you have a big data capture, storage and processing platform in place immediately. Capture everything that you can now, and use technologies such as Hadoop to provide economies of storage that allow for a “keep it all” strategy.

Simultaneously, start developing in-house data science skills or find partners who can provide these for you. Many corporations have professionals with the necessary personal and professional attributes (statistical, programming, business) to develop into the next generation of data scientists.

Exactly how the application of smarter algorithms to more extensive data sets will transform a business differs across industries. But for almost all industries, this transformation is real and imminent. Even if it’s not entirely clear how big data will transform your business, it’s prudent to start data acquisition and data science skill-up immediately. When paradigm shifts occur, those who hesitate are often lost.

 

Matthew Johnston is managing director of Dell Software, South Asia.