Big Data analysis has never been a simple task. The sheer quantity and lack of uniform data poses a logistical nightmare for traditional analytical means. Frameworks like Hadoop and multi-server NoSQL databases allow for greater flexibility, but disparate data still needs a human to decode the core message. Is it time that artificial intelligence is incorporated into the big data playbook?

An Impending Big Data Talent Gap

IDC predicts that there will be 181,000 deep analytics roles by 2018. More than five times that number of other roles will require some level of data management or interpretation. Big data is becoming an integral part of the workforce and there’s no sign of that slowing.

Hadoop started the trend toward more comprehensive big data tools.  Now, new products are expanding and diversifying the ways that big data operates. The speed at which these products are advancing makes it difficult for all but the exceptional few to keep pace.

The Limitations of Technology

Today’s analytics technology is powered by some impressive components, but  major impediments to better and faster data analytics still remain. By 2020, there will 44 zettabytes of data in existence. That will certainly require an advanced level of computer to parse it all. At a glance, it seems that our best hardware may be capable of handling it.

The fastest supercomputer in the world, China’s Tianhe-2, can process 33.86 petaflops per second. (To give some perspective, one petaflop equals about a quadrillion or a thousand trillion operations). But even that impressive machine requires oversight and guidance to produce any worthwhile results. The greatest potential lies in providing programs with heuristic capabilities, or the ability to discover and learn on their own.

IBM’s famous experimentation with computer learning, Watson, has achieved a great deal but still has its limitations. Though it is capable of taking big data queries – an esoteric question in Jeopardy, the symptoms reported by a medical patient, or the potential location of a shale deposit – and reviewing them for an incisive answer, it still, in the words of long-time IBM researcher Dave Ferrucci, can’t “simulate the world at a level deeper than statistics.”

That era might someday come about, but not without further exploration and breakthroughs in the realm of Deep Learning, which hopes to create the mental pliancy of a human being in a collective of processors. And that technology is not there yet.

Until True AI Appears, What Do We Do About Big Data?

Though we’ve made progress in processing speeds and predictive analysis, the enterprise market is not yet at the point where programs will be able to completely take over for data analysts. Meanwhile, the big data skills gap still threatens to slow the progress of businesses worldwide. To remain competitive, businesses will need to employ a variety of tactics to attract and acquire big data professionals.