Open-source tools are beginning to emerge, providing faster analysis of large volumes of data. Businesses won't always be able to wait several days for decisions...
The topic of Big Data is in the media and starting to become a reality. But curiously, partly due to the inaccuracy of the name, much attention is paid to the volume factor. The capacity to store data is growing rapidly, while its cost is falling. But an important variable is the speed with which we can quickly and effectively access, analyze, and make decisions based on this information. The velocity variable, or the speed at which the flow of information travels through the organization, is one of the main Vs of the concepts embedded in Big Data.
Access speed is already a reality in some sectors, such as finance, where the competitive advantage in buying and selling stocks is measured in microseconds. But as we interact with companies continuously, using our smartphones and tablets, they are beginning to identify the importance of interacting with their customers during the interactions themselves. Thus, an e-commerce business, like Amazon, proposes recommendations based on your purchasing patterns the moment you are active on the site. And banks try to identify fraud the moment you are using a credit card, anywhere on the planet.
But have you stopped to think about the immense volume of data and processing demand behind these operations? Identifying a purchasing pattern or a fraud attempt involves analyzing a mountain of structured and unstructured data.
With a greater variety of available data, the chances of algorithms being able to identify patterns increase. In Big Data, analysis is therefore not restricted only to larger volumes, but to a more complex amount (data diversity) and at a speed that meets business needs. An interesting example is the company Dataminr, which scours about 500 million tweets per day to look for market-moving information before it hits the news. Dataminr categorizes and analyzes each tweet in real time, filtering out spam, and comparing the information with other news sources, market prices, weather patterns, and other data to determine its importance.
This aspect, velocity, has been the target of numerous research and technological projects. Some examples are Druid and Drill. Drill is the open-source version of Google's Dremel. It is worth studying Dremel technology a bit more, and for that, I suggest reading the text http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36632.pdf. I also recommend looking at Storm, used by Twitter. It is a low-latency open-source system based on the concept of "stream processing". This processing modality is necessary when the application demands an immediate response to incoming data. Notice how we have cutting-edge technology solutions in open source! It seems to me that gradually, proprietary solutions are starting to give way to open-source solutions as they mature and build a consistent ecosystem around themselves.
These technologies complement the ubiquitous Hadoop, which was developed to operate in batch (a much slower time...) and will probably consolidate in the coming years. In turn, Hadoop distributors, such as Cloudera, develop solutions that allow fast analysis on top of the mass of data, without the need to go through batch processes. Impala is a very emblematic example.
But we have an interesting fact here. Being fast does not mean being real-time. In practice, there is no real-time, but "near real-time". Real-time means you process the data the instant it arrives. You process the data in the present. This is different from systems like Drill and Dremel, which process already stored data, albeit at high speed. The same goes for Dataminr, which processes tweets that have already been posted. In daily life, there are few cases of real-time. A recommendation algorithm like Amazon's does not need to be in real-time. It does, however, need to interact with the customer while they browse the company's website, which may take a few seconds or minutes. Real-time is necessary for an autonomous vehicle like Google's self-driving car, as it has to make a decision the instant the situation occurs.
Planning Big Data initiatives necessarily involves, to a greater or lesser degree, the various Vs, such as volume and variety. However, the velocity aspect cannot and should not be underestimated.
Businesses won't always be able to wait several days for decisions... Perhaps we need to analyze and decide at the very moment the event is happening.
(*) Cezar Taurion is CEO of Litteris Consulting, author of six books on Open Source, Innovation, Cloud Computing, and Big Data