It feels like every time you turn around these days, someone (mainly in the media) is talking about Hadoop. How it is changing how businesses work. How Hadoop is innovating big data technologies. How everyone is using Hadoop to bolster data analytics. But is that really the case? Fortune posted an interesting article earlier this week discussing, you guessed it, Hadoop. What makes this discussion different from all the rest? Instead of talking about how Hadoop is the ultimate in data analytics, it actually looks at stats and insight from Gartner Research, which details the truth behind the hype. Mainly that the Hadoop hype isn’t all it’s cracked up to be, and in fact, the numbers from a recent Gartner survey paint a very different picture than the one you normally see. “And yet … adoption isn’t setting the world on fire, according to a new Gartner survey which shows less than 50% of 284 respondents have invested in Hadoop technology or even plan to do so. Just over a quarter (26%) of respondents said they are deploying or experimenting with Hadoop and only 11% said they plan to invest in Hadoop within 12 months.” ~ Barb Darrow, Fortune These stats are incredibly telling about how people feel about Hadoop in actual practice and not just in theory. It’s not the need or desire for data analytics itself that is the problem. Organizations are actively searching for a way to quickly, efficiently and effectively analyze batch, streaming, unstructured and multi-structured data in order to make better decisions. So why? Why is this solution that has been billed as the answer to big data problems actually realizing less than stellar adoption numbers? Let’s take out the fact that it’s difficult to still gain the answers that big data needs. Put to the side that data is often stale by the time it can actually be analyzed because of lengthy ETL and indexing processes. Don’t consider the fact that business questions aren’t answered until after the data is likely out of date because of the stop-and-go nature of the traditional data pipeline. Let’s simply look at the initial implementation, and Gartner has some interesting insight. “The upshot seems to be that while Hadoop can handle huge data sets and make them useable, the capabilities needed to set up and run Hadoop remain scarce and expensive. And, for at least a subset of the corporate population, the perceived advantages do not yet outweigh the cost and complications.” ~ Darrow Now the adoption numbers make sense in comparison to the amount of hype seen in the market. And this is only compounded by the fact that Gartner also found that a lack of skilled Hadoop workers is also hindering organizations’ desires to move forward. How good is a solution if the users find it to be too costly and complex to bother with? It’s time for a dramatic shift in the data analytics market. Organizations have data, and being able to extract timely value from that data often has a very real impact on not only their bottom line but, in some cases—including fraud detection, forensics, healthcare and more—the general public as well. That’s not to say that Hadoop doesn’t have its place. Hadoop was the answer during a time when many organizations wanted data analytics, but the market hadn’t caught up with the technology required for advanced analytics. It also has its place moving forward with certain types of data (mainly when looking at purely historical information) and when mean time to decision doesn’t have to be condensed in order to preserve the data’s value. However, the growing needs of the data analytics market, and the need for a simple, easy-to-use solution that doesn’t take a “Ryft expert” to operate, are the main drivers for how we build our solutions. Data is an important part of day-to-day operations for many organizations, and even more important to planning for their successful future. It shouldn’t be held hostage by unneeded cost and complexity. 2 responses to “Hadoop vs. the Hype” Pingback: Supercharging Spark by 100X with New Ryft Connector for Spark Pingback: Ryft VP of Engineering Pat McGarry Talks Analytics and Live Music Leave a Reply Cancel reply Your email address will not be published. Required fields are marked *Comment Name * Email * Website Save my name, email, and website in this browser for the next time I comment.