Many companies, especially mid-size ones, have avoided some big data projects because the high-end, high-performance storage they specify as standard for those projects costs half a million dollars to store 20 to 40 terabytes, according to an interview consultancy Sandhill Partners did with Fred Gallagher, general manager of big data cloud developer Actian.
Actian's main product, Vectorwise, scales more efficiently as users add more processing cores than it does simply adding more servers. That approach--making relatively inexpensive storage hardware perform up to the level of its more-expensive relatives--is the more effective way to scale storage networks to keep up with data that gets bigger and bigger ad infinitum, Gallagher said.
Scale-out NAS boxes do much the same thing: they make big data projects with budget-busting levels of growth slightly more palatable, or at least more affordable.
Dell launched a big data storage package July 23--a rack of products based on Apache Hadoop, that starts with two TB of storage and ranges up into petabytes. The product includes Dell's Cloudera data-management software, Apache Hadoop, Force 10 networking, and Dell PowerEdge servers. It also includes data compression technology capable of shrinking data at ratios of 40:1. This frees up disk space, reduces the number of units needed for a big data project, and saves customers money—but they still must spend vast amounts on storage for data that, in most cases, has yet to prove its worth.
It will, predicts Alvarez and other analysts.
Massive amounts of data and the ability to analyze it quickly enough that the results are still useful are so important as decision-making tools that data forms the fourth paradigm of computer science – a whole new way of considering, analyzing, and making use of data, according to computer scientist Jim Gray, who just published a book of essays called The Fourth Paradigm: Data-Intensive Scientific Discovery.
The first three paradigms, according to the British computer scientist Amnon Eden, differed radically in the assumptions with which they approached computers. The first treated computers as a branch of mathematics in which applications were formulae designed to produce a practical result. The second treated computer science as an engineering discipline and programs as data. The third, the scientific paradigm, treats applications as processes on a par with those of the human mind, an approach that assumes programs will eventually develop their own intelligence.
The idea is a little abstract for IT, but each paradigm brought with it a new way of analyzing problems: first according to observation, then by theory, and finally by simulation. Big data goes beyond all of those by promising to deliver insights so integrally concealed in massive amounts of data that direct observation and analysis by humans can never coax it out. Finding those answers requires enough data to make indirect correlations clear, however. Having enough data to mine for indirect correlations requires having enough storage hardware to house all that data and access it quickly.
Having that much storage hardware--there's no other way to say it, according to Alvarez, and Woo--means spending a lot more money on storage, no matter how efficiently it can be made to run or how cheaply it can be bought.
Big data places heavy demands on storage infrastructure. In the new, all-digital Big Storage issue of InformationWeek Government, find out how federal agencies must adapt their architectures and policies to optimize it all. Also, we explain why tape storage continues to survive and thrive.
