(click image for larger view and for slideshow)
When information security defenses fail, what happens next?
Ideally, chief security officers (CSOs) could simply order all network traffic from around the time of the attack to be replayed and analyzed in depth by their incident response team. Except that to date, that hasn't been possible.
"The idea of full packet capture has been likened to network TiVo, and largely discounted," said Michael Baker, CTO at Packetloop, in an interview at this week's Black Hat Europe conference in Amsterdam. That's because although capturing and storing packet data is simple, no one has had the big data tools required to effectively analyze terabytes' worth of packet captures (a.k.a. pcaps).
[ How to secure a videoconferencing system.
See Videoconferencing Systems Vulnerable To Hackers. ]
So Baker and two other members of a new Australian company, Packetloop, built such a tool. Dubbed Packetpig, Baker released the open-source tool on Wednesday, just minutes before delivering a related presentation, "Finding Needles In Haystacks (The Size Of Countries)" at the Amsterdam conference.
"I built the product to answer two questions: Am I overly targeted, and am I vulnerable?" said Baker. "This is what CSOs would love to know: How many attacks am I getting? How do I compare with other CSOs? If you have an attacks-per-hour ratio, that's the sort of thing you could give away, because it's an attack thing, it's not about how secure you are." Furthermore, many CSOs don't want perfect security, said Baker. They just want to be better enough than their peers that attackers will prefer the easier target.
Baker said one challenge with analyzing large sets of data traditionally has been that as pcaps age, tools tend to aggregate the data, meaning that a minute-by-minute look at packets becomes averaged into an hour, and hours later rolled up into days or even months. "The key point is to not lose fidelity," he said. But that's only possible by doing full packet capture, and then analyzing all of those packets for a desired timeframe.
Enter Packetpig, which is designed to analyze packets' IP headers, protocols, and conversations and flows, as well as to handle threat analysis, geo-location, operating system fingerprinting, and file dissection across large sets of data.
Packetpig is built on Pig, which is a platform--programmed in a language called Pig Latin--for creating MapReduce jobs, a concept Google outlined in a 2004 research paper. The jobs spread problems involving large amounts of data across multiple nodes. In particular, Packetpig is a series of data-analysis jobs that run on Hadoop, an open-source implementation of MapReduce, to handle the replication of data across multiple nodes. These nodes could be anything from spare servers or compute time scrounged by the information security group to Amazon Simple Storage Service (Amazon S3).
Packetpig offers "loaders" that extract pcap data from Argus, Bro, Flowgrep, Network Miner, Sguil, and Snort. "They're generally used like electron microscopes, as isolated tools [to analyze] very small packet captures," said Baker. But visualizing all pcaps over a long period of time helps an incident response team spot historical attacks that it might have missed when they were unfolding.
"This is helpful for finding zero-day threats in old data," he said. "Remember, Snort is a signature-based system, mostly, so as I update my signatures, I can go back and find zero days in data." In addition, the bigger the data set--meaning, the more pcaps analyzed, over a long period of time--the easier it is to calculate background noise, which makes unusual activity easier to spot. "Let's say you find something interesting, how do you go back and dump it, saying, 'Show me everything from that box for the past three months that got sent to China, or Iran'?"
