Of course, mere packet count wasn't the primary factor determining whether a target suffered an outage. Invasive tests, such as brute-forcing accounts and executing DoS attacks, can also crash a target system.
OS fingerprinting: Scanners send targets malformed IP requests in an attempt to extract a response. The manner in which an OS responds to these requests helps the scanner identify the type of OS that has replied. Depending on the request, as well as the maturity of the OS's IP stack, a system might encounter a failure. For example, Nessus' methods will crash older systems, while FoundScan's more RFC-friendly approach to fingerprinting rarely does.
Furthermore, we tested each product for its ability to remain stable while scanning large address ranges. Although our test bed contained fewer than 30 machines, a VA scanner must examine any range of systems designated as its target base. Our network was segmented into four class "C" address ranges, so that's what we submitted to our scanners. Most of the products handled the load with ease. We input all our addresses into each of the products; however, Beyond Security's scanner wasn't able to finish the workload, and Vigilante.com's SecureScan NX failed several times before presenting us with a completed scan.
In enterprise environments, a more distributed deployment method--as opposed to deploying a single scanning device--can prove beneficial. Enterprises do not want to burn WAN bandwidth with vulnerability-scanner traffic, and scanners often encounter problems with system identification across multiple routers, proxy servers and firewalls. In fact, we found one segment of our mock environment especially tricky for several products under test--on TCP- and UDP-based identification scans, Rapid7's NeXpose and Vigilante.com's SecureScan reported responses from systems that didn't exist! Best we could tell, our Cisco PIX firewall (acting as a simple router in this case) was sending replies to the scanner, indicating that there was no host on the other end; the scanner interpreted the PIX's response as a positive host finding. This caused a tremendous amount of overhead, as these scanners spent hours attempting to identify what services were running on nonexistent servers. This is where products such as eEye Retina and Tenable Lightning can prove useful, by allowing multiple scanners to be deployed throughout the environment, all reporting back to a single aggregator.
Price: We waited until we were nearly finished testing to look at prices because we didn't want our opinions skewed by our perception of what a particular product "should" provide for its price. We found that product pricing accurately matched the features being offered, with a few exceptions: Tenable's Nessus appliance, which retails at $20,000 with an additional $12,000 to license Lightning for five users; Beyond Security's Automated Scanning Server, which retails at $12,000; and Rapid7's NeXpose software, starting at $8,750 for only 64 specified IP addresses. These products don't seem worth the price.