The Web-crawling wizard experience detailed earlier left us with Microsoft Visual C++ exception windows covering the screen and a crashed version of Protocol Modeler. We encountered similar disappointments on other long-running tests. For example, we tried running a SQL Disclosure attack on a single user-supplied input on a Web page being posted back to a Web server. We watched the logs on the HTTP server (the user will often find him or herself monitoring closer to the target application) and got a glimpse of the types of attempted queries. We observed 8,600 attack queries before Protocol Modeler finally crashed (Cenzic said the fault injector was probably close to finishing, given that number). Unfortunately the product doesn't do any checkpointing, so we couldn't find out if any vulnerabilities were discovered in those queries.
A more critical feature gap is the lack of any indication of the approximate and relative run-times of the fault injectors, some of which can run for hours or days, depending on the size of the test. Even nicer would be a: "This test requires about 7 minutes per iteration, times 60 loops = 7 hours" message.
Art Meets Science
The Protocol Modeler experience is, in many ways, like staring at a blank 6-foot canvas with a full palette in your hands. Using the tool successfully takes creativity. We don't claim to be the Picasso of the Protocol Modeler world--frankly, much of our work was painting by numbers with the wizards, though we did begin to devise some interesting tests as we became more comfortable with the Protocol Modeler environment. Make no mistake: The product is difficult to use. Allow at least a full week to ramp up on the tool and assemble a preliminary test network. This assumes you have advanced knowledge of both IP networking protocols (at every level) and advanced knowledge of security vulnerability theory and practice. If not, allow much more time. According to Cenzic, a few days of on-site training come with the purchase price. We recommend taking full advantage of this.
At press time Cenzic had launched a new product, Hailstorm Web. The core engine technology is the same as Hailstorm Protocol Modeler but also includes extensive workflow management for structured use within an organization. Security expert and QA analyst roles let application security testing tasks be distributed logically. Predictably, the focus is on HTTP-based applications.