Barracuda Networks Responds

The following post contains the correspondence between Barracuda Networks and Frank Bulk in response to Frank???s blog on Barracuda???s representation of its Spam Firewall e-mail capacity.
Barracuda Networks' Official Response:

Barracuda Networks stands behind its stated e-mail capacity specifications for each of our Barracuda Spam Firewall models. We also recognize, and clearly indicate on our marketing materials, that the e-mail capacity for customers will vary depending on the customer's environment. Further, the datasheet specifications for e-mail capacity reflect an environment in which over 99% of e-mail can be blocked through connection management, a situation typical of directory harvest and other denial of service attacks. A significant percentage of our customers, including small ISPs, have successfully used the Barracuda Spam Firewall 300 model to not only successfully block spam but to also protect their e-mail infrastructure against ongoing attacks at the stated specifications.

We would welcome your further feedback on this issue.

Frank Bulk replies:

It's hard to take a vendor's press department comments about a system???s operational capacity seriously when the vendor's own senior tech and a customer's experience confirm otherwise.

1. Your senior tech stated that "Active E-mail User" most directly affects performance when quarantining is used. No one at Barracuda has pointed to the specific impact of more e-mail users on the Barracuda's system, nor quantified it. I believe that's because additional e-mail users, when there is no quarantining in place, have a very negligible impact on the system's ability to process e-mail. In the ISP???s case, no quarantining was in use except for four accounts. If anything, that should help the box exceed the stated 2M messages/day.

2. I agree with you that several factors, specifically e-mail mix and configuration, affect system load, but they shouldn't affect the ability of a system to achieve its own stated metrics, within a few percentage points. If I had discovered that scanning 50% of the messages would reduce the processing capacity from 2M messages per day to 1.8M per day I would consider a 10% drop in capacity quite reasonable, but in this case it's a reduction of 87.5%. Better for the marketing literatures to state:
Capacity in relation to initial connections blocked: 2M/day at 99%, 1M/day for 80%, 306K/day for 75%, 167K/day for 50%, etc.
Then potential buyers can refer to their current metrics and identify the model that best suits their environment. Without that context, potential buyers are left to assume that the system can handle 2M messages/day, give or take a few percentage points depending on their setup. They are left totally in the dark that scanned messages impact capacity so dramatically.

3. The numbers stated in the marketing sheet ought to reflect an average configuration, but they do not. All the extremes are listed: 2M message per day, 1000 accounts, 250 domains. The reality is that not all those metrics can be achieved simultaneously unless at least 99%-plus of the messages are blocked upon connection using Invalid Recipient checking, Rate Control enforcement, IP Blacklist, RBLs and DNS lookup.

4. Using more reasonable but still rather high connection blocking rates, such as 75%, pulls the performance down from 2M messages/day to 306K messages per day, just 15.3% of the voulume on the marketing sheet. When the senior tech used a scanning rate of 49.5% he calculated that each Model 300 should be able to handle 167K messages/day, which is just 8.4% of the marketing literature's documented capacity. The fact that each of the ISP???s boxes can process 250K messages/day suggests that quarantining does have a huge operational impact, and the lack of it more than mitigates any minor load that more users has on the system (back to point 1).

5. The 99%-plus connection blocking is absolutely atypical. Yes, I'm sure there are times in the day that some of Barracuda's customers experience directory harvesting, e-mail denial of service, or other attacks, but if you look at MessageLabs statistics, spam rates range from 38% to 68% between verticals. Those rates include all those attacks. 99% is not sustained, day in and day out.

6. So 2M/messages per day only applies when under attack? That's definitely not in the marketing literature.

7. Customers, when they see 2M messages/day, should be able to expect the system to process 2M accepted messages, not just block 1.98M messages and let 20K through.

8. The ISP???s message volume didn't double in one year, it actually increased 550%. The only reason the ISP did not go for the Model 200 was missing features, not capacity. But it expected the pairs to be able to handle 2M messages/day, each.
--Frank Bulk

Barracuda Responds:

Thank you for your further feedback. Our intention is not to mislead customers but rather simply tout the benefits we offer with each model. A Barracuda Spam Firewall 300 does stand up to 2M connection attempts per day when under attack, and even smaller corporate customers appreciate this point. From our correspondence with you, we will plan to do more in terms of educating our customers on how our products are designed, and for what environments, so that they can be sized to achieve peak performance. Along these lines we plan to publish a white paper, specifically for ISPs, that explains various issues encountered in different network environments and recommends consultation with a Barracuda Networks system engineer for final sizing. We will also ensure that our sales and systems engineers are fully trained on all of the issues involved in appropriately sizing a unit leading up to sale.