We also considered the products' security and performance. Because of the difficulties inherent in testing performance for these products, our performance evaluations focused on how long it took for a report to be generated. We didn't clock specific times, but during our tests it was obvious that all the products except MicroStrategy are on an even footing in this category. The other products returned results lickety-split, but MicroStrategy took noticeably longer. Performance should not be a huge factor when deciding on a business-intelligence solution, but any tool that requires users to wait for a report for too long will sit unused. For this reason, we kept an eye on this aspect during our product testing.
Security, on the other hand, is a huge issue--not in the omigod-a-hacker-just-got-onto-our-system way, but in ensuring that data access is limited to authorized employees. MicroStrategy provided the most comprehensive security model, offering RBAC (role-based access control) and ACL (access-control lists) to limit access not only to the data but to specific functionality within the product. Cognos Series 7 and Brio Intelligence do an excellent job of addressing this issue, but do not extend their security models to the feature-use level as MicroStrategy does. By comparison, Microsoft's Data Analyzer relies solely on the security provided by a database--which may or may not be active if the data is aggregated from multiple sources in a warehouse.
Pricing
Although pricing was only a small factor in our evaluation--our scenario was based on a small to midsize company, and because package deals were available, cost was less of a factor than it would be in a large organization--it is still important to consider. Microsoft and Cognos use a flat per-user fee. Pricing for the other products depends on the deployment platform of the server as well as the specific number of users and their needs. To put all the participants on even footing, we asked for pricing for a company with 100 users (business analysts), three report developers and one administrator. The products would be deployed on a dual-processor, 1.2-GHz Pentium 4 server running Windows 2000. We also included the vendors' annual maintenance fees, which ranged from 0 (Microsoft) to 25 percent (Cognos).
Depending on your needs, a business-intelligence solution's pricing could range from affordable for even small organizations to prohibitive (and note that IT, not sales or marketing, will pay, because it's an application that requires installation and maintenance). Because most of the vendors license their products on a per-user basis, it's often feasible to start with a small implementation that you can later expand as needed.