Package Ranking Technique >
Rule001 - Evaluation Criteria Definition
Determine the criteria that qualified application packages must satisfy to become a finalist. Use a prioritized list of requirements as a starting point. Determine which requirements are the key factors for screening the field down to a set of finalists. Translate each key requirement into one or more evaluation criterion that will be used to evaluate whether the package meets the specified requirement. Each evaluation criterion should be measurable and observable by examining the package, its documentation, or a demonstration.
Evaluation criteria fall into three major categories: business, technical, and vendor. The number and rigor of criteria defined in each category should be driven by the Statement of Principles. For example, if a long-term relationship with the package vendor is sought, the vendor criteria should be more detailed and rigorous.
1. Business Evaluation Criteria
Examine the key business requirements relating to:
- business rules
- data structure, format, and validation
- events, processing, and workflow
- screens, reports, and queries
- frequency, volume, and growth
- performance and security
- other important features
Define one or more measurable and observable evaluation criterion for each key business requirement. Review the final list from two perspectives: 1) the set of criteria should be sufficient to narrow the field down to the desired number of finalists; and, 2) the set of criteria should be able to be scored for each qualified package within the allotted time. Define a single final criterion that could be used as a "tie breaker" for the business criteria in the event that the scoring is close.
2. Technical Evaluation Criteria
Examine the key technical requirements relating to:
- methods for customization, interfacing, and conversion
- openness of architecture and technical platforms
- methods for performance tuning and implementation
- methods for security, audits, backup, and recovery
- methods for change and configuration management (e.g., rolling forward customized features, upward compatibility)
- other important technical features
Define one or more measurable and observable evaluation criterion for each key technical requirement. Review the final list from two perspectives: 1) the set of criteria should be sufficient to narrow the field down to the desired number of finalists; and, 2) the set of criteria should be able to be scored for each qualified package within the allotted time. Define a single final criterion that could be used as a "tie breaker" for the technical criteria in the event that the scoring is close.
3. Vendor Evaluation Criteria
Examine the key vendor requirements relating to:
- size and location
- stability and financial strength
- quality and types of services
- quality and types of agreements and policies
- strategic vision of the marketplace and products
- development and release management approach
- product delivery track record
- pricing
- other important vendor criteria
Define one or more measurable and observable evaluation criterion for each key vendor requirement. Review the final list from two perspectives: 1) the set of criteria should be sufficient to narrow the field down to the desired number of finalists; and, 2) the set of criteria should be able to be scored for each qualified vendor within the allotted time. Define a single final criterion that could be used as a "tie breaker" for the vendor criteria in the event that the scoring is close.