AI Model performance evaluation
Service Description
Overview
This service provides an independent and reproducible evaluation of AI model performance using quantitative metrics adapted to the model task and intended application. Multitel evaluates customer-provided AI models on customer-provided test datasets in a controlled and documented execution environment. Performance metrics are selected from recognized standards or well-established scientific references, and the associated measurement uncertainties are analysed and reported. The resulting technical evidence can support model development decisions, comparison of model versions, validation activities and, where relevant, the preparation of technical documentation for AI-based medical applications.
How can the service help you?
The service is intended for SMEs and technology developers that need an objective assessment of the performance of an existing AI model, independently from the team that developed it.
Before the service, the customer may have a trained AI model and a test dataset but need independent and documented evidence of how the model performs against defined technical objectives. This may be required during model development, before selecting or releasing a model version, as part of verification and validation activities, or to generate technical evidence supporting the documentation of an AI-based medical application.
Multitel brings expertise in AI testing, quantitative performance evaluation and metrology to define appropriate evaluation metrics, execute the agreed tests, and analyse the associated measurement uncertainties.
After the service, the customer receives an independent and interpretable assessment of the model’s performance, including the quantitative results, associated measurement uncertainties and the methodology used to obtain them. These results can support development decisions, subsequent validation activities and, where relevant, technical documentation.
How will the service be delivered?
The service starts with the preparation and discussion of a ** Welcome Package**, used to align Multitel and the customer on the objectives and expected outcomes of the performance evaluation before the tests are executed. It covers the AI model and task to be evaluated, the available test dataset, the performance characteristics and metrics to be assessed, and the practical conditions for executing the tests.
Once the Welcome Package has been agreed, the customer provides the model and the required test data in the agreed format. Multitel performs the evaluation in a controlled and documented execution environment using quantitative metrics adapted to the model task and agreed evaluation objectives. The associated measurement uncertainties are systematically analysed and reported. All activities are performed under ISO 9001:2015 certified processes.
At the end of the service, the results are delivered as a detailed, structured, and interpretable evaluation report describing the evaluation methodology, quantitative performance results, and associated measurement uncertainties. The results are discussed with the customer to support their interpretation and subsequent development or validation activities.
The execution time depends on the model, dataset, evaluation task, and testing scope agreed in the Welcome Package and is defined during the quotation phase.
Use cases/examples
A healthcare technology developer has developed an AI model that classifies medical images into predefined categories. The developer needs an independent quantitative assessment of the model’s performance on a test dataset that was not used for model training.
During the Welcome Package, Multitel and the customer define the evaluation protocol, including the model and test dataset to be provided, the classes to be evaluated, and the relevant performance metrics. Depending on the evaluation objective, these may include metrics such as sensitivity, specificity, precision, recall, F1-score, and other relevant classification metrics.
Multitel executes the model on the provided test dataset using its dedicated assessment framework and calculates the agreed performance metrics. The associated measurement uncertainties are analyzed and reported to provide information on the uncertainty of the obtained performance estimates.
Outcome: the customer receives a documented and independent evaluation report containing the test methodology, quantitative performance results, associated measurement uncertainties, and the information needed to interpret the model’s measured performance. The results can be used to identify whether the model meets the predefined performance objectives and to provide technical evidence for subsequent validation activities and technical documentation.
