MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities

kaestner@cs.cmu.edu (K. Maffey, K. Dotterrer, J. Niemann, I. Cruickshank, G. Lewis, C. Kästner)
Fri, 3 May 2024 10:33:49 GMT

Katherine R. Maffey, Kyle Dotterrer, Jennifer Niemann, Iain Cruickshank, Grace Lewis, and Christian Kästner. MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities. In Proceedings of the Proc. International Conference on Software Engineering -- New Ideas Track (ICSE-NIER), pages 31--36, May 2023.

Many organizations seek to ensure that machine learning (ML) and artificial intelligence (AI) systems work as intended in production but currently do not have a cohesive methodology in place to do so. To fill this gap, we propose MLTE (Machine Learning Test and Evaluation, colloquially referred to as “melt”), a framework and implementation to evaluate ML models and systems. The framework compiles state-of-the-art evaluation techniques into an organizational process for interdisciplinary teams, including model developers, software engineers, system owners, and other stakeholders. The MLTE tooling supports this process by providing a domain-specific language that teams can use to express model requirements, an infrastructure to define, generate, and collect ML evaluation metrics, and the means to communicate results.