Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis

kaestner@cs.cmu.edu (P. Jamshidi, N. Siegmund, M. Velez, C. Kästner, A. Patel, Y. Agarwal)
Fri, 09 Feb 2018 09:07:32 -0500

Pooyan Jamshidi, Norbert Siegmund, Miguel Velez, Christian Kästner, Akshay Patel, and Yuvraj Agarwal. Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 497--508, Los Alamitos, CA: IEEE Computer Society, November 2017.

Modern software systems provide many configuration options which not only influence their functionality but also non-functional properties such as response-time. To understand and predict the effect of configuration options, several sampling, analysis, and learning strategies have been proposed, albeit often with significant cost to cover the highly dimensional configuration space. Recently, transfer learning has been applied to reduce the effort of constructing performance models by transferring knowledge about performance behavior across environments. While this line of research is promising to learn more accurate models at lower cost, it is unclear until now why and when transfer learning works for performance modeling and analysis in highly configurable systems. To shed light on when it is beneficial to apply transfer learning, we conducted an empirical study on four popular software systems, varying software configurations and environmental conditions, such as hardware, workload, and software versions, to identify the key knowledge pieces that can be exploited for transfer learning. Our results show that in small environmental changes (e.g., homogeneous workload change), by applying a linear transformation to the performance model of the source environment, we can understand the performance behavior of the target environment, while for severe environmental changes (e.g., drastic workload change) we can transfer only knowledge that makes sampling in the target environment more efficient, e.g., by reducing the dimensionality of the configuration space.