Impact of landscape ruggedness on performance modeling

<p>Previous efforts in our community has been focusing on developing new performance models, whereas little efforts are devoted to interrogate the bottlenecks of existing approaches. Here we demonstrate the accuracy of predictive models in fitting configuration performance can heavily rely on the ruggedness of the underlying.</p> <p>The set of plots presented here depicts the R2 score of a XGBoost regressor fitting on the configuration data of an entire landscape against the number of local optima/autocorrelation of each landscape. Each point in each subplot represents a workload, with the x and y values showing the R2 score and the number of local optima/autocorrelation of the corresponding landscape. A linear regression fit linear, along with 95% confidence interval of the fit, as well as Spearman's p, are also shown in the plots.&nbsp;</p> <p>From the results, it is very clear to see that the accuracy of the fit is significantly correlated with the ruggedness of the landscape, despite we have employed the same model. Notably, on LLVM, the R2 score can drop from nearly 0.9 all the way down to around 0.2 with the increase in landscape ruggedness.</p> <p>Therefore, local optima and landscape ruggedness can be play a critical role in performance modeling, which have not been previously reported. We intend to add this finding to the main text, since it can potentially inspire new performance modeling methods that can tackle rugged landscapes.</p>