Rigorous performance evaluation made easy

Empirical systems research is facing a dilemma. Minor aspects of an experimental setup can have a significant impact on its associated performance measurements and potentially invalidate conclusions drawn from them. Examples of such influences, often called hidden factors, include binary link order, process environment size, compiler generated randomized symbol names, or group scheduler assignments. The growth in complexity and size of modern systems will further aggravate these issues, especially with the given time pressure of producing results. So how can one trust any reported empirical analysis of a new idea or concept in computer science?

DataMill is a community-based, easy-to-use, open benchmarking infrastructure for performance evaluation. DataMill facilitates producing robust, reliable, and reproducible results. The infrastructure incorporates the latest results on hidden factors and automates the variation of these factors.

DataMill is hosted and developed by the Real-Time Embedded Software Group at the University of Waterloo. Contact us at <datamill@uwaterloo.ca>.


Associated research papers

  • A. Oliveira, J-C. Petkovich, T. Reidemeister, and S. Fischmeister, "DataMill: Rigorous Performance Evaluation Made Easy", Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), Prague, Czech Republic, April, 2013.
  • A. Oliveira, S. Fischmeister, A. Diwan, M. Hauswirth, and P. Sweeney, "Why You Should Care About Quantile Regression", Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2013), Houston, USA, March, 2013.
  • A. Oliveira, J-C. Petkovich, T. Reidemeister, and S. Fischmeister, "How Much Does Memory Layout Impact Performance? A Wide Study", Workshop on Reproducible Research Methodologies (REPRODUCE 2014), Orlando, Florida, USA, February 2014.