Performance Evaluation

General Informations

The coordinator for these lectures is Jean-Marc Vincent . The lecturers are Jean-Marc Vincent and Arnaud Legrand .

The next lecture will be on Friday 21/10 in room H102 from 13h15 to 16h30.

Objectives

The aim of this course is to provide the fundamental basis for performance evaluation of computer systems. Two approaches are developed:

• performance measurement: based on experimental platforms (benchmarks or owner instrumented code execution), how to analyze data and synthesize performance indexes
• performance modeling: from a description of resources and the behavior of applications, how to predict the performance of the application

Program and expected schedule

• Friday 14/10/2011 (13h30-16h45): Jean-Marc Vincent Introduction, performance indexes, data analysis, modeling and inference.
• Documents: slides EP_{01Introduction}.pdf
• References:
  • Preliminary guidelines for empirical research in software engineering Kitchenham, B.A.; Pfleeger, S.L.; Pickard, L.M.; Jones, P.W.; Hoaglin, D.C.; El Emam, K.; Rosenberg, J.; Software Engineering, IEEE Transactions on , Volume: 28 Issue: 8, Aug 2002 Page(s): 721 -734
  • Partie I : R. Jain, [[http://www.cs.wustl.edu/~jain/books/perfbook.htm][The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling]] Wiley- Interscience, New York, NY, April 1991.
• Friday 21/10/2011 (13h30-16h45): Arnaud Legrand Measurement on computer systems (benchmarking, observation, tracing, monitoring, profiling).
• Documents: slides EP_{02Measurements}.pdf
• References:
  • David J. Lilja. Measuring Computer Performance: A Practitioner’s Guide Hardcover. Cambridge University Press, 2000.
  • R. Jain, [[http://www.cs.wustl.edu/~jain/books/perfbook.htm][The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling]] Wiley- Interscience, New York, NY, April 1991.
• Friday 4/11/2011 (13h30-16h45): Arnaud Legrand Introduction to discrete event simulation and emulation of computer systems. The simulation crux: validation. Introduction to network topology generation.
• Documents: slides EP_{03Simulation1}.pdf, EP_{03Simulation2}.pdf, EP_{03Simulation3}.pdf
• References:
  • Banks J., J. S. Carson and B. L. Nelson 1996. Discrete-Event System Simulation, second edition. Prentice-Hall, Englewood Cliffs, NJ. GORDON G. 1996.
  • R. Jain, [[http://www.cs.wustl.edu/~jain/books/perfbook.htm][The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling]] Wiley-Interscience, New York, NY, April 1991.
  • On Traffic Phase Effects in Packet-Switched Gateways by Sally Floyd and Van Jacobson.
  • Difficulties in Simulating the Internet by Sally Floyd and Vern Paxson.
• Monday 7/11/2011 (13h30-16h45): cancelled session
• Work to do: Read the following document and make a 1-page checklists for designing/conducting/analyzing your experiments.
• Other suggested readings: How to read an article? and How to write a review?
• Monday 14/11/2011 (13h30-16h45): Arnaud Legrand and Jean-Marc Vincent Visualization of distributed systems and Workload generation
• Documents: visualization slides EP_{04Visualization}.pdf, workload generation slides EP_04Workload.pdf
• References:
  • Lucas Schnorr's PhD thesis: Some Visualization Models applied to the Analysis of Parallel Applications
  • Again, R. Jain's book.
  • Books by Edward Tufte
  • GGen: Random DAG Generation
• Friday 25/11/2011 (13h30-15h00): Arnaud Legrand Introduction to design of experiments.
• Documents: slides EP_05DoE.pdf
• References:
  • Again, R. Jain's book.
  • Design and Analysis of Experiments, by Douglas C. Mongtgomery
  • One-Factor-at-a-Time Versus Designed Experiments by Veronica CZITROM
  • There are three kinds of lies: lies, damned lies, and statistics (Benjamin Disraeli ?). Two articles on the (mis)-usage of statistics in medical science: Lies, Damned Lies, and Medical Science and Do We Really Know What Makes Us Healthy?
• Monday 28/11/2011 (14h00-15h30): Jean-Marc Vincent

Bibliography

• R. Jain, [[http://www.cs.wustl.edu/~jain/books/perfbook.htm][The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling]], Wiley- Interscience, New York, NY, April 1991.
• Jean-Yves Le Boudec. Methods, practice and theory for the performance evaluation of computer and communication systems, 2006. EPFL electronic book.
• David J. Lilja, Measuring Computer Performance: A Practitioner’s Guide, Cambridge University Press 2005
• R. Nelson, Probability stochastic processes and queuing theory: the mathematics of computer performance modeling. Springer Verlag 1995