M2R Performance Evaluation

The coordinator for these lectures is Jean-Marc Vincent . The lecturers are Jean-Marc Vincent and Arnaud Legrand .

The next lecture will be on Friday 21/10 in room H102 from 13h15 to 16h30.

The aim of this course is to provide the fundamental basis for performance evaluation of computer systems. Two approaches are developed:

• performance measurement: based on experimental platforms (benchmarks or owner instrumented code execution), how to analyze data and synthesize performance indexes
• performance modeling: from a description of resources and the behavior of applications, how to predict the performance of the application

• Friday 14/10/2011 (13h30-16h45): Jean-Marc Vincent Introduction, performance indexes, data analysis, modeling and inference.
  • Documents: slides EP_01_Introduction.pdf
  • References:
    • Preliminary guidelines for empirical research in software engineering Kitchenham, B.A.; Pfleeger, S.L.; Pickard, L.M.; Jones, P.W.; Hoaglin, D.C.; El Emam, K.; Rosenberg, J.; Software Engineering, IEEE Transactions on , Volume: 28 Issue: 8, Aug 2002 Page(s): 721 -734
    • Partie I : R. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling Wiley- Interscience, New York, NY, April 1991.
• Friday 21/10/2011 (13h30-16h45): Arnaud Legrand Measurement on computer systems (benchmarking, observation, tracing, monitoring, profiling).
  • Documents: slides EP_02_Measurements.pdf
  • References:
    • David J. Lilja. Measuring Computer Performance: A Practitioner’s Guide Hardcover. Cambridge University Press, 2000.
    • R. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling Wiley- Interscience, New York, NY, April 1991.
• Friday 4/11/2011 (13h30-16h45): Arnaud Legrand Introduction to discrete event simulation and emulation of computer systems. The simulation crux: validation. Introduction to network topology generation.
  • Documents: slides EP_03_Simulation1.pdf, EP_03_Simulation2.pdf, EP_03_Simulation3.pdf
  • References:
    • Banks J., J. S. Carson and B. L. Nelson 1996. Discrete-Event System Simulation, second edition. Prentice-Hall, Englewood Cliffs, NJ. GORDON G. 1996.
    • R. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling Wiley-Interscience, New York, NY, April 1991.
    • On Traffic Phase Effects in Packet-Switched Gateways by Sally Floyd and Van Jacobson.
    • Difficulties in Simulating the Internet by Sally Floyd and Vern Paxson.
• Monday 7/11/2011 (13h30-16h45): cancelled session
  • Work to do: Read the following document and make a 1-page checklists for designing/conducting/analyzing your experiments.
  • Other suggested readings: How to read an article? and How to write a review?
• Monday 14/11/2011 (13h30-16h45): Arnaud Legrand and Jean-Marc Vincent Visualization of distributed systems and Workload generation
  • Documents: visualization slides EP_04_Visualization.pdf, workload generation slides EP_04_Workload.pdf
  • References:
    • Lucas Schnorr's PhD thesis: Some Visualization Models applied to the Analysis of Parallel Applications
    • Again, R. Jain's book.
    • Books by Edward Tufte
    • GGen: Random DAG Generation
• Friday 25/11/2011 (13h30-15h00): Arnaud Legrand Introduction to design of experiments.
  • Documents: slides EP_05_DoE.pdf
  • References:
    • Again, R. Jain's book.
    • Design and Analysis of Experiments, by Douglas C. Mongtgomery
    • One-Factor-at-a-Time Versus Designed Experiments by Veronica CZITROM
    • There are three kinds of lies: lies, damned lies, and statistics (Benjamin Disraeli ?). Two articles on the (mis)-usage of statistics in medical science: Lies, Damned Lies, and Medical Science and Do We Really Know What Makes Us Healthy?
• Monday 28/11/2011 (14h00-15h30): Jean-Marc Vincent

• R. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, Wiley- Interscience, New York, NY, April 1991.
• Jean-Yves Le Boudec. Methods, practice and theory for the performance evaluation of computer and communication systems, 2006. EPFL electronic book.
• David J. Lilja, Measuring Computer Performance: A Practitioner’s Guide, Cambridge University Press 2005
• R. Nelson, Probability stochastic processes and queuing theory: the mathematics of computer performance modeling. Springer Verlag 1995