PUF/JLPC Summer school on Performance Metrics, Modeling and Simulation of Large HPC Systems

Jesus Labarta (Director of the Barcelona Supercomputing Center) and Juan Gonzalez (Junior Researcher at Barcelona Supercomputing Center)

     cd puf_jlpc/bsc
     for i in * ; do echo " - [[./puf_jlpc/bsc/$i][$i]]" ; done

• BSC_Tools_SophiaAntipolis_2014.pdf
• BSC_Tools_SophiaAntipolis_2014.pptx

• Tutorial
• Hand on (demo of Paraver)

Obtaining traces from HPC applications is the first step before analysis, understanding, optimization or simulation. At large scale, issues such as tracing overhead, intrusiveness or trace size become so important that classical tracing mechanisms have to be redesigned.

In a first part, we will present the basics of tracing and what the major issues are. We will present some of the main tracing environments and try to compare them. We will focus on extrae and present some mechanisms that allow to increase scalability. We will also describe some of the analysis capabilities motivating such mechanisms and how to use them. In particular, we will present a couple of recent developments with clustering and processing sampled data ("folding"). We will illustrate the resulting analysis with paraver.

In a second part, a few traces will be provided and attendees will have the opportunity of analyzing them or even to obtain traces from their own code if time allows

We will illustrate and use several tools and traces from the BSC toolchain (e.g., extrae, dimemas, paraver). Such tools can be downloaded from the BSC software page.

Sanjay Kale (Professor in the CS dept., University of Illinois, Urbana Champaign) and Ronak Buch (PhD candidate, University of Illinois, Urbana Champaign)

     cd puf_jlpc/uiuc
     for i in * ; do echo " - [[./puf_jlpc/uiuc/$i][$i]]" ; done

• 1_charmIntroShort.pdf
• 1_charmIntroShort.pptx
• 2_Projections_Overview.pdf
• 2_Projections_Overview.pptx
• 3_Case_Studies_with_Projections.pdf
• 3_Case_Studies_with_Projections.pptx

Unavailable at the moment. I'm working on it…

Performance of parallel applications is notoriously hard to optimize. This gets even more challenging when running a large number of processors, and/or during strong scaling of an application to the limit of its scaling. Fortunately, one can obtain detailed performance data in the form of traces or summaries. However, it requires skill and expertise to use this data to identify the major bottleneck that is holding up performance. This is further compounded by the "whack a mole" nature of performance debugging: when you fix one problem, another problem that was masked by the first one emerges as the next bottleneck.

In this tutorial, we will learn about techniques and methodologies that one can use to hold onto and saw that the major performance problems faced by your applications. You will learn about different views of performance, such as time profiles, processor profiles, communication graphs, outlier analysis, histograms, and the richest of them all: detailed timelines. You will then learn about the rules, heuristics, and idioms (i.e. sequence of analysis/visualizations in pursuit of a conclusion or inference) that experts use in performance tuning.

The tutorial will use the projections performance analysis tool that is part of the Charm++ parallel programming system. It will include several case studies from applications such as NAMD (biophysics), ChaNGa (astronomy), and several mini-applications.

It is required that the attendees download and test "Projections" on their laptops. In addition, several log files will be provided that must be downloaded prior to the workshop. Projections can be downloaded from http://charm.cs.uiuc.edu/software

Martin Quinson (Lorraine University/Inria) and Augustin Degomme (CNRS/University of Grenoble/Inria)

     cd puf_jlpc/sg
     for i in * ; do echo " - [[./puf_jlpc/sg/$i][$i]]" ; done

• SMPI.tar.gz
• simgrid-smpi-101.pdf
• simulation.pdf

• Tutorial
• SimGrid/SMPI Presentation
• Hands on (demo of SMPI)

Modern computing systems have become increasingly complex and large scale. This irreducible complexity of creates a large gap between our understanding of the system and its reality, between the facts and our analysis. Simulation is thus an appealing alternative to study such systems. Indeed, in silico studies have proved their usefulness in most other scientific and engineering disciplines. SimGrid is an open source framework to study the behavior of large-scale distributed systems such as Grids, Clouds, HPC or P2P systems. It can be used to evaluate heuristics, prototype applications or even assess legacy MPI applications. It is developed and optimized since more than 10 years and allows to develop simulators that are several orders of magnitude faster than many ad hoc simulators while relying on fluid (coarse-grain) resource models whose validity has been thoroughly studied. These models allow to account for network topology (e.g., contention, performance heterogeneity, communication locality) and resource capacity fluctuations as well as non trivial network protocol peculiarities (e.g., protocol switches, fairness, reverse traffic impact).

This tutorial will provide attendees with clear perspectives on the challenges for experimental research in the area of parallel and large-scale distributed computing, and on current technology for conducting experiments with real-world testbeds, emulated testbeds, or simulated testbeds. It will particularly emphasize on the capabilities and limitations of simulation.

The first part of the tutorial will present and contrast current experimental methodologies, giving attendees in-depth understanding of the scientific and technological issues at hand.

The second part of the tutorial will focus on simulation, giving a state of the art of current simulation technology and discussing challenges for the development of sound simulation models. The tutorial will use the SimGrid simulation framework as an exemplar since it implements sophisticated and validated simulation models.

The last part of the tutorial will be a hands-on and focus on an in-depth presentation how SMPI can be used to predict the performance of MPI codes.

By the end of this tutorial attendees should have a clear understanding of current technology and best practice for experimental parallel large-scale distributed computing research, and in particular on the use of simulation.

The attendees should be familiar with MPI and HPC systems. If they have an idea of system they would like to study through simulation, we will be happy to brainstorm with them. It is however essential to fully benefit from the hands-on that they bring their own laptop with an already working recent coding environment (C/C++, cmake, whatever IDE they prefer).

The simplest option for attendees is to install the libsimgrid-dev package from the official servers of recent debian or ubuntu distributions. If you have another OS, you should install from the sources (or have a VM with an up-to-date debian).

• The SimGrid project and in particular the Tutorial section.
• Toward Better Simulation of MPI Applications on Ethernet/TCP Networks.
• On the Validity of Flow-level TCP Network Models for Grid and Cloud Simulations.

Arnaud Legrand (CNRS/University of Grenoble/Inria).

   cd /home/alegrand/Work/SimGrid/infra-songs/WP8/140613-Nice-PUF_JLPC-RR/
   cp PUF_JLPC_intro.pdf /home/alegrand/org/public_html/research/events/puf_jlpc
   cp RR_orgmode.pdf /home/alegrand/org/public_html/research/events/puf_jlpc/rr/

• Slides

• Tutorial
• Hands-on (setting up a laboratory notebook with emacs/org-mode)

Reproducibility of experiments and analysis by others is one of the pillars of modern science. Yet, the description of experimental protocols (particularly in computer science articles) is often lacunar and rarely allows to reproduce a study. Such inaccuracies may not have been too problematic 20 years ago when hardware and operating systems were not too complex. However nowadays are made of a huge number of heterogeneous components and rely on an software stack (OS, compiler, libraries, virtual machines, …) that are so complex that they cannot be perfectly controlled anymore. As a consequence some observations have become more and more difficult to reproduce and to explain by other researchers and even sometimes by the original researchers themselves. Although computer systems are theoretically deterministic systems, the state explosion problem and the inability to perfectly observe and control their state leave no other option than considering them as rather stochastic systems.

Yet, the analysis of stochastic systems is more complex and the graphs provided in articles are generally insufficient to draw solid conclusions. Although simple graphs may illustrate the point of view of the authors, they rarely convey information about the variability of the system, which is yet critical to evaluate how much confidence can be put in the analysis.

In the last decade there has been an increasing number of article withdrawal even in prestigious journals and the realization by both the scientific community and the general public that many research results and studies were actually flawed and wrong.

Open science is the umbrella term of the movement to make scientific research, data and dissemination accessible to all levels of an inquiring society. In particular, it encompasses practices such as the use of open laboratory notebooks and reproducible research, which refers to the idea that the ultimate product of academic research is the paper along with the full computational environment used to produce the results in the paper such as the code, data, etc. that can be used to reproduce the results and create new work based on the research.

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

– Jonathan Buckheit and David Donoho

The aim of this tutorial is to sensibilize the audience to the experiment and analysis reproducibility issue in particular in computer science. I will present tools that help answering the analysis problem and may also reveal useful for managing the experimental process through notebooks.

More precisely, I will introduce the audience to the following tools:

• R and ggplot2 that provide a standard, efficient and flexible data management and graph generation mechanism. Although R is quite cumbersome at first for computer scientists, it quickly reveals an incredible asset compared to spreadsheets, gnuplot or graphical libraries like matplotlib or tikz.
• knitR is a tool that enables to integrate R commands within a LaTeX or a Markdown document. It allows to fully automatize data post-processing/analysis and figure generation down to their integration to a report. Beyond the gain in term of ease of generation, page layout, uniformity insurance, such integration allows anyone to easily check what has been done during the analysis and possibly to improve graphs or analysis.
• I will explain how to use these tools with Rstudio, which is a multi-platform and easy-to-use IDE for R. For example, using R+Markdown (Rmd files) in Rstudio, it is extremely easy to export the output result to Rpubs and hence make the result of your research available to others in no more than two clicks.
• I will also mention other alternatives such as org-mode and babel or the ipython notebook that allow a day-to-day practice of reproducible research in a somehow more fluent way than knitR but is mainly a matter of taste.

Depending on the question of the audience, I can also help the attendees analyzing some of their data and introduce them to the basics of data analysis.

To make sure that everyone uses its time in the most possible efficient way, it is required that the attendees have installed before hand R, Rstudio et ggplot2 on their laptop.

Here is how to proceed on a recent debian distribution

     sudo apt-get install r-base r-cran-ggplot2 r-cran-reshape

Rstudio and knitr are unfortunately not packaged within debian so the easiest is to download the corresponding debian package on the Rstudio webpage and then to install it manually.

     wget http://download1.rstudio.org/rstudio-0.98.490-amd64.deb
     sudo dpkg -i rstudio-0.98.490-amd64.deb
     sudo apt-get -f install # to fix possibly missing dependencies

You will also need to install knitr. To this end, you should simply run R (or Rstudio) and use the following command (you'll have to answer yes to the first question about installing in your home and then you'll have to select a mirror).

     install.packages("knitr")

If r-cran-ggplot2 or r-cran-reshape could not be installed for some reason, you can also install them through R by doing:

     install.packages("ggplot2")
     install.packages("reshape")

Those of you who whould like to play with org-mode should install emacs24, org-mode (>=8.2!!!) and ess:

     sudo apt-get install emacs24 org-mode ess

You may want to use my emacs configuration.

• Although this is not specifically about reproducible research, every PhD student should have read this standard textbook: Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, Wiley- Interscience, New York, NY, April 1991. A must read!
• R for Beginners
• If you don't like reading books, you can also follow an online course on R and data analysis like: https://www.coursera.org/course/compdata. There is also an upcoming one on reproducible research with R: https://www.coursera.org/course/repdata
• The performance evaluation lecture that I give with Jean-Marc Vincent to master and graduate students. It is intended for a computer scientist audience and we present the basics of data analysis (test, confidence interval, linear regression) and design of experiments (randomization, replication, factorial plan)
• With a little experience, the incredible documentation of ggplot2 will allow you to build the graphical representation you need.
• http://www.r-bloggers.com/ is a fun way to access a bunch of enlightening graphical representations and analysis from a various domains.

• Workshop on Duplicating, Deconstructing and Debunking (WDDD) (2014 edition)
• Reproducible Research: Tools and Strategies for Scientific Computing
• Working towards Sustainable Software for Science: Practice and Experiences
• REPPAR'14: 1st International Workshop on Reproducibility in Parallel Computing
• Reproducibility@XSEDE: An XSEDE14 Workshop
• Reproduce/HPCA 2014
• TRUST 2014