TotalView: The Leader in MPI and OpenMP Debugging

TotalView Debugger is the de facto “gold standard” for MPI and OpenMP debugging. Considered an indispensible tool in the most advanced and prestigious research, commercial, and academic institutions in the world, TotalView has remained the proven leader in parallelism during more than 19 years of continuous development and refinement.

Isolating interprocess communication problems can be like trying to find your way through a maze – in the dark. TotalView has many built-in features that illuminate and map your path. Features like our visual process grid and Message Queue Graph give you immediate feedback on the state of any process in your application. And TotalView makes debugging remote processes just as easy as debugging local processes.

Automatic Process Acquisition

TotalView makes remote debugging as easy as local. Your complex MPI code can have hundreds of MPI processes running in parallel and perhaps distributed across several machines. TotalView automatically detects that a new MPI process has started and automatically attaches to it. MPI provides a mechanism for us to know which processes on what hosts are part of a parallel job, and TotalView makes it a simple operation to attach to them all. And advanced users can debug an arbitrary subset of the MPI processes.

Custom Process Groups

TotalView lets you easily create custom process groups. Then, when you select that custom group to control, all TotalView commands will act on that group. The Processes tab provides visual feedback, making it easy to keep track of which processes you are controlling at any given time.

Message Queue Graph

Our Message Queue Graph gives you a visual representation of the message queue information, making it easy to see where interprocess communication has gone wrong. Simply choose Tools > Message Queue Graph from the Process Window, select the types of messages you would like to see, and click Update. TotalView will show you an easy-to-understand graph. A click on any process provides more detail.

Message Queue Window

TotalView provides visibility and insight into how your MPI processes are communicating with one another. Since MPI process communication problems are both common and often hard to detect, the TotalView Message State Window is an essential feature for your MPI debugging. This window makes it easy to see when processes aren't communicating as you intended, and to get the details you need to fix the problem.

Barrier Points

TotalView lets you control your MPI processes any way you like. Special breakpoints called barrier points put you in control of synchronizing groups of processes at strategic points in your debugging session. Using barrier points, you won’t lose control of all your processes – you’re in the driver’s seat.

Comprehensive Support

TotalView supports many flavors of MPI, such as MPICH, LAM-MPI, and vendor-specific version. Yours is probably on our list! To find out, consult our platforms document.

OpenMP Debugging

OpenMP provides a powerful and easy way to introduce threads-based parallelism into your program. TotalView brings you the same advantages in debugging OpenMP programs as it does in MPI. You can control individual OpenMP threads as you would individual MPI processes. You can set breakpoints and barriers, which can stop a single thread or the entire OpenMP program.

Using TotalView’s “laminate” command, you can view common variables across all threads in your process, just as you can across all the processes of an MPI group.

Mixed MPI and OpenMP Support

Some developers are using both OpenMP with MPI in the same application, a technique called "hybrid programming." TotalView handles hybrid programs as easily as it handles non-hybrid MPI or OpenMP programs. Chapter 6 of the TotalView Users Guide has more information on debugging OpenMP programs.