- What is the Visualization Tool?
- What Does VT Look Like?
- Visualization Tool Views
- Using VT for Trace Visualization
- Using VT for Performance Monitoring
- Adjusting a View
- Creating and Using Your Own VT Configuration File
- Index of VT Views
- Acknowledgements and References
- Exercises
What is the Visualization Tool?
- The Visualization Tool (VT) is a graphical user interface which
enables you to perform both:
- Trace Visualization - post execution analysis of your application's behavior within IBM's Parallel Environment
- Performance Monitoring - real time behavior of the SP system as your program (and other) programs execute
- VT's primary purposes are to help you analyze, understand, debug and
tune a parallel SP application by graphically demonstrating execution
behaviors such as:
- Communication patterns and characteristics
- Processor load balance
- Computation versus communication ratios
- Timing characteristics
- Processor idle time
- Lengthy message passing operations
- Differences between different algorithms
- VT is comprised of numerous graphical displays or "views" which present information about your application and the system. You, as the user, decide which views to display. For example:
- Views are displayed in common and easy to understood formats, such as bar charts, pie charts, strip graphs and grids. Most information can be displayed in more than one format, allowing you to select the most appropriate display.
- Many views allow you to see the information in more than one way:
- Instantaneous (snapshot)
- Average over time
- Cumulative over time
- Streaming - information over a range of time.
- Aggregate over all processors
- Individual processors
- Most views will allow you to "point and click" at any location in the view to obtain that location's instantaneous, average and cumulative values.
- Most views allow you to set your own preferences for parameters such as colors, time resolution, display rate, chart gradiations, etc.
- You can save your VT preferences in a configuration file. This allows you to quickly return to your favorite "setup" on subsequent VT sessions.
What Does VT Look Like?
VT "looks like" an X Windows Graphical User Interface (GUI). It opens with two initial windows which then allow you to select and control a number of different "views".- Trace Visualization Window
This is the "main" VT window. Its primary purpose is for monitoring and controlling the playback of trace files.
Several sets of analog (VCR-like) controls and two pulldown menus are used to:
- Play, step, loop, reset or stop tracefile viewing
- Set time resolution for playback of "streaming" type views
- Adjust tracefile playback speed
- Display trace clock and the name of the tracefile being viewed
- Select tracefiles
- Select the Performance Monitor
- Bring back the View Selector window if it is currently closed
- Save and load custom VT configuration files
- Get context sensitive Help from IBM's InfoExplorer database
- Print/save PostScript output of VT displays
- VT View Selector Window
Used to select the desired views for both trace visualization and performance monitoring.
Both an individual view or, an entire view group can be selected by a single mouse click on the desired icon. An entire view group can be "deselected" by a single click also.
If performance monitor is selected (discussed later), a second View Selector window will appear. Use it instead of the initial one.
- Visualization Tool Views
- over 30 different views which are organized and selected from the View Selector Window. Organized into five main categories:- Computation: utilization of processor nodes running a particular program. Can be used for both trace visualization and performance monitoring.
- Communication / Program: information regarding message-passing events between processor nodes while running a particular program. Also shows source code of a program whose trace records are being played back. Can only be used for trace visualization.
- System: system activity and event information such as page faults and context switches. Can be used for both trace visualization and performance monitoring.
- Network: number of TCP/IP packets sent and received. Can be used for both trace visualization and performance monitoring.
- Disk: disk activity such as disk reads, writes and transfers. Can be used for both trace visualization and performance monitoring.
- Any number of views can be selected simply by clicking on the associated icon. An entire view group can be selected by clicking on the group name icon.
- To "deselect" a view or view group, click on that view or view group's associated icon.
- Use the left mouse button in any open view to determine information about the values at that point. A pop-up window will appear to display information such as clock time, processor number, instantaneous, average and cumulative values.
- Use the right mouse button to display and/or change view configuration parameters, such as display rate and color.
Using VT for Trace Visualization
About Trace Records and Files
- Trace visualization enables you to play back statistical and event records (trace records) generated during a program's execution.
- Trace records are stored in a trace file. By playing a trace file and opening views to visualize its trace records, you can perform algorithm characterization and study your program's performance.
- There are four types of trace records used by VT:
- MPL/MPI Message Passing - generated by message-passing events such as blocking sends and receives among your program tasks. Each of these events is the result of a call to a message passing subroutine.
- MPL/MPI Collective Communication -generated by communication events involving all tasks of your program such as broadcasts, scatters and gathers. Each of these events is the result of a call to a message passing subroutine.
- AIX Kernel Statistics - contain a sampling of statistics from the
kernel, including:
- Percent of CPU utilization (user, kernel, wait, and idle)
- Number of system calls
- Number of page faults
- Number of transfers to and from disk
- Number of blocks read from disk
- Number of blocks written to disk
- Number of TCP/IP packets received
- Number of TCP/IP packets sent
- Application Markers - generated as a result of using the function mpc_marker ( C ) or MP_MARKER (FORTRAN) in your program for the Program Marker Array.
- VT uses its own routines to create trace records - it does not utilize the AIX trace facility.
Using VT for Trace Visualization
How to Generate Trace Files
- Step 1: Control Trace Record Generation Within the Program. Use the
following routines within your application program to turn on and turn
off trace record generation anywhere within your program. For example:
For FORTRAN programs
INTEGER FLAG, RC C Turn tracing on somewhere near the beginning of program FLAG = 3 CALL VT_TRC_START(FLAG,RC) ..... program execution ...... C Turn tracing off somewhere near the end of program CALL VT_TRC_STOP(RC)For C programs#include <VT_trc.h> int flag, rc; /* Turn tracing on somewhere near the beginning of program */ flag = 3; rc = VT_trc_start_c(flag); ..... program execution ...... /* Turn tracing off somewhere near the end of program */ rc = VT_trc_stop_c();The return code for successful subroutine completion should be zero. The flag argument to the VT_TRC_START / VT_trc_start_c routine is an integer that specifies one, some, or all trace record types. The following table shows the possible values of this flag and the trace record type(s) indicated by each.AIX Message Collective Kernel Application Flag Passing Communication Statistic Markers ---- ------- ------------- ---------- ------------ 0 1 X 2 X X 3 X X X 9 X X X X - Step 2: Compile the Program
Use either mpcc or mpxlf to compile your program. Note: If you plan on using the "Source Code" view, you will need the -g option.
Examples:
mpcc myprogram.c -o myprogram or mpcc -g myprogram.c -o myprogram mpxlf myprogram.f -o myprogram or mpxlf -g myprogram.f -o myprogram - Step 3: Set the Necessary VT Environment Variables
Minimally, you must set the MP_TRACELEVEL environment variable. By default, tracing is turned off within POE. To generate a trace file, you need to run your program with tracing turned on.
Examples:
C Shell: setenv MP_TRACELEVEL 3 Ksh Shell: export MP_TRACELEVEL=9The integer you use with the MP_TRACELEVEL environment variable and its associated flags are the same ones you use on the VT_trc_start statement within your programs - if they do not match, the lesser value is used.
Other VT environment variables may also be set to specify temporary file sizes, name, etc. These are discussed later.
- Step 4: Run the Program to Generate a Trace File
As your program executes on each SP node, trace records are created for execution events on that node. The trace level setting determines which events cause trace record generation:
- Message passing communication events - written by instrumentation within the communications library
- AIX kernel statistics - written by a spawned process to a separate file
Three-tiered memory buffering approach is used for storing trace files
- Node memory - on each processor during execution
- Node temp file space - on each processor during execution
- Permanent file space - final merged file in user file space (usually)
Trace files from all nodes are merged after execution into a single serialized collection on your home (login) node. This is the file which VT can use to display your application's execution events.
By default, VT will name your trace file the same as the program name with the suffix .trc added.
Optional: When running your program, you can specify a command-line flag (such as -tracelevel) to override its associated environment variable.
Using VT for Trace Visualization
Additional Options
- Specifying a Trace File Name
To specify a trace file name other than the default, you can set the environment variables MP_TRACEDIR and MP_TRACEFILE or use the -tracedir/tdir and -tracefile/-tfile flags when invoking the program. As with all PE command-line flags, they temporarily override their associated environment variable. Note that the default value of MP_TRACEDIR is the current directory and that you are required to set this variable if using a different directory.
Examples:
setenv MP_TRACEDIR /u/user3/test1 setenv MP_TRACEFILE tfile -or- myprogram -tracedir /u/user3/test1 -tracefile tfile - Writing the Temporary Trace File to a Specified Directory
By default, VT writes the memory buffer to a file /tmp/username. You can override this by setting the MP_TMPDIR environment variable or using the -tmpdir flag when invoking the program. Note that the -tmpdir flag temporarily overrides its associated environment variable.
Examples:
setenv MP_TMPDIR /tmp1 -or- myprogram -tmpdir /tmp1 - Changing the Sampling Interval for AIX Kernel Statistics
If you are generating trace records for AIX Kernel Statistics, VT gets a sampling of those statistics at set intervals. By default, the set interval is every ten milliseconds.
You can change the sampling interval by setting the environment variable MP_SAMPLEFREQ, or using the -samplefreq or -sfreq flag when invoking the program. Note that the -samplefreq and -sfreq temporarily override their associated environment variable.
Examples:
setenv MP_SAMPLEFREQ 50 -or- myprogram -sfreq 50More frequent sampling is more detailed, but requires more storage. If storage is a consideration, set the sampling frequency to a higher value. - Managing Storage for Trace Files
By default, the system uses the following three-tiered approach to manage the storage of trace files:
- First, trace records are written to a 1 MB buffer in memory.
- When filled, the system flushes the buffers contents to a temp file with 10 MB of memory.
- When the temp file would exceed 10 MB, the system flushes its contents to a permanent trace file whose default maximum size is 10 MB.
You have several options regarding the storage of trace files. You can specify:
- the maximum size of the buffer by setting the MP_TBUFFSIZE environment variable or using the -tbuffsize or -tbsize command-line flag;
- the maximum size of the temp file by setting the MP_TTEMPSIZE environment variable or using the -ttempsize or -ttsize command-line flags;
- the maximum size of the permanent trace file by setting the
MP_TPERMSIZE environment variable or using the -tpermsize or -tpsize
command-line flags;
Examples:
setenv MP_TBUFFSIZE 5M setenv MP_TTEMPSIZE 20M setenv MP_TPERMSIZE 300M -or- myprogram -tbuffsize 20M -ttempsize 50M -tpermsize 300M -
a wraparound storage approach instead of the default three-tiered
approach, by setting the MP_TBUFFWRAP environment variable or using
the -tbuffwrap or -tbwrap command-line flag. With this approach, the
system overwrites the buffer instead of flushing it to a temp file.
Examples:
setenv MP_TBUFFWRAP yes -or- myprogram -tbuffwrap yes
- VT_trc_set_params_c / VT_trc_set_params
You may use this VT tracing routine (either C or Fortran version) within your application code, instead of environment variables and command-line flags, to accomplish all of the options just discussed. Please see the "IBM AIX Parallel Environment Operation and Use" manual for more information.
Using VT for Trace Visualization
Playing Trace Files
The Trace Visualization window is used to
monitor and control the playback of trace files.
- Selecting the Trace File
You can start VT with the name of your trace file to have it automatically load that file for playback. For example:
vt -tfile mytrace.fileAlternately, if you start VT without specifying a tracefile, you can use the Trace Visualization window's "File" pull-down menu to select a trace file.
- Basic Trace Visualization Controls. Allow you to control the playback
of your trace files. Similar to VCR or CD player controls.
From left to right, these are:
- Reset Control Button -used to return playback to the start (or an earlier point) in the trace file.
- Step Back Control Button - allows you to step back a single trace record.
- Stop Control Button - allows you to stop playback.
- Play Control Button -allows you to start or resume playback.
- Step Forward Control Button - allows you to step forward a single trace record.
- Loop Control Button - allows you to continuously play a trace file or a portion of a trace file.
- Application Time Display. Shows the time recorded in the trace file
during the application's execution. Displayed as hours, minutes, seconds,
and nanoseconds.
Keep in mind that the time is the original processing time, not the current trace file's playback time.
- Trace File Field. Displays the full path name of the selected trace file. You can also select a file for playback by entering it in this field.
- The Trace File Time Control. This slide bar shows your current playback
position within the trace file. You can choose the scale units according
to time or number of events.
As you play the trace file, a Playback Position Line moves from left to right to show your current playback position.
You can change the current playback position by dragging the small triangle at the base of the Playback Position Line to a new position within the Trace File Time Control.
Two range markers allow you to set the range of playback within your trace file.
- Replay Speed Control. Permits you to change the speed of playback.
- Magnification Control. Allows you to increase or decrease the amount of time represented in certain views. By varying the magnification of time, you present more or less detail in these views.
Using VT for Performance Monitoring
- Performance monitoring enables you to monitor the operational status and activity of your SP processor nodes.
- Performance monitoring does not require a trace file - it is done online.
- Each processor node has a Statistics Collector daemon that, upon request, supplies VT with samplings of AIX kernel statistics at a configurable interval. It is these statistics that you visualize during performance monitoring.
- Starting the Performance Monitor
Select "Performance Monitor" from the "File" pull-down menu of the Trace Visualization window. This will cause the Performance Monitor window to appear.
A second window will also appear, called the VT Performance Monitor View Selector. This window is very similar to the VT View Selector window in appearance and function. You may want to close the original View Selector window to avoid confusion.
- See Visualization Tool Views to review the description on VT views.
Using VT for Performance Monitoring
Description of the Performance Monitor Window
The VT Performance Monitor window consists of:
- Job List: This lists all the jobs currently running on the SP by their process identifier using data provided by the Resource Manager. If there are jobs running which do not communicate with the Resource Manager, VT cannot track these jobs and they will not be listed here.
- Node Matrix: This takes up the main part of the window. Each square on
the matrix represents one of the processor nodes. The square's appearance
indicates its status.
- gray box - unavailable -you cannot select it for monitoring.
- pink box - available -you can select it for monitoring.
- green square in a pink box - active, job running.
- yellow box in a pink box - selected for monitoring.
- yellow box in a green square in a pink box -active job on selected node.
Note: the meaning of these colors/symbols can be displayed at any time by selecting "Legends" from the Performance Monitor's "File" pull-down menu.
- Node List: This lists all the processor nodes. VT gets this information from the SP Resource Manager
- Selected Nodes Area: This area lists the processor nodes selected for performance monitoring.
- Frequency Control: This control displays, and lets you reset the interval between which VT advances the views with new AIX kernel statistics from the Statistics Collector daemons. Interval is measured in seconds.
Using VT for Performance Monitoring
Getting Started
After starting the VT Performance Monitor, you
can do the following to begin monitoring.
- Select Processor Nodes for Monitoring
There are three ways to select processor nodes for monitoring.
- From the Node Matrix - click on the square representing the processor node in the Node Matrix. VT colors this node appropriately, adds the node's identifier to the Selected Nodes Area, and highlights the node in the node list.
- From the Node List - click on the processor node's entry in the Node List. VT highlights the the processor node's entry in the node list, adds that node's identifier to the Selected Nodes Area, and colors the node appropriately.
- From the Job List - click on the job's process identifier in the Job list. VT highlights the job's process identifier, colors all the node squares associated with the job, and lists the nodes in the Selected Nodes Area.
- Start Monitoring
First, select one or more views from the VT Performance Monitor View Selector window. This is required before monitoring can begin.
Then, from the "File" pull-down menu of the Performance Monitor, select "Toggle Monitoring". The Statistics Collector daemon(s) on the selected processor node(s) will begin sending samplings of AIX kernel statistics to VT. All open views will begin visualizing these statistics.
If communication between VT and a selected node cannot be established by the network, it may take a few minutes for the network to respond with an error.
- Selecting Additional Nodes and Views
If you select additional processor nodes or additional views after you have already started monitoring, you must stop and restart monitoring. Use "Toggle Monitoring" from the "File" pull-down menu.
- Changing the Kernel Statistics Sample Interval
The default sampling frequency interval, expressed in seconds, appears in the "Frequency" box at the bottom of the Performance Monitor. You can change this by simply entering a new value into the small box.
For the new value to have effect, you must stop and restart monitoring. Use "Toggle Monitoring" from the "File" pull-down menu.
- Stop Monitoring
Select "Close Performance Monitor" from the Performance Monitor's "File" pull-down menu. The Performance Monitor View Selector window can be closed by selecting "Done" from its "Actions" pull-down menu.
Adjusting a View
Most of VT's viewspermit you to adjust certain parameters such as display rate and color. Many views permit you to adjust other parameters as well.- Determining What Parameters Can Be Adjusted
Place the mouse cursor in any open view and press the right mouse button. A pop-up menu will appear showing you which parameters and configuration settings you may change.
- Adjusting the View Display Rate
When doing trace visualization or performance monitoring, views redisplay using new information from VT after set intervals. By default, this interval is one second.
To change this rate, select the "Config" option from the pop-up menu. The view's Time Resolution window will open. Enter the new display rate and press "OK".
The view will now display itself at the new display rate.
- Adjusting the View Colors
To change a view's colors, select the "Parameters" option from the pop-up menu. The view's Key Spectrum window will open.
Click on the spectrum to see what other spectrums are available. When the desired spectrum is displayed, press "OK" or "APPLY".
The view will now display itself with the new colors.
Note that the colors used in the VT views are requested from the default X-Windows color map. The appearance of your views could be changed if the default color map is changed. To avoid this, instruct VT to create a private copy of the color map using the -cmap flag on the vt command when starting VT.
- Other View Parameters
These will differ from view to view and are not discussed here.
Creating and Using Your Own VT Configuration File
- When you have a screen configuration you like, you can save it in a configuration file. Later you can load that configuration file to display the same screen configuration.
- A VT configuration file controls:
- which VT windows that are displayed
- window characteristics, such as size and location on the screen
- the configuration parameters for each view, such as display rate and color.
- To save the desired configuration, select "Save Configuration" from
the "File" pull-down menu on the Trace
Visualization Window.
A Save Configuration Dialog windowwill then appear. Enter the name you want to give the configuration file. VT will save the file as specified.
- To load a previously saved configuration, select "Load Configuration"
from the "File" pull-down menu on the Trace
Visualization Window.
A Configuration File Selection Dialog window will appear. Locate and then select the name of the desired configuration file. VT will load the specified file.
You can also load a configuration file when starting a VT session by using the -cfile or -configfile flags with the vt command.
Index of VT Views
The following index is arranged according to VT view groups. A graphical example of the group views can be obtained by clicking on the group name. Text descriptions of the individual views can be obtained by clicking on the view name.- Communication / Program
-
Connectivity Graph
- Interprocessor Communication
- Message Status Matrix
- Source Code (includes example)
- Interprocessor Communication
- Computation
-
Processor Utilization (Bar Chart)
- Processor Utilization (Graph)
- Processor Utilization (3D)
- User Load Balance
- Processor Utilization (Graph)
- Disk
-
Disk Reads (Bar Chart)
- Disk Reads (Graph)
- Disk Transfers (Bar Chart)
- Disk Transfers (Graph)
- Disk Writes (Bar Chart)
- Disk Writes (Graph)
- Disk Reads (Graph)
- Network
-
Packets Received (Bar Chart)
- Packets Recieved (Graph)
- Packets Sent (Bar Chart)
- Packets Sent (Graph)
- Packets Recieved (Graph)
- System
-
Context Switches (Bar Chart)
- Context Switches (Graph)
- Page Faults (Bar Chart)
- Page Faults (Graph)
- System Calls (Bar Chart)
- System Calls (Graph)
- System Summary
- Context Switches (Graph)