SP Parallel Programming Workshop
Parallel Operating Environment (POE)

© Copyright Statement

Table of Contents

Prerequisites

What is the Parallel Operating Environment?

POE Definitions

Executing Parallel Programs Using the POE

Setting your Path
Creating a .rhosts File
Compiling and Linking a Parallel Program
Setting Environment Variables
Some Preset POE Environments
Creating a Host List File
Invoking the Executable

CPU and Communications Adapter Usage

Miscellaneous Environment Variables

Parallel File Copy Utilities

Recommendations for Running on the MHPCC SP2

References, Acknowledgements, WWW Resources

Exercises

Prerequisites

This tutorial assumes that the reader is already familiar with the concepts covered in the following tutorials:
- Parallel Programming Introduction
- IBM SP Hardware / Software Overview

What is the Parallel Operating Environment?

IBM's environment for developing and running distributed memory, parallel Fortran, C or C++ programs. Runs on the IBM RS/6000 platform using AIX.
Can also be used to run serial jobs and shell commands across a partition of processors.
The POE consists of a set of software components for developing, executing, debugging, profiling, and tuning parallel programs.
- Parallel Compiler Scripts
- Environment Variables
- Visualization Tool
- Program Marker Array
- System Status Array
- Parallel Debugging Facility
- Parallel Profiling Capability
- Parallel File Copying Utilities
The operation of POE is influenced by a number of POE environment variables.

POE Definitions

Partition

The group of processor nodes on which you run your program is called your partition. There may be multiple active partitions for multiple users across the SP system.

Partition Manager

The Partition Manager establishes and controls your partition. It consists of a set of subroutines that are linked into your program and an Internet daemon process called pmd2. The Partition Manager is responsible for:

Requesting the nodes for your parallel job
Acquiring the nodes needed for your parallel job (if the Resource Manager is not used)
Copying the executables from the initiating node to each node in your partition
Connecting stdin and stdout on your "home node" to all other nodes in your partition.

Resource Manager

The Resource Manager keeps track of all nodes and all POE jobs on an SP system. When your job requests nodes - via your Partition Manager process, the Resource Manager will allocate nodes for your use. It attempts to enforce a "one parallel task per node" rule. Generally, there is only one Resource Manager process for an entire SP system. The Resource Manager may also be referred to as "JM" or Job Manager.

The jm_status command allows you to query the Resource Manager for information about SP pools, nodes and POE jobs currently running on the system:

     
    jm_status -Pv      - verbose list of all nodes and POE jobs
    jm_status -j       - list of all POE jobs

Specific Node Allocation

Refers to the user explicitly choosing the nodes on which the job will run. This is most typically done by placing the node names in a host list file.

Non-Specific Node Allocation

Refers to the Resource Manager automatically selecting the nodes which will be used to run a user job.

Home Node

The SP node where you start your POE job. Generally, this is the SP node where you are currently logged in.

Remote Node

Any other non-home node in your partition.

RM Agent

A pmd2 process (called the Partition Manager daemon) is automatically started for each remote task in the partition. It receives the user's environment, validates the user, and executes the program requested for the particular task. The daemon terminates when the partition is released.

Processor Pool

The local systems administrator may divide the processor nodes into disjoint pools of processors for management purposes. If this is the case, you may then request that your parallel tasks run on specific pools. You may determine your system's pools and associated nodes by using the command:

    
    jm_status -P

Communication SubSystem (CSS)

The Communication SubSystem (CSS) is the set of library routines which implement one of the two protocols for message passing communications between SP nodes. The two different protocols are User Space Protocol and Internet Protocol.

User Space Protocol

The fastest method of communication between nodes. Can be used only with the high performance switch. Often referred to simply as US protocol.

Internet Protocol

A slower method of communication between nodes. Can be used with ethernet or the high performance switch. Often referred to simply as IP protocol.

Executing Parallel Programs Using POE

In order to execute a parallel program, you need to:

Set your path to include the necessary POE executables.
Create a .rhosts file
Compile and link the program using one of the POE compile scripts.
Set up your execution environment by setting the necessary POE environment variables.
Create a host list file (optional)
Invoke the executable

Setting Your Path

The POE executables are usually located in the directory /usr/lpp/poe/bin(versions before 2.1) or /usr/lpp/ppe.poe/bin (version 2.1). There may be symbolic links pointing to them from /usr/bin or some other location. This may vary from system to system.
To determine if the POE executables are in your path, use a command such as which poe or whence poe. If the poe executable can not be found, then you will need to find where they are installed and include that directory in your path.
If the POE executables are found with the which or whence command, you need to do nothing to your path.

Creating a .rhosts File

The Unix .rhosts file is required to run parallel programs with POE. The format of the .rhosts file is:
```
    hostname1  userid
    hostname2  userid
    hostname3  userid
    hostname4  userid
      ...
```
where "hostname" is the actual name of an SP node and "userid" is your actual login userid on that node.
You must create the .rhosts file in your home directory
It is recommended that you include the full domain name of all SP node names in your .rhosts file.
View a sample MHPCC .rhosts file here. This file may also be copied from $WORKSHOP/samples/rhosts.

Compiling and Linking a Parallel Program

POE includes three parallel compile scripts. These scripts cause dynamic linking of the communication libraries at run time. The compile scripts actually call the native IBM Fortran, C or C++ compiler (xlf, cc, CC) to perform the compilation.
- mpxlf - Fortran
```
    mpxlf   options  program.f 
    
```
- mpcc - C language
```
    mpcc  options  program.c
    
```
- mpCC - C++
```
    mpCC  options  program.C
    
```
The communication subsystem library (IP or US) which will be linked is determined dynamically by the MP_EUILIB environment variable at run time.
To statically link either the IP or US library, follow the instructions here.
Options include all valid options available with the native compiler (xlf, cc, CC). There are numerous compile options available with the IBM Fortran, C and C++ compilers, many of which can dramatically improve performance. Users are advised to consult the IBM documentation for details.
Several commonly used options are listed below and described in more detail in the mpxlf, mpcc or mpCC man pages.
- -c compiles only, producing a ".o" file. Does not link the object files.
- -g compiles the program suitable for debugging with pdbx or xpdbx (see corresponding man pages or use the info -l pe command to view IBM online documentation for these debuggers). This option is also necessary to use the Source Code view in the Visualization Tool.
- -I (upper case i) names directories for additional include files.
- -l (lower case L) names additional libraries to be searched.
- -L specifies pathname where additional libraries reside Directories will be searched in the order of their occurrence on the command line.
- -o specifies the name of the executable (a.out by default)
- -p permits profiling with prof or gprof
- -qarch=pwr2 -qtune=pwr2 -O3 permits maximum optimization for the SP processor architecture
- -v causes a "verbose" output listing of the compile shell script

Setting Environment Variables

There are many environment variables and command line flags that you can set to influence the operation on PE tools and the execution of parallel programs. A complete discussion and list of the PE environment variables can be found in the IBM AIX Parallel Environment Operation and Use manual. They can also be reviewed (in less detail) in the POE man page.
Environment variables may be set on the shell command line or placed within your shell's "dot" files (.cshrc, .profile). Alternately, they may be put into a file which is "sourced" prior to execution.
PE environment variables can be overridden by supplying the appropriate flag when the program executable is invoked. See the POE man page for details.
Although there are many PE environment variables, only a few are required. These important environmental variables are listed below.

MP_PROCS
The number of nodes to allocate for your partition. If not set, the default is 1.

MP_RESD
Specifies whether or not the Partition Manager should connect to the SP system Resource Manager to allocate nodes. Valid values are either "yes" or "no", and there is no default. In most cases, this should be set to "yes".

MP_EUILIB
Specifies which CSS library implementation to use for communication. Valid values are either "ip" for Internet Protocol CSS or "us" for User Space protocol. The default is "ip".

MP_EUIDEVICE
Used to specify which network adapter should be used for communications. Valid values are: "en0" (ethernet), "fi0" (FDDI), "tr0" (token-ring), or "css0" (high-performance switch). Note that valid values will also depend upon the actual physical network configuration of the node. For example, to specify "fd0" when a node does not physically have a FDDI adapter is an error.

MP_HOSTFILE
Specifies the filename containing the names of nodes you wish to use. Can be set to any file name. The default filename is "host.list" in the current directory. You do not need to set this variable if your host list file is the default "host.list".
You must specify a host list file if you need specific node allocation, request non-specific node allocation from different system pools or use a host list file named something other than the default "host.list".
If you prefer to let the Resource Manager automatically allocate nodes then this variable should be set to NULL or "".

MP_RMPOOL
The number of the SP system pool that should be used by the Resource Manager for non-specific node allocation. This is only valid if you are using the SP Resource Manager for non-specific node allocation (from a single pool) without a host list file. You may obtain information about available pools by typing "jm_status -P" at the Unix prompt.

Some Preset POE Environments

The following four sets of POE environment variables should cover most users' needs and are provided to help simplify getting started.
These files may all be copied to your current directory with the command:
```
    cp  $WORKSHOP/samples/poe/poe_vars*  .
    
```
If you copy these for your use, you will need to "source" them before they take effect. For example:
```
    C Shell:      source  poe_vars1 
    Korn Shell:   .   poe_vars1
    
```
You may also add the variable settings of your choice to your shell's "dot" file (.cshrc, .profile, etc.) so that you don't have to remember to source them later.

The Host List File

The host list file is not required if you let the Resource Manager allocate which nodes your job uses (MP_HOSTFILE set to NULL or ""). The host list file is required if you want to do any of the following:
1. Explicitly specify the actual nodes which your job should use. Node names are listed one per line. Note: actual names will depend upon your system.
```
    ! Specific node allocation for 4 tasks
    fr3n05.mhpcc.edu
    fr3n01.mhpcc.edu
    fr3n03.mhpcc.edu
    fr3n07.mhpcc.edu
    
```
2. Use non-specific node allocation from more than one pool. Each line specifies a pool number preceded by the "at" symbol (@). In the example below, the first two tasks will be allocated to processors in pool 6, the third task will go to pool 9 and the fourth task will execute in pool 12. Note: the actual pool numbers will depend upon your system.
```
    ! Non-specific allocation from different pools for 4 tasks
    @6
    @6
    @9
    @12
    
```
3. Override the default use of the node's CPU and / or adapter. This is not recommended in most cases. See the CPU and adapter usage discussion for details.
By default, the host list file is called host.list and resides in your current directory. If you want to use a filename other than the default name host.list , then you will need to set the MP_HOSTFILE environment variable to that file's name. For example:
```
    C Shell:      setenv  MP_HOSTFILE  /u/jsmith/parallel/16hosts
    Korn Shell:   export  MP_HOSTFILE=/u/jsmith/parallel/16hosts
    
```
The actual number of nodes used by your job depends upon the MP_PROCS variable. Some considerations:
- Nodes are used in the order they are listed in the host list file.
- If a listed host is unavailable, the Partition Manager returns an error and the program is not executed - even if there are more than the number of nodes needed in your host list file.
- For jobs using IP communications, if a host is listed more than once in the host list file, then multiple tasks will be allocated to that particular host. This can not be done for US communications because the switch adapter must be dedicated.

If the environment variable MP_SAVEHOSTFILE is used during a program execution , the resulting output file can be saved and used as a host list file later if desired. An example appears below. Note that it automatically includes additional information about CPU usage, adapter usage and adapter network addresses. Comment lines begin with an exclamation mark (!).


    ! myhosts: allocation file 4 tasks
      fr29install4.mhpcc.edu  dedicated  multiple 
    ! external_IP 164.122.53.46  MPL_IP 164.122.217.113
    ! @3
      fr29n02.mhpcc.edu  dedicated  multiple 
    ! external_IP 164.122.189.102  MPL_IP 164.122.217.114
    ! @3
      fr29n03.mhpcc.edu  dedicated  multiple 
    ! external_IP 164.122.189.103  MPL_IP 164.122.217.115
    ! @3
      fr29n04.mhpcc.edu  dedicated  multiple 
    ! external_IP 164.122.189.104  MPL_IP 164.122.217.116
    ! @3

Invoking the Executable

Once the environment is setup and the executables are created, invoking the executables is relatively easy.

Optional - start any X-Windows analysis tools (ie. pmarray) before the executable is invoked.
For SPMD programs, simply issue the name of the executable, specifying any command line flags that may be required. Command line flags may be used to temporarily override any MP environment variables that have been set. See the POE man page for a complete listing of flags.
For MPMD programs, the following steps must be performed:
1. Set the MP_PGMMODEL environment variable to "mpmd". For example:
```
    C Shell:      setenv MP_PGMMODEL mpmd
    Korn Shell:   export MP_PGMMODEL=mpmd
    
```
2. Enter poe at the Unix prompt. You will then be prompted to enter the executable which should be loaded on each processor.
3. Manually load each program on the processor nodes. Alternatively, the program names may be listed in a file and poe will automatically load the nodes based on that file. Set the environment variable MP_CMDFILE to that file name. This example loads a "master" program on the first processor and 4 "slave" tasks on the remaining four processors.
```
    0:node1> master 
    1:node2> slave 
    2:node3> slave
    3:node4> slave
    4:node5> slave
    
```
4. Execution starts automatically after the last node has been loaded.

For serial programs or commands, use the poe command followed by your program name or the command you wish to run across your partition. For example:


    poe  cp ~/input.file  /tmp/input.file
    poe  my_serial_job
    poe  rm /tmp/input.file

CPU and Communications Adapter Usage

POE jobs typically require both CPU and communications adapter resources. The manner in which a job uses these two resources effects both job performance and whether or not other users can run jobs on the same node.
CPU Usage: may be either "unique" or "multiple"
- IP: default = multiple. Permits any number of other IP jobs and up to one US job to run on the same node.
- US specific node allocation: default = multiple. Permits any number of other IP jobs and NO other US job to run on the same node.
- US non-specific node allocation: default = unique. No other jobs permitted on same node - either IP or US. Set the environment variable MP_CPU_USE to "multiple" to override this.
Communications Adapter Usage: may be either "shared" or "dedicated"
- IP: default = shared. Permits any number of other IP jobs and up to one US job to share the communications adapter on a node.
- US: default = dedicated. Permits any number of other IP jobs and NO other US jobs to share the communications adapter on a node. Can not be overridden.
Best performance is usually obtained running with US communications when CPU use is "unique" and the adapter is "dedicated".
The best "good neighbor" policy is for all POE jobs to run with IP communications using the defaults of CPU use "multiple" and adapter "shared".
For the most part, users do not need to change the default settings for CPU usage and communications adapter usage. One instance where the default might be changed concerns the use of US communications in an interactive pool of nodes shared by many users. In this case, it might be considered "good neighbor policy" to set MP_CPU_USE to multiple so that at least other IP jobs can run also.

Miscellaneous Environment Variables

A complete list of the POE environment variables can be viewed in the POE man page. A list of the more commonly used "miscellaneous" variables appears below.

MP_CPU_USE

Specifies CPU usage. Valid values are "unique" and "multiple". The default for IP communications is "multiple". US communications defaults are "multiple" if specific node allocation is used, "unique" otherwise.


    C Shell:    setenv MP_CPU_USE multiple
    Korn Shell: export MP_CPU_USE=multiple

MP_RETRY

The period (in seconds) of time between processor node allocation retries if there are not enough processor nodes immediately available.


    C Shell:    setenv MP_RETRY 10
    Korn Shell: export MP_RETRY=10

MP_RETRYCOUNT

The number of times that the Partition Manager should attempt to allocate processor nodes before returning without running your program.


    C Shell:     setenv MP_RETRYCOUNT 15
    Korn Shell:  export MP_RETRYCOUNT=15

MP_SAVEHOSTFILE

The name of an output host list file to be generated by the Partition Manager.


    C Shell:     setenv MP_SAVEHOSTFILE progname.hosts.used
    Korn Shell:  export MP_SAVEHOSTFILE=progname.hosts.used

MP_PGMMODEL

Determines the programming model you are using. Valid values are spmd or mpmd. If not set, the default is spmd. If set to "mpmd" you will be enabled to load different executables individually on the nodes of your partition.


    C Shell:     setenv MP_PGMMODEL mpmd
    Korn Shell:  export MP_PGMMODEL=mpmd

MP_CMDFILE

Determines the name of a POE commands file used to load the nodes of your partition. If set, POE will read the commands file rather than STDIN. Valid values are any file specifier. Generally used only when MP_PGMMODEL=mpmd.


    C Shell:     setenv MP_CMDFILE mpmd.hosts
    Korn Shell:  export MP_CMDFILE=mpmd.hosts

MP_EUIDEVELOP

Causes MPL to do more detailed checking during program execution.


    C Shell:     setenv MP_EUIDEVELOP yes
    Korn Shell:  export MP_EUIDEVELOP=yes

MP_STDOUTMODE

Enables you to manage the STDOUT from your parallel tasks. If set to "unordered" all tasks write output data to STDOUT asynchronously. If set to "ordered" output data from each parallel task is written to its own buffer. Later, all buffers are flushed, in task order, to stdout. If a task id is specified, only the task indicated writes output data to stdout. The default is unordered.


    C Shell:     setenv MP_STDOUTMODE ordered
    Korn Shell:  export MP_STDOUTMODE=ordered
            -or-
    C Shell:     setenv MP_STDOUTMODE unordered
    Korn Shell:  export MP_STDOUTMODE=unordered
            -or-
    C Shell:     setenv MP_STDOUTMODE 6
    Korn Shell:  export MP_STDOUTMODE=6

MP_INFOLEVEL

Determines the level of message reporting. Default is 1. Valid values are:

Note that use of this feature can consume significant system resources and should not be set above zero routinely.


    C Shell:     setenv MP_INFOLEVEL 2
    Korn Shell:  export MP_INFOLEVEL=2

SP_NAME

Specifies the name of the Control Workstation. This variable is used by the System Status Array tool.

Parallel File Copy Utilities

POE provides four utilities which may be used to copy file(s) to and from a number of nodes.
Three of these utilities (mcp, mcpscat, mcpgath) are actually message passing applications designed for efficiency. One utility (mprcp) is a shell script which uses the standard UNIX utility rcp to perform the copy operations and may not be as efficient.
Note: see the associated hyperlinked man page for examples of each utility's use.

mcp
Copies a single file from the home node to a number of remote nodes.

mcpscat
Copies a number of files from task 0 and scatter them in sequence to all tasks, in a round robin order.

mcpgath
Copies a number of files from all tasks back to task 0.

mprcp
Copies a file from the home node to a list of remote hosts.

Recommendations for Running on the MHPCC SP2

Set MP_PROCS to the number of nodes you wish to use.
Inform the Resource Manager about your job (MP_RESD = yes).
Use dynamic linking of the communication libraries instead of static linking. This will give you more flexibility at run time - you can alternate between IP and US communications simply by changing the MP_EUILIB variable.
Always use the high performance switch for SP communications (MP_EUIDEVICE = css0).
Write scratch files to local scratch space whenever possible. At the MHPCC, each node has a /localscratch directory - it is faster than writing scratch files to your home directory.
Interactive Users:
- Always use IP communications (MP_EUILIB = ip). This is "good neighbor" use of the limited number of interactive nodes. It will permit other POE jobs to run at the same time.
- If you need (for testing purposes) to run with US communications in the Interactive pool, be sure to share the CPU with other users (MP_CPU_USE = multiple). If this is not done, even IP jobs can not run on the same node as your job.
- Unless there is a particular reason for selecting certain nodes, do not use a host list file. Allow the Resource Manager to automatically allocate your nodes (MP_HOSTFILE = NULL or "").
- During program development it may be helpful to set MP_INFOLEVEL to a higher value (>3).
- If you are having trouble getting the requested number of nodes for your job, be sure that you are using IP communications (MP_EUILIB = ip). If you still have problems, check what other users are doing in the Interactive pool with the command: jm_status -Pv | more . Most likely, somebody is running a User Space job without MP_CPU_USE set to multiple. Then notify the MHPCC User Services staff by sending email to "help@mhpcc.edu".
LoadLeveler (batch) Users:
- POE environment variables and adapter usage are specified in your LoadLeveler command file. For example:
```
    #@ requirements = (Adapter == "hps_user")
    #@ environment = MP_EUILIB=us;MP_INFOLEVEL=3;MP_LABELIO=yes
    
```
- Alway use US communication over the switch. The above example demonstrates how to do this.
- Not all POE variables are used by LoadLeveler. The following variables will have no effect if used in a LoadLeveler job command file.
```
    MP_PROCS
    MP_RMPOOL
    MP_EUIDEVICE
    MP_HOSTFILE
    MP_SAVEHOSTFILE
    MP_PMDSUFFIX
    MP_RESD
    MP_RETRY
    MP_RETRYCOUNT
    MP_ADAPTER_USE
    MP_CPU_USE 
    
```
- Don't forget that certain POE environment variables in your "dot" files (.cshrc, .profile, .login) will override the settings you specify in LoadLeveler.
- LoadLeveler always acquires nodes for POE jobs directly from the Resource Manager and ignores user host list files.

References, Acknowledgements and WWW Resources

Additional Information on the WWW

"The IBM SP Parallel Environment and Parallel Operating Environment" tutorial on the Cornell Theory Center's WWW server, located at http://www.tc.cornell.edu/Edu/Tutor/POE/index.html.

References and Acknowledgements

"IBM Parallel Environment for AIX: Operation and Use, Version 2.1.0". IBM Corporation. Available online at the MPHCC by using InfoExplorer with the command: info -l pe .
"IBM AIX Parallel Environment Operation and Use, Release 2.0". IBM Corporation.
We gratefully acknowledge the IBM Corporation for providing much of the original material included in this document.

Documents located on the Maui High Performance Computing Center's WWW server are copyrighted by the MHPCC. Educational institutions are encouraged to reproduce and distribute these materials for educational use as long as credit and notification are provided. Please retain this copyright notice and include this statement with any copies that you make. Also, the MHPCC requests that you send notification of their use to help@mail.mhpcc.edu.

Commercial use of these materials is prohibited without prior written permission.

Revised: 17 July 1996 blaise@mhpcc.edu

SP Parallel Programming Workshop Parallel Operating Environment (POE)

Additional Information on the WWW

References and Acknowledgements

SP Parallel Programming Workshop
Parallel Operating Environment (POE)