HPCPOWER User Guide

Index

1. General

2. Compilers / Debuggers for parallel environment

3. Resource Management System

4. Software Libraries/Packages

1.General

1.1 System overview

The hpcpower uses the Redhat Linux 9.0 as its operating system. Reading of this user guide and using of the system assumes familiarity with the Linux/Unix software environment. In order to get an understanding of the Linux/UNIX, please study the  UNIX user's guide in the Computer Centre's web page.

The hpcpower uses the Portable Batch System (PBS) software to distribute the computational workload across the processors. PBS is a batch job scheduling application that provides the facility for building, submitting and processing batch jobs on the system.

Jobs are submitted to the system by creating a PBS job command file that specifies certain attributes of the job, such as how long the job is expected to run and, in the case of parallel programs, how many processors are needed, and so forth. PBS then schedules when the job is to start running on the cluster (based in part on those attributes), runs and monitors the job at the scheduled time, and returns any ouput to the user once the job completes.

1.2 Logging into & transferring file to the system

1.3 Editing the program

Important notice for Microsoft Windows users: do not use a standard Microsoft Windows editor such as Notepad to edit files that will be used on the Linux or other Unix systems. The two systems use different sequences of control characters to mark the end of line (EOL). If you are using the system from a Microsoft Windows desktop machine, please SSH to the master node and edit the program directly using pico.

1.4 Configuring your account

You can copy the most up-to-date system files from the directory /etc/skel in case your copies are deleted by accident.

When a job is submitted to the cluster through PBS a new login to your account is initiated, and any initialization commands in your startup files (.bashrc, .bash_profile, etc) are executed. In this case (running in batch mode) it is necessary not to put  interactive commands such as setting tset and stty or generating outputs in the startup files. If these precautions are not taken then error messages will be written to the batch jobs error file and your program cannot run.
 

2. Compilers

2.1 The PGI compilers

Quick list of arguments for the compiler can be found using the compile command with the argument -help .

Most commonly used arguments for the compilers are :
 

-tpp7                   Choose Pentium IV target processor
-fast             Common optimizations
-Mfixed           Assume fixed-format source for Fortran programs
-Mfree            Assume free-format source for Fortran programs
-Mmpi             Link with mpi libraries
-Mprof=lines,mpi  Profiling a mpi program
-Mprof=lines      Profiling a serial program

2.2 Program Compilation and testing

Compile commands

MPI C programs :
      pgcc -Mmpi program.c -o program.exe
MPI Fortran 77 or Fortran 90 programs in fixed form :
      pgf90 -Mmpi program.f -o program.exe
MPI Fortran 77 or Fortran 90 programs in free form :
      pgf90 -Mmpi -Mfree program.f -o program.exe
High Performance Fortran Programs :
      pghpf -Mfreeform program.f -o program.exe

Example of MPI programs is located in /etc/skel/  .
 

Test serial program

The "./" tells the system to run the program in the current directory.
 

Test MPI program

mpirun  ./program.exe

You do not need to specify the number of processes to test your job. For interactive testing, the default number of processes is 4. You can add "-np xxx" to specify the number of processes for testing.
 

Profiling a program

After the program is run, serial profile program would output a file called pgprof.out. You can display this file using the command pgprof .

For parallel profile program, after the program is run, there are 4 files called pgprof.out, pgprof1.out, pgprof2.out and pgprof3.out . Only the last 3 can be displayed. You can use this command to display the result :

pgprof -merge pgprof1.out pgprof2.out pgprof3.out

2.3 The PGI debuggers

Use "pgdbg" as the debugger for PGI compiled program. It will show GUI if launched at X-windows environment.

pgdbg

3. Resource Management System

3.1 PBS (Portable Batch System)

To use PBS, you create a batch job command file which you submit to the PBS server to run on the system. A batch job file is simply a shell script containing the set of commands you want to run on the batch nodes. It also contains directives which specify the characteristics (attributes), and resource requirements (e.g. number of nodes and maximum runtime) that your job needs. Once you create your PBS job file, you can reuse it if you wish or modify it for subsequent runs.

Since the system is set up to support large computation jobs, the following maximum number of CPUs and maximum processing time are allowed for each batch job:

- Maximum number CPUs  for each program job = 64 (i.e. 32 compute-nodes of 2 CPUs)
- Maximum processing time for each program job = 10 Hours (wall clock time)

Furthermore, the job scheduling is set in such a fashion that higher priority will be given parallel jobs requiring a larger number of processors.

In order to provide a fair share environment for all users, the system is set such that each user can put no more than 4 jobs on the job queue and no more than 2 of them should be running at the same time.
 

3.2 PBS Job command file

To submit a job to run on the SMP system, a PBS job command file must be created. The job command file is a shell script that contains PBS directives which are preceded by #PBS.

The following is an example of a PBS command file to run a parallel job, which would require 4 nodes and 1 processors on each node. This file is also located in the system as /etc/skel/pbs.cmd.
 

#!/bin/sh
### Job name
#PBS -N test
### Declare job non-rerunable
#PBS -r n
### Mail to user
#PBS -m ae

### Queue name (qprod or qdev)

### qprod is the queue for running production jobs.
### 126 nodes can run jobs in this queue.
### Each job in this queue can use 1-32 nodes.
### Parallel jobs will be favoured by the system.
### Walltime can be  00:00:01 to 10:00:00

### qdev is the queue for program testing.
### 2 nodes can run jobs in this queue.
### Each job in this queue can use 1-2 nodes.
### Walltime can be  00:00:01 to 00:15:00

#PBS -q qprod
#####PBS -q qdev

### Wall clock time required. This example is 2 hours
#PBS -l walltime=02:00:00

### Number of nodes

### The following means 1 node and 1 processor.
### Clearly, this is for serial job
###PBS -l nodes=1:ppn=1

### The following means 4 nodes required. Processor Per Node=1,
### i.e., total 4 processors needed to be allocated.
### ppn (Processor Per Node) can be either 1 or 2.
#PBS -l nodes=4:ppn=1

### Another example
### 4 nodes required. Processor per node=2,
### total 8 processors need to be allocated.
###PBS -l nodes=4:ppn=2

###The following stuff will be executed in the first allocated node.
###Please don't modify it
 

cd $PBS_O_WORKDIR

# Define number of processors
NPROCS=`wc -l < $PBS_NODEFILE`

echo $PBS_JOBID : $NPROCS processors allocated : `cat $PBS_NODEFILE`
echo =======================================================

# Run serial executable "a.out" in one batch node like this :
# ./a.out > ${PBS_JOBNAME}.`echo ${PBS_JOBID} | cut -c 1-5 `

# Run the parallel MPI executable "a.out"
# Remember, users don't need to specify "-np" nor "-machinefile"
# in batch nodes.
mpirun ./a.out > ${PBS_JOBNAME}.`echo ${PBS_JOBID} | cut -c 1-5 `
 

After the PBS directives in the command file, the shell executes a change directory command to $PBS_O_WORKDIR, a PBS variable indicating the directory where the PBS job was submitted and nominally where the progam executable is located. Other shell commands can be executed as well. In the last line, the executable itself is invoked.

If we are running MPI program, then the command "mpirun ./programfile " should be used. There is no need to tell the MPI how many nodes and where the machine file is because the PBS has already passed these information to the MPI system.

The parameter > ${PBS_JOBNAME}.`echo ${PBS_JOBID} | cut -c 1-5 `  would redirect the standard output of the program to a text file jobname.jobid . You can inspect this file from time to time to check the progress of the program.
 

3.3 Submitting a Job

[h0xxxxxx@hpcpower test]$ qsub pbs.cmd
2234.hpcpower

Upon successful submission of a job, PBS returns a job identifier of the form jobid.hpcpower where jobid is an integer number assigned by PBS to that job. You'll need the job identifier for any actions involving the job, such as checking job status or deleting the job.

When the job is being executed, it stores the program outputs to the file jobname.xxxx where xxxx is the job identifier of the job. At the end of the job, the file jobname.oxxxx and jobname.exxxx would also be copied to the working directory to show the standard output and error which were not explicited redirected in the job command file.

Also, an e-mail message would be sent to the user when the job is finished. The following is an example of a PBS email notification to the user at the end of the job:

Date: Fri, 12 Dec 2003 17:16:52 +0800
From: adm <adm@hpcpower.hku.hk>
To: h0xxxxxx@hpcpower.hku.hk
Subject: PBS JOB 1717.hpcpower

PBS Job Id: 1717.hpcpower
Job Name:   pd2-sn1
Execution terminated
Exit_status=0
resources_used.cput=00:18:07
resources_used.mem=92104kb
resources_used.vmem=330380kb
resources_used.walltime=00:18:24
 

Note that the walltime-used information in the e-mail message should be used to accurately estimate the walltime resource requirement in the PBS job command file for future job submissions so that PBS can more effectively schedule the job. When submitting a particular PBS job for the first time, the walltime requirement should be overestimated to prevent premature job termination.
 

3.4 Manipulating a Job

List all jobs

hpcpower:
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
2230.hpcpower   ycleung  large    FORCED_CHA   5597  16  --    --  10:00 R 07:17
2231.hpcpower   ycleung  large    CHANNEL_LI  11048  16  --    --  10:00 R 04:02
2233.hpcpower   chliu    large    HUGE_CAVIT   4485  16  --    --  10:00 R 01:51
 
 

List all nodes

List detail of the job

Delete a job

4. Software Libraries/Packages

4.1 Gaussian 03 Job (structure calculation software)

The following is a brief description of how to run Gaussian. Please also study the detail information about Gaussian in hpcpower.

Serial G03

To run Gaussian 03, it is easy to trace the input and output files if the input file and output file has the same prefix as the jobname. In the following example, the Gaussian input file input.com is used (other PBS commands were omitted) :

### PBS -N means setting the job name
#PBS -N input

#PBS -l nodes=1:ppn=1
g03 < ${PBS_JOBNAME}.com  \
         >   ${PBS_JOBNAME}.`echo ${PBS_JOBID} | cut -c 1-5 `
 
 

Running G03 using 2 processors

#PBS -l nodes=1:ppn=2

and put the following line in the G03 input file :

%NprocShared=2

4.2 Parallel Gaussian 03 Jobs

4.3 IMSL library (International Mathematics and Statistics Library)

. /share/bin/startintel

For Fortran 77 standard program, compile your_program.f by:

/share/bin/imsl your_program

For Fortran 90 stardard program, compile your_program.f90 by:

/share/bin/imsl90 your_program

4.4 BLAS & LAPACK library (Basic Linear Algebra Subprograms & Linear Algebra PACKage)

E.g. Compile C/C++ program by:

pgcc -Mmpi -lblas -llapack your_program.c

E.g. Compile Fortran program by:

pgf77 -Mmpi -lblas -llapack your_program.f

pgf90 -Mmpi -lblas -llapack your_program.f90

4.5 R statistical package (Statistical package)

To debug R interactively :

/share/bin/R

To run R in batch mode:

/share/bin/R --no-save < INPUT_FILE > OUTPUT_FILE

User are advised to run R in batch mode by submitting the above command to serial job queue.

An Introduction to R

The R Manuals

4.6 GROMACS (Versatile package to perform molecular dynamics)

. /share/bin/gromacs.sh

GROMACS Online Reference

GROMACS Tutorial

4.7 Abaqus (Package for finite element analysis)

4.8 HYPHY (Hypothesis Testing Using Phylogenies)

mpirun /share/bin/HYPHYMPI HYPHY_BATCH_FILE

A sample PBS job command file can be copied from /share/HYPHY/Tutorial/hyphy.cmd and a sample HYPHY batch file can be found at /share/HYPHY/Tutorial/MPITest.bf.

HYPHY Documentation

4.9 Mrbayes (Parallel Bayesian estimation of phylogeny)

mpirun /share/bin/mb MRBAYES_BATCH_FILE

It is reminded that you need to include the mrbayes_batch_file and the NEX files in the same directory. A sample PBS job command file can be copied from /share/mrbayes/pbs-mb.cmd and a sample mrbayes batch file can be found at /share/mb/run_mb_batch.

MRBAYES Documentation

4.10 PHYML (Parallel Phylogenies by Maximum Likelihood)

mpirun /share/bin/phyml sequences_file data_type format data_sets bootstrap_sets model [kappa] invar nb_categ alpha tree opt_topology opt_lengths

A sample PBS job command file can be copied from /share/phyml/pbs-phyml.cmd

PHYML Documentation

4.11 Garli (Parallel Genetic Algorithm for Rapid Likelihood Inference)

mpirun /share/bin/ParGarli.94 GARLI_CONFIGURATION_FILE

A sample PBS job command file can be copied from /share/GARLI/example/garli-pbs.cmd

GARLI Documentation

4.12 VNC Server (X-Window emulation on MS Windows)

[hpcpower]$ vncserver
(On the first time you are running VNC Server, you will be asked to set the password.)
You will require a password to access your desktops.
Password: ********
Verify: ********
New 'X' desktop is hpcpower.hku.hk:3
(The number at the end may not be 3, it will vary from session to session)

Remember this number (this example is 3) and use it for the steps below. You can use the same number whenever you need to connect. There is no need to re-run "vncserver" again.
If you have forgotten this number, you need to run "vncserver" again as above to get a new number.

3. Make sure your client PC is connected to HKU campus network, and then run the VNC Viewer program ("Start" -> "Programs" -> "Real VNC" -> "VNC Viewer" -> "Run VNC Viewer") in your windows platform, then type "hpcpower.hku.hk:3". The number 3 at the end will vary, depending on what you see after running vncserver.

Then type in the password you create at step 2.

Then you will see the Linux GNOME X-window at your client PC. Press right mouse button at the desktop and choose "New Terminal". Then execute "xclock" at the new terminal will launch a X-window clock interface in the VNC session.