PBS commands reference

Supercomputers that are shared among many users typically use a job management system such as Torque to manage submission and execution of jobs. Jobs are not run directly from the command line- the user needs to create a job script which specifies both the required compute resources and the job that is to be run. The script is submitted to the job management system (or queueing system) and if the requested resources (processors, memory, etc) are available on the system, the job will by run. If not, it will be placed in a queue until such time as the resources do become available. In order to provide a fair share of the resources among users, the priority of jobs in the queue may be varied based on how much resources someone has used, so it is possible that jobs may not run in the order in which they have been submitted to the queue.

Users wanting to use the supercomputers need to understand how to use the queuing system, how to create the job submission script, as well as check its progress or delete a job from the queuing system.

PBS command vs SGE commands

See PBS command vs SGE commands

Examples:

pbsnodes -a | egrep '^[a-z]|state' -- get only states
 

Note: TORQUE (Big Red II, Karst, and Mason) relies on Moab to dispatch jobs; SGE (Rockhopper) does not. For a list of useful Moab commands, see Common Moab scheduler commands.

Torque PBS job management cheat sheet Genome Informatics Facility (GIF)

Job management commands

Job script basics

A typical job script will look like this:

#!/bin/bash
#PBS -l nodes=1:ppn=16
#PBS -l walltime=48:00:00
#PBS -N 
#PBS -o ${PBS_JOBNAME}.o${PBS_JOBID} 
#PBS -e ${PBS_JOBNAME}.e${PBS_JOBID}
#PBS -m ae -M [email protected]
cd $PBS_O_WORKDIR
module use /data003/GIF/software/modules/
module load yourmodule
your_commands_goes_here

Lines starting with #PBS are for Torque resource manager to request resources for HPC. Some important options are as follows:

Interactive session

To start a interactive session you need to use the option -I, with all the other options you normally put in a PBS script file. For example:

qsub -I -N stdin -l nodes=1:ppn=16 -l walltime=1:00:00

Torque PBS queuing system eResearch SA

• About
• Torque PBS queuing system commands
  • The most useful commands
• Tutorial using PBS
  • Configure your QSUB script
  • Submit the Job to the queue: qsub
  • Look at the current Queue Status: qstat
  • Delete a Job From the Queue: qdel
  • See only your jobs on the Queue: qstat -u $USER
  • Analysing the PBS Output file : $PBS_JOBNAME.o$PBS_JOBID
    • Side-by-side comparison of PBS Submit and PBS Output Files
    • Explanation of each section in PBS Output File
• Tutorial using PBS to run an MPI program
  • Tutorial Output Files
    • Hello World Output File
    • Matrix Output File

Torque PBS queuing system commands

• The queue software Torque PBS is used to manage access to the backend compute clusters.
• Torque user manual

The most useful commands

Tutorial using PBS

See the HPC quick start guide for how to connect to the HPC Machines.

Configure your QSUB script

• You may first need to take a look at Most Basic Vi Commands You Need, and when you are more advanced, see the Vi Cheat Sheets: Vi cheat sheet and Vi editor cheat sheet.
• First step is to copy a pre-configured template from the folder ~/.templates in your home folder to the current directory for the specific cluster.
  • Make an Experiment Directory:

    mkdir MyFirstExperiment
    cd MyFirstExperiment
    

  • Copy the pre-configured template:

    cp ~/.templates/tizard.sub myexperiment-1.sub
    

• Customise the template myexperiment-1.sub by editing it using the vi text editor.
  • Change: MyJobName to give the job a descriptive name so you can recognise the output files:

    ### Job name
    #PBS -N  MyJobName
    

  • Change: Your-email-Address to your actual email address:

    ### email address for user
    #PBS -M Your-email-Address
    

  • Change the Requested Resources: nodes, ppn, mem, vmem, walltime:

    ### Request Resources
    #PBS -l nodes=1:ppn=Y
    #PBS -l mem=Xgb,vmem=Xgb
    #PBS -l walltime=HH:MM:SS
    

  • The modules you need:

    #Load module(s) if required
    module load application_module
    

  • The Application name and input files:

    # Run the executable
    MyProgram+Arguments
    

• Here are the differences between the two files - indicated by the | when the lines are different:

[demouser@tizard1 ~/MyFirstExperiment]$ sdiff myexperiment-1.sub ~/.templates/tizard.sub

#!/bin/csh							#!/bin/csh

#PBS -V								#PBS -V

### Job name							### Job name
#PBS -N MyFirstExperiment				      |	#PBS -N MyJobName

### Join queuing system output and error files into a single 	### Join queuing system output and error files into a single
#PBS -j oe							#PBS -j oe

### Send email to user when job ends or aborts			### Send email to user when job ends or aborts
#PBS -m ae							#PBS -m ae

### email address for user					### email address for user
#PBS -M [email protected]				      |	#PBS -M Your-email-Address

### Queue name that job is submitted to				### Queue name that job is submitted to
#PBS -q tizard							#PBS -q tizard

### Request nodes NB THIS IS REQUIRED				### Request nodes NB THIS IS REQUIRED
#PBS -l nodes=1:ppn=2					      |	#PBS -l nodes=1:ppn=Y
#PBS -l mem=2gb,vmem=4gb				      |	#PBS -l mem=Xgb,vmem=Xgb
#PBS -l walltime=00:10:00				      |	#PBS -l walltime=HH:MM:SS

# This job's working directory					# This job's working directory
echo Working directory is $PBS_O_WORKDIR			echo Working directory is $PBS_O_WORKDIR
cd $PBS_O_WORKDIR						cd $PBS_O_WORKDIR
echo Running on host `hostname`					echo Running on host `hostname`
echo Time is `date`						echo Time is `date`

#Load module(s) if required					#Load module(s) if required
module load intel/11.9.293 openmpi/intel64/1.4.3	      |	module load application_module

# Run the executable						# Run the executable
mpirun -np 2 hostname					      |	MyProgram+Arguments

#SLEEP for 60 seconds - so you get time to see the job in the queue
sleep 60

Submit the Job to the queue: qsub

qsub

  qsub myexperiment-1.sub
  92853.tizard1

• NOTE: the full jobid is 92853.tizard1 (but just 92853 is usually sufficient)

Look at the current Queue Status: qstat

qstat -a

	  qstat -a
tizard1:
                                                                         Req'd  Req'd   Elap
Job ID               Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
-------------------- ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
92853.tizard1        demouser    tizard   MyFirstExperimen    --      1      2    2gb 00:10 Q   --

• We see a 'Q' under 'S', meaning that the job has been queued and is waiting to run.

Once the job has started, let's look again.

qstat -an

	  qstat -an
tizard1:
                                                                         Req'd  Req'd   Elap
Job ID               Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
-------------------- ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
92853.tizard1        demouser    tizard   MyFirstExperimen    986     1      2    2gb 00:10 R   --
   tizard20/1+tizard20/0

• Here we see the job is running (indicated by 'R') on node: tizard20 and it's using 2 cpu's
  We can get the same output with qstat followed by the jobid : 92853

qstat -f

Below we see a snippet of qstat -f

	qstat -f 92853
[demouser@tizard1 ~/MyFirstExperiment]$ qstat -f 92853
Job Id: 92853.tizard1
    Job_Name = MyFirstExperiment
    Job_Owner = demouser@tizard1
    job_state = R
    queue = tizard
    server = tizard1
    ...

Run the qstat -f command to see full results.

Delete a Job from the queue: qdel

qdel 92853

... nothing to see .... but the job is deleted when you look at qstat

See only your jobs on the queue: qstat -u $USER

Let's submit four jobs and see them on the queue:

[demouser@tizard1 ~/MyFirstExperiment]$ qsub myexperiment-1.sub
92863.tizard1
[demouser@tizard1 ~/MyFirstExperiment]$ qsub myexperiment-1.sub
92864.tizard1
[demouser@tizard1 ~/MyFirstExperiment]$ qsub myexperiment-1.sub
92865.tizard1
[demouser@tizard1 ~/MyFirstExperiment]$ qsub myexperiment-1.sub
92866.tizard1

Now let's see only our own jobs:

qstat -u username -an

• Note $USER will automagically fill in your username for you, or you can supply it.

[demouser@tizard1 ~/MyFirstExperemnt]$ qstat -u $USER -an
tizard1:
                                                                         Req'd  Req'd   Elap
Job ID               Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
-------------------- ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
92863.tizard1        demouser    tizard   MyFirstExperimen  25064     1      2    2gb 00:10 R   --
   tizard46/47+tizard46/46
92864.tizard1        demouser    tizard   MyFirstExperimen   1293     1      2    2gb 00:10 R   --
   tizard20/1+tizard20/0
92865.tizard1        demouser    tizard   MyFirstExperimen   1333     1      2    2gb 00:10 R   --
   tizard20/3+tizard20/2
92866.tizard1        demouser    tizard   MyFirstExperimen   1338     1      2    2gb 00:10 R   --
   tizard20/5+tizard20/4

• Now let's delete one of these jobs:

    [demouser@tizard1 ~/MyFirstExperiment]$ qdel 92865
  

• And Examine the Queue to see this job has gone:

  [demouser@tizard1 ~/MyFirstExperiment]$ qstat -u $USER -an
  tizard1:
                                                                         Req'd  Req'd   Elap
  Job ID               Username    Queue    Jobname          SessID NDS   TSK    Memory Time  S Time
  -------------------- ----------- -------- ---------------- ------ ----- ------ ------ ----- - -----
  92863.tizard1        demouser    tizard   MyFirstExperimen  25064     1      2    2gb 00:10 R   --
   tizard46/47+tizard46/46
  92864.tizard1        demouser    tizard   MyFirstExperimen   1293     1      2    2gb 00:10 R   --
   tizard20/1+tizard20/0
  92866.tizard1        demouser    tizard   MyFirstExperimen   1338     1      2    2gb 00:10 R   --
   tizard20/5+tizard20/4
  

Analysing the PBS Output file: $PBS_JOBNAME.o$PBS_JOBID

• The output file from PBS is created after your job has finished.
• The file name is a concatenation of the JOBNAME .dot. o JOBID
• e.g. for this job it will be: MyFirstExperiment.o$PBS_JOBID which is 92853
  • i.e. MyFirstExperiment.o92853
  • cat MyFirstExperiemnt.o92853

    Warning: no access to tty (Bad file descriptor).
    Thus no job control in this shell.
    Working directory is /home/users/demouser/MyFirstExperiment
    Running on host tizard20
    Time is Tue Oct 30 16:49:32 CST 2012
    tizard20
    tizard20
    .
    Resources Requested:
    =========================================
    mem=2gb
    neednodes=1:ppn=2
    nodes=1:ppn=2
    vmem=4gb
    walltime=00:10:00
    =========================================
    .
    Resouces Used:
    =========================================
    cput=00:00:00
    mem=0kb
    vmem=0kb
    walltime=00:00:14
    =========================================
    .
    Exit Status : [0]
    

Explanation of each section in PBS Output File

• If we look at the first two lines:

  Warning: no access to tty (Bad file descriptor).
  Thus no job control in this shell.
  

  • This is normal. It is not an error, it is simply a warning message.
    It tells you that this is a NON-INTERACTIVE job so you can not input keyboard commands via tty
    • which is:

      no access to tty
      

    • and this means:

      no job control in this shell
      

• Next three lines:

  Working directory is /home/users/demouser/MyFirstExperiment
  Running on host tizard20
  Time is Tue Oct 30 16:49:32 CST 2012
  

  • This corresponds to these lines in the qsub script:

    # This job's working directory
    echo Working directory is $PBS_O_WORKDIR
    cd $PBS_O_WORKDIR
    echo Running on host `hostname`
    echo Time is `date`
    

• The lines:

  tizard20
  tizard20
  

  • This is the output of the mpi program itself.

• The next VERY useful lines - tell you about the resources that were used:
  
  • Resources that were actually used:

    Resouces Used:
    =========================================
    cput=00:00:00
    mem=0kb
    vmem=0kb
    walltime=00:00:14
    =========================================
    

  • Compared to what was requested:

    Resources Requested:
    =========================================
    mem=2gb
    neednodes=1:ppn=2
    nodes=1:ppn=2
    vmem=4gb
    walltime=00:10:00
    =========================================
    

• Here we see that we asked for 10 minutes and only ran for 14 seconds.
• Also the resources used are all zero - this is not because you used nothing; rather, it is due to the program completing before the system could do a count. (Typically your program would run for more than 14 seconds and the Resources Used will be non-zero numbers.)
• You'll need to compare what you asked for and change your next job to match what you actually used. Sometimes you need more than you asked for; this will help you see what a more realistic figure is.
• last line: Exit Code

  Exit Status : [0]
  

  • an Exit Status of [0] equals success; it means ZERO ERRORS.
  • if it is non-zero, you need to check what the exit code means so you can decide how to resolve it.
    • See PBS exit codes

Tutorial using PBS to run a MPI program

The following are some MPI Examples for you try.

• Starting in your home directory copy the whole tree to your current directory:

  cp -ax /opt/shared/training/MPI-EXAMPLES . 
  

• Enter the directory and read the HOW-2:

  cd MPI-EXAMPLES
  more HOW-2-compile-MPI-Programs
  

• This explains you need to do the following:
  1. Load the openmpi modules

     module load intel/11.10.319  openmpi/intel64/1.6.2
     

  2. Change into each directory and compile the programs:
     • C Program 'HelloWorld'

           cd HelloWorld
           mpicc -o helloworld helloworld.c
           cd ..
       

     • C++ Program 'Matrix'

           cd Matrix
           mpicc -o matrix matrix.cc
           cd ..
       

  3. Make a QSUB sumbmisson script for each program in its directory:

      cd HelloWorld
      cp ~/.templates/tizard.sub helloworld.sub
      cd ..
     
      cd Matrix
      cp ~/.templates/tizard.sub matrix.sub  
      cd ..
     

  4. Edit the qsub file as detailed above:

     *For each qsub script, change the resources to four nodes with one cpu on each, 2GB of mem and vmem and a walltime of 10 minutes:

        #PBS -l nodes=4:ppn=1
        #PBS -l mem=2gb,vmem=2gb
        #PBS -l walltime=00:10:00
     

     • Add the following code to the line after : echo Time is `date`

       echo Time is `date`
       echo Using Nodes from PBS_NODEFILE:
       cat $PBS_NODEFILE
       

       (We add this so we can see what NODES PBS gave us to run our job.)

     • For helloworld.sub * Change the 'JobName' and the 'Executable Line'

           ### Job name
           #PBS -N HelloWorld
       
       
           # Run the executable
           mpirun -np 4 ./helloworld_program
       

     • For matrix.sub
       • change the 'JobName' and the 'Executable Line'

         ### Job name
         #PBS -N Matrix
         
         
         # Run the executable
         mpirun -np 4 ./matrix_program
         

  5. Submit the the jobs to the queuing system:
     • 'HelloWorld'

       cd HelloWorld
       qsub helloworld.sub
       cd ..
       

     • Matrix

       cd Matrix
       qsub matrix.sub
       cd ..
       

     • NOTE: there is a precompiled and preconfigured qsub script in each directory if you find things going astray on you.

       -rw-r--r-- 1 demouser demouser 2024 Oct 31 18:17 /opt/shared/training/MPI-EXAMPLES/HOW-2-compile-MPI-Programs
       

       /opt/shared/training/MPI-EXAMPLES/Matrix:
       total 112
       -rw-r--r-- 1 demouser demouser    746 Oct 31 18:15 matrix-tizard.sub
       -rwxrwxr-x 1 demouser demouser 101652 Oct 31 17:53 matrix_program
       -rw-r--r-- 1 demouser demouser   4424 Oct 31 17:12 matrix.cc
       drwxrwxr-x 4 demouser demouser     69 Oct 31 18:17 ..
       drwxrwxr-x 2 demouser demouser     67 Oct 31 18:15 .
       

       /opt/shared/training/MPI-EXAMPLES/HelloWorld:
       total 108
       -rw-r--r-- 1 demouser demouser   755 Oct 31 18:15 helloworld-tizard.sub
       -rwxrwxr-x 1 demouser demouser 99054 Oct 31 17:54 helloworld_program
       -rw-r--r-- 1 demouser demouser   951 Oct 31 17:47 helloworld.c
       drwxrwxr-x 4 demouser demouser    69 Oct 31 18:17 ..
       drwxrwxr-x 2 demouser demouser    78 Oct 31 18:15 .
       

Tutorial output files

Hello World output file

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
Working directory is /home/users/demouser/MyFirstExperiemnt/MPI-EXAMPLES/HelloWorld
Running on host tizard47
Time is Thu Nov 1 11:53:45 CST 2012
Using Nodes from PBS_NODEFILE:
tizard47
tizard47
tizard47
tizard47

Greetings from process 1 on host[tizard47]!
Greetings from process 2 on host[tizard47]!
Greetings from process 3 on host[tizard47]!


Greetings from the MASTER process [0] on  host[tizard47]!



Resources Requested:
=========================================
mem=2gb
neednodes=4:ppn=1
nodes=4:ppn=1
vmem=2gb
walltime=00:10:00
=========================================

Resouces Used:
=========================================
cput=00:00:00
mem=4320kb
vmem=228364kb
walltime=00:00:03
=========================================




Exit Status : [0]

Matrix output file

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
Working directory is /home/users/demouser/MyFirstExperiemnt/MPI-EXAMPLES/Matrix
Running on host tizard47
Time is Thu Nov 1 11:53:26 CST 2012
Using Nodes from PBS_NODEFILE:
tizard47
tizard47
tizard47
tizard47
Preparing the matrix...
The matrix is:
0	0	0
0	0	0
0	0	0
Sending the matrix to the other nodes...
Waiting for responses...
The result is...
Level 0:
0	1	2
0	0	0
0	0	0
Level 1:
0	0	0
0	2	4
0	0	0
Level 2:
0	0	0
0	0	0
0	3	6
Collapsing results...
0	1	2
0	2	4
0	3	6


Resources Requested:
=========================================
mem=2gb
neednodes=4:ppn=1
nodes=4:ppn=1
vmem=2gb
walltime=00:10:00
=========================================

Resouces Used:
=========================================
cput=00:00:00
mem=0kb
vmem=0kb
walltime=00:00:08
=========================================

NEWS CONTENTS

Old News ;-)

Recommended Links

Softpanorama Recommended

Top articles

Sites

PBS Professional - Quick Reference Guide for users

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 20, 2020