PBSpro submission scripts examples

Writing a submission script is typically the most convenient way to execute job on the cluster  but jobs can also be submitted on the command line.

A submission script (also often called a job script) consists of 3 parts:

1. Shell specification (PBSpro always uses specified interpreter be is Bash, Perl or Python)
2. PBSpro directives (preudocomments that inform PBS how to execure this particular jobs and what resouces it needs)
3. Actual script executed on target node to which scheduler allocate this particular job

Here is a very basic submission script example:

#!/bin/bash
#PBS -l mem=1gb,nodes=1:ppn=1,walltime=1:00:00
#PBS -m abe -M your-email-address

module add python3

WORK_DIR=~/tmp/$USER\_workdir            # define your work directory
mkdir  -p $WORK_DIR/DATA                 # make the directory for data
cp ~/DATA $WORK_DIR/DATA                 # copy data to the host on this the job will be executed 
~/bin/my_script.py ~/DATA $WORK_DIR/DATA # execute my Python pipeline
rsync -a $WORK_DIR/DATA ~                # copy changed files back to your home directory 
rm -r $WORK_DIR                          # clean the temp data folder after the execution finished

NOTE: Make sure there is a blank line at the end of your PBSpro submission script

This file consists of three parts.

1.  The shell to use.  In this case

 #!/bin/bash

means that we use the bash interpreter for out submission script. You can use any other scripting language, but bash is the most popular for this particular purpose,  and you can get better help in case of problems  if you use bash. 

2.  Two line of preudocomments with PBS parameters which specify wall-time, number of cores, memory and more.  In order for PBS to recognize them as directives they should start with #PBS. The command

#PBS -l mem=10gb,nodes=1:ppn=1,walltime=8:00:00

requests 10 gigabytes of memory, 1 core, and 8 hours of wall clock time (wall clock time is the period from the moment the job starts to the moment when it completes).

Units to specify memory include kb (kilobytes), mb (megabytes) and gb (gigabytes).  The unit must be an integer.

The "nodes=1:ppn=1" requests one core.  This means you requesting one node and one core per node.  The ppn notation is historic and means processors per node. It is from a time when every processor had one core.  You can also request multiple cores. for example, ppn=4 requests four cores . Please note that even if your job is single threaded you can use multithreaded archivers (pigz, pbzip2) to unpack and at the end pack your data.

The directive #PBS -m abe -M your-email-address is optional.  It specifies email notification options.

You can use any combination of a,b, and e.  The meaning is as follows:

• a  mail is sent when the job is aborted by the batch system.
• b  mail is sent when the job begins execution.
• e  mail is sent when the job terminates.

There can be as many preudocomments as you need, not just two as in this example. 

3.  The third part of the submission script consists of batch commands that you want to be executed on the target host -- the host on which scheduler will allocate your jobs.

Typically, the the first command in this part loads the necessary environment modules. After it you position the actual commands to prepare your data (for example, copy them to /tmp or unpack them into /tmp) and them the command that executes the program you want to run. After execution finish you need command that put results back into your home folder and clean temporary folder, if any. 

NOTES:

• PBS starts executing your submission script in your home directory.  You can add a cd command to take you to the directory where your data reside in necessary, copy he data to temp and them launch your program so that it will read the data to /tmp the commands.
• For bioinformatics jobs you can unpack (for example using pbzip2 ) your data files into /tmp instead of copyting them. that will be faster.  It does not make sense you run your jobs with your large data files being on a shared filesystem -- you can kill GPFS performance this ways quite easily.
• You you use small amount of space for your data and temp results you can use /dev/shm as your temporary folder instead of /tmp. In many case, especially for bioinformatics jobs that dramatically speed up the execution of your pipeline. 
• Make sure the last line of the pbs file has a carriage return in it (Be sure it has a blank line at the end)

Requesting Specific Nodes

Some software is licensed only on specific nodes.  To request nodes in which commercial program, for example amail,  is licensed you can specify

#PBS -l mem=1gb,nodes=aimall+1:ppn=1,walltime=1:00:00

You may also request the type of node.  If you don't request a type of node, your job will run on a node with  any nuumber of avalable cores.  To see the type of nodes, type

 pbshosts

The "properties" column shows the types of nodes that you can request

To specify  4 HPintel8 nodes using 16 cores per node with 32 GB of total memory:

#PBS -l mem=32g,nodes=HPintel8+4:ppn=8,walltime=1:00:00

Parallel Jobs

To request more than one core, you can request nodes and cores per node (recommended) or just request the total number of cores

a) To request 4 cores, on a single node:

#!/bin/bash
#PBS -l pmem=4gb,nodes=1:ppn=4,walltime=1:00:00
#PBS -m abe -M your-email-address

b) To request 8 cores, on two nodes:

#!/bin/bash
#PBS -l pmem=1gb,nodes=2:ppn=4,walltime=1:00:00
#PBS -m abe -M your-email-address

Notes:

1. In this example, we used pmem, not mem.  This request memory per process, you can also use mem, which is total memory requested.
2. It is best to specify either intel8 or HPintel8 when running multiprocessor jobs on multiple nodes.  This will give you nodes with the same processor speed.

Optional Job Control Parameters

The following parameters are optional.  If you don't specify them, torque will use default names.

 a) To name your job, add:

#PBS -N your-choice-of-job-name

b) To specify the name of the output file (where the standard output is sent), add:

#PBS -o output-file-name

This will send the output file to the output-file-name in the directory from which the job was submitted.  If you want to the output file to go to another directory,  use the complete path to the file.

#PBS -o complete-path/output-file-name

c) To specify the name of the error file. You can use just a file name, or give a complete path to the file where you want the standard error as for the output file.

#PBS -e error-file-name

Environment Variables

You can use the environment variable PBS_JOBID  in your script.  For example

#PBS -o output.$PBS_JOBID

will send the output to a file named output.Job-Number.nas-1-0.local.

Other useful commands

To delete a job:

qdel job-number

To see the status of the queues, use the command

showq

To check the status of your job, use the command

checkjob -v job-number

To see the number of cores per node, and the number of free cores,

pbshosts

or to see information about only nodes with free cores

pbshosts -f

NEWS CONTENTS

• 210608 : PBS Pro job script examples Computational Information Systems Laboratory ( cisl.ucar.edu )
• 210608 : For tcsh users ( )

Old News ;-)

PBS Pro job script examples Computational Information Systems Laboratory

cisl.ucar.edu

When using any of these examples, remember to substitute your own job name and project code, and customize other directives and commands as necessary. If you need to convert an LSF script that you used on Yellowstone to work in the PBS scheduler, you can use the comparison chart below as a guide.

These examples assume that you are using SGI's Message Passing Toolkit (MPT) MPI library and that the programs being run were compiled with Intel 16.0.3 or a later version. You will have loaded the MPT and Intel modules into your module environment before you submit the batch jobs shown on this page. Contact [email protected] for assistance with adapting them for other cases.

The examples also assume that you are not using the Cheyenne "share" queue. That queue requires setting a special environment variable and a special invocation of mpirun as explained here.

When your script is ready, submit your batch job for scheduling as shown here.

---

Batch script to run an MPI job

For tcsh users

#!/bin/tcsh
### Job Name
#PBS -N mpi_job
### Project code
#PBS -A project_code
#PBS -l walltime=00:01:00
#PBS -q queue_name
### Merge output and error files
#PBS -j oe
### Select 2 nodes with 36 CPUs each for a total of 72 MPI processes
#PBS -l select=2:ncpus=36:mpiprocs=36
### Send email on abort, begin and end
#PBS -m abe
### Specify mail recipient
#PBS -M email_address

### Run the executable
mpiexec_mpt dplace -s 1 ./executable_name.exe

For bash users

#!/bin/bash
### Job Name
#PBS -N mpi_job
### Project code
#PBS -A project_code
#PBS -l walltime=00:01:00
#PBS -q queue_name
### Merge output and error files
#PBS -j oe
### Select 2 nodes with 36 CPUs each for a total of 72 MPI processes
#PBS -l select=2:ncpus=36:mpiprocs=36
### Send email on abort, begin and end
#PBS -m abe
### Specify mail recipient
#PBS -M email_address

### Run the executable
mpiexec_mpt dplace -s 1 ./executable_name.exe

---

Batch script to run a pure OpenMP job

To run a pure OpenMP job, specify the number of CPUs you want from the node, which will provide you with that number of threads. You will still be charged for 36 CPUs in exclusive queues.

For tcsh users

#!/bin/tcsh
#PBS -A project_code
#PBS -N OpenMP_job
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00
### Request 10 CPUS for 10 threads
#PBS -l select=1:ncpus=10:ompthreads=10

### Run OpenMP program with thread pinning using omplace
omplace ./executable_name

For bash users

#!/bin/bash
#PBS -A project_code
#PBS -N OpenMP_job
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00
### Request 10 CPUS for 10 threads
#PBS -l select=1:ncpus=10:ompthreads=10

### Run OpenMP program with thread pinning using omplace
omplace ./executable_name

---

Batch script to run a hybrid MPI/OpenMP job

If you want to run a hybrid MPI/OpenMP configuration where each node uses threaded parallelism while the nodes communicate with each other using MPI, activate NUMA mode and run using the MPI launcher.

For tcsh users

#!/bin/tcsh
#PBS -A project_code
#PBS -N hybrid_node
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00
### Request two nodes, each with one MPI task and 36 threads
#PBS -l select=2:ncpus=36:mpiprocs=1:ompthreads=36
 
### Run the hybrid OpenMP/MPI program
mpiexec_mpt omplace ./executable_name

For bash users

#!/bin/bash
#PBS -A project_code
#PBS -N hybrid_node
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00
### Request two nodes, each with one MPI task and 36 threads
#PBS -l select=2:ncpus=36:mpiprocs=1:ompthreads=36
 
### Run the hybrid OpenMP/MPI program
mpiexec_mpt omplace ./executable_name

---

Batch script to run a command file (MPMD) job

Multiple Program, Multiple Data (MPMD) jobs run multiple independent, sequential executables simultaneously. The executable commands appear in the command file (cmdfile) on separate lines. The command file, the executable files, and the input files should reside in the directory from which the job is submitted. If they don't, you need to specify adequate relative or full pathnames in both your command file and job scripts.

The command file used in the example job scripts has these four lines.

./cmd1.exe < input1 > output1
./cmd2.exe < input2 > output2
./cmd3.exe < input3 > output3
./cmd4.exe < input4 > output4

The job will produce output files that reside in the directory in which the job was submitted.

In place of executables, you can specify independent shell scripts, MATLAB scripts, or others, or you can mix and match executables with scripts. Each task should execute in about the same wall-clock time as the others.

If any of your command file lines invoke a utility such as IDL, MATLAB, NCL, R and so on, invoke it in batch mode rather than interactive mode or your job will hang until it reaches the specified walltime limit. See the user guide for the utility for how to invoke it in batch mode.

For tcsh users

#!/bin/tcsh
#PBS -A project_code
#PBS -N cmd_file
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00
### Request one chunk with ncpus and mpiprocs set to
### the number of lines in the command file
#PBS -l select=1:ncpus=4:mpiprocs=4

setenv MPI_SHEPHERD true

mpiexec_mpt launch_cf.sh cmdfile

For bash users

#!/bin/bash
#PBS -A project_code
#PBS -N cmd_file
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00
### Request one chunk with ncpus and mpiprocs set to
### the number of lines in the command file
#PBS -l select=1:ncpus=4:mpiprocs=4

export MPI_SHEPHERD=true

mpiexec_mpt launch_cf.sh cmdfile

---

Pinning tasks/threads to CPUs

The omplace wrapper script pins processes or threads to specific CPUs. This is particularly useful if you want to use threads to parallelize at the socket level (using 18 CPUs per socket, two sockets per node, for example), which can improve performance if your program depends strongly on memory locality.

For purposes of many Cheyenne users, using omplace as shown below is sufficient to ensure that processes do not migrate among CPUs and adversely affect performance. To learn about using omplace and dplace for more precise control of process placement, see Using omplace and dplace.

For tcsh users

#!/bin/tcsh
#PBS -A project_code
#PBS -N hybrid_socket
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00
### Request two nodes, each with two MPI tasks and 18 threads per task
#PBS -l select=2:ncpus=36:mpiprocs=2:ompthreads=18
 
### Run hybrid program with explicit task/thread placement using omplace
### to ensure that each group of 18 threads runs on a separate socket
mpiexec_mpt omplace ./executable_name

For bash users

#!/bin/bash
#PBS -A project_code
#PBS -N hybrid_socket
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00
### Request two nodes, each with two MPI tasks and 18 threads per task
#PBS -l select=2:ncpus=36:mpiprocs=2:ompthreads=18
 
### Run hybrid program with explicit task/thread placement using omplace
### to ensure that each group of 18 threads runs on a separate socket
mpiexec_mpt omplace ./executable_name

---

Dependent jobs

It is possible to schedule jobs to run based on the status of other jobs. For example, you might schedule a preprocessing job to run; start a computation job when the preprocessing job is complete; then start a post-processing job when the computation is done.

To schedule such a series or chain of jobs, use qsub -W [job-dependency-expression] to specify the job dependencies you need.

Example 1 below shows how to do this by submitting each job individually.
Example 2 is more automated.

---

Dependent jobs - example 1

Let's say you have you have three jobs to run:

1. pre.pbs: a preprocessing job
2. main.pbs: a computation job
3. post.pbs: a post-processing job

The main job can be run only when the preprocessing job finishes, and the post-processing job can be run only when the computation job finishes.

To schedule the jobs, follow this process:

• Submit the preprocessing job script first:

qsub pre.pbs

Note the job number that is assigned; you will need it for the next step. We'll use 895101 in this example.

• Submit the main job, using the -W depend=afterok:[jobID] option to indicate that it should start when the preprocessing job is done.

qsub -W depend=afterok:895101.chadmin1 main.pbs

Again, note the job ID for the next step.

• Submit the post-processing job, setting the final job dependency with the -W depend=afterok:[jobID] option to indicate that it should start when the main job is done.

qsub -W depend=afterok:895102.chadmin1 post.pbs

To monitor the jobs you submitted, use the qstat command.

qstat -u $USER

To determine why an individual job is pending, enter qstat followed by the job ID. Such a job typically would be pending because the job on which it depends has not met the required conditions.

Dependent jobs - example 2

This script gets the PBS jobID when the job is submitted, then uses it to execute a qsub command with the dependency expression to start a second job when the first one is done.

#!/bin/bash
#PBS -A project_code
#PBS -N job1
#PBS -j oe
#PBS -m abe
#PBS -M email_address
#PBS -q queue_name
#PBS -l walltime=01:00

### Request 2 nodes, each with two MPI tasks and 18 tasks per node
#PBS -l select=2:ncpus=36:mpiprocs=2

### Run job1
./job1.exe

### Run job 2 if dependency is satisfied
qsub -W depend=afterok:$PBS_JOBID main.pbs

Comparing basic PBS and LSF MPI scripts

PBS Pro	Platform LSF
#!/bin/tcsh	#!/bin/tcsh
#PBS -N job_name	#BSUB -J job_name
#PBS -A project_code	#BSUB -P project_code
#PBS -l walltime=00:05:00	#BSUB -W 00:05
#PBS -l select=2:ncpus=32:mpiprocs=32	#BSUB -n 64 #BSUB -R "span[ptile=16]"
#PBS -j oe	#BSUB -o job_name.%J.out #BSUB -e job_name.%J.err
#PBS -q queue_name	#BSUB -q queue_name
mpiexec_mpt -n 64 ./hw_mpi.exe	mpirun.lsf ./myjob.exe

Recommended Links

Softpanorama Recommended

Top articles

Sites

Top articles

Sites

Useful PBS Commands ( usc.edu )

TORQUE:

• The Official Users Guide (link is external)
• Adaptive's documentation

PBSpro

• [PDF] PBS Professional® 11.1 user's guide pbsworks.com