Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Monitoring and Controlling Jobs

News Commands SGE Troubleshooting Troubleshooting Steps Using qacct command for job that failed
qstat Job or Queue Reported in Error State E Resource requirements and limitations SGE Submit Scripts Slot hacking
Unreachable or error in node problem. Restricting number of slots per server qacct  qhost qmod
Monitoring Queues qping  SGE Array Jobs Humor Etc

Introduction

After submitting your job to Grid Engine you may track its status by the number or name. Both can be determined from the output of the qstat command. Sometimes job is stuck in queue, typically with State E or State AU

Reasons can be connected with the node (full filesystem, not running execd, not mounted NFS partition, etc) or with SGE.

You can get general classification of stuck jobs using command qstat -j

qstat -j
invalid error number
        289,    290,    291,    292,    293,    294,    295,    296,
        207

invalid error number
        289,    290,    291,    292,    293,    294,    295,    296,
        207,    305,    306,    307,    308,    309

Jobs can not run because queue instance is not contained in its hard queue list
        305,    306,    307,    308,    309

invalid error number
        305,    306,    307,    308,    309

Listing  jobs that use particular parallel environment

qstat  -pe  ms -u "*"

For example, Accelrys forms job name by prefixing MS_ to the name of the subdirectory in jobs directory

If job is no longer in the queue, the output of the command

qacct -j MS_YBJ2I 

were MS_YBJ2I is Accelrys Material Studio name for the job gives info why job failed. 

 

Location of submitted jobs

Jobs for each server can be found at directory

$SGE_HOME/default/spool/ node_name/job_scripts
For example:
[0]# ls
301
---sge server: /sge/default/spool/mysite52/job_scripts
[0]root@mysite17: # ll 301

-rwxr-xr-x 1 root root 1442 Jul 10 13:00 301*

---sge server: /sge/default/spool/mysite52/job_scripts

[0]root@mysite17: # cat 301
#!/bin/sh -f
# This file is submitted to the queuing system
# The command for the job submission:
# qsub -q mysite17 -S /bin/bash -V -N MS_8WX9Z -cwd  -pe ms 10    ./sge.sh
# Appropriate environment should be set before the job submission

 if [ -f /sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../share/license/data/lic_setup.sh ]; then
          eval `/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../share/license/data/lic_setup.sh /sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../ -s sh`
 fi
 if [ -f /sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../share/bin/ms_setup.sh ]; then
          eval `/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../share/bin/ms_setup.sh /sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../ -s sh`
 fi

DSD_QUEUED_RUN=1
export DSD_QUEUED_RUN
cd "/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/jobs/8WX9Z"
awk '{print $1":"$2}' $PE_HOSTFILE  > machines.LINUX

DSD_MachineList="/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/jobs/8WX9Z/machines.LINUX"
export DSD_MachineList
DSD_NumProc=10
export DSD_NumProc

/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../gwperl "/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/servers/MatServer.pl" -standalone -fromdsd fromdsd.txt -todsd todsd.txt > stdout.txt
---sge server: /sge/default/spool/mysite52/job_scripts
		

Troubleshooting Steps

  1. Get the list of jobs and locate problematic job in it:
    qstat -u <name of the user with problem jobs> 

    or

    qstat -u "*"
  2. Determine whether your job show "Eqw" or "qw" state when you run qstat, and just sits there refusing to run?  If status is "E" see Unreachable or error in node problem

State E

If the "state" column in qstat -f  has a big E, that host or queue is in an error state due to... well, something. Sometimes an error just occurs and marks the whole queue as "bad", which blocks all jobs from running in that queue, even though there is nothing otherwise wrong with it.

State AU

au - means that communication with node is broken. possible that execd daemon is not running.  in this case you have -NA- in load average: 

all.q@wx3481              BIP   0/0/8          -NA-     lx24-amd64    au

Maybe there is some network problem preventing the SGE master from communicating with the exec host, such as routing problems or a firewall misconfiguration.  Once I have the case when there was a mismatch of hostnames

You can troubleshoot these things with qping, which will test whether the SGE processes on the master node and the exec nodes can communicate.

N.B.: remember, the execd  process on the exec node is responsible for establishing a TCP/IP connection to the qmaster  process on the master node, not the other way around. The execd  processes basically "phones home". So you have to run qping  from the exec nodes, not the master node!

Syntax example (I am running this on a exec node, and sheridan is the SGE master):

qping sheridan 6444 qmaster 1

where 6444  is the port that qmaster  is listening on, and 1  simply means that I am trying to reach a daemon. Can't reach it? Make sure your firewall has a hole on that port, that the routing is correct, that you can ping using the good old ping  command, that the qmaster  process is actually up, and so on.

Of course, you could ping the exec nodes from the master node, too, e.g. I can see if I can reach exec node kosh like this:

 $ qping kosh 537 execd 1

but why would you do such a crazy thing? execd  is responsible for reaching qmaster, not the other way around.

If the above checks does not help, check the messages log in /var/log/sge_messages on the submit and/or master node:

 $ tail /var/log/sge_messages

Personally, I like running:

 $ tail -f /var/log/sge_messages

before I submit the job, and then submit a job in a different window. The -f  option will update the tail of the file as it grows, so you can see the message log change "live" as your job executes and see what's happening as things take place.

(Note that the above is actually a symbolic link I put in to the messages log in the qmaster  spool directory, i.e. /opt/sge/default/spool/qmaster/messages .)

One thing that commonly goes wrong is permissions. Make sure that the user that submitted the job using qsub  actually has the permissions to write error, output, and other files to the paths you specified.

Submitting test job for the node

See qsub -- Submitting Jobs To Queue Instances

maybe the problem is unique only to some nodes(s) or some queue(s)? To pin it down, try to run the job only on some specific node or queue:

 $ qsub -i hostname=<node/host name> <command> -- <command params>  
 $ qsub -l qname=<queue name> <other job params> 

Maybe you should also try to SSH into the problem nodes directly and run the job locally from there, as your own user, and see if you can get any more detail on why it fails.

Does your job actually get dispatched and run

(that is, qstat no longer shows it -- because it was sent to an exec host, ran, and exited), but something else isn't working right?

NOTE: You can also use:

$ qacct -j 

But in this case you should pipe that to something like tail -n <number of lines> because you only want the end of the massive log that it will spit out.

Alternately, you can use:

$ qacct -j <job number> 

There are many other useful options, such as getting all jobs by a specific user, etc. See man qacct  for them.

If the job is still running, use qstat  instead of qacct  above to get the same info.

 

Cleaning stuck queue

NOTE: Use qmod -c <queue list>  to clear the error state for a queue.

[0]root@mysite17: # qmod -c m12a.q m32a.q
root@mysite17 changed state of "m12a.q@mysite52" (no error)
Queue instance "m32a.q@mysite16" is already in the specified state: no error
---sge server: /sge/default/spool/qmaster
[1] root@mysite17: # qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
all.q@mysite16                 BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite17                 BIP   0/0/12         0.06     lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite52                 BIP   0/0/12         12.02    lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite53                 BIP   0/0/80         39.72    lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite54                 BIP   0/0/80         0.02     lx24-amd64
---------------------------------------------------------------------------------
all.q@wx3481-ustc              BIP   0/0/8          -NA-     lx24-amd64    au
---------------------------------------------------------------------------------
c12.q@mysite52                 BIP   0/0/12         12.02    lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite16                 BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite53                 BIP   0/0/32         39.72    lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite54                 BIP   0/0/32         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
c40.q@mysite53                 BIP   0/0/40         39.72    lx24-amd64
---------------------------------------------------------------------------------
c40.q@mysite54                 BIP   0/0/40         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
m12a.q@mysite52                BIP   0/12/12        12.02    lx24-amd64
---------------------------------------------------------------------------------
m32a.q@mysite16                BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
m40a.q@mysite54                BIP   0/0/40         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
m40b.q@mysite53                BIP   0/40/40        39.72    lx24-amd64
 qmod -c command in cost cases can clean the state (but not if the node is unreachable -- au):
qmod -c c40.q c32.q m40a.q
root@mysite17 changed state of "c40.q@mysite54" (no error)
Queue instance "c40.q@mysite53" is already in the specified state: no error
Queue instance "c32.q@mysite16" is already in the specified state: no error
root@mysite17 changed state of "c32.q@mysite54" (no error)
Queue instance "c32.q@mysite53" is already in the specified state: no error
root@mysite17 changed state of "m40a.q@mysite54" (no error)

 

If all else fails...

Sometimes, the SGE master host will become so FUBARed that we have to resort to brute, traumatizing force to fix it. The following solution is equivalent to fixing a wristwatch with a bulldozer, but seems to cause more good than harm (although I can't guarantee that it doesn't cause long-term harm in favor of a short-term solution).

Basically, you wipe the database that keeps track of SGE jobs on the master host, taking any problem "stuck" jobs with it. (At least that's what I think  this does...)

I've found this useful when:

The solution:

ssh sheridan
su -
service sgemaster stop
cd /opt/sge/default/
mv spooldb spooldb.fubared
mkdir spooldb
cp spooldb.fubared/sge spooldb/
chown -R sgeadmin:sgeadmin spooldb
service sgemaster start

Wipe spooldb.fubared when you are confident that you won't need its contents again.


Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Oracle

Ubuntu Manpage qstat - show the status of Sun Grid Engine jobs and queues

sge-tutorial - The Center for Computational Biology and Bioinformatics - UT Austin Wikis

W. M. Keck Laboratory for Integrated Biology II



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019