Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Job or Queue Reported in Error State E

News  SGE Troubleshooting Recommended Links Client Commands Parallel Environment Gridengine diag tool Queue instance in AU state
Job or Queue Reported in Error State E Monitoring and Controlling Jobs Monitoring Queues Creating and modifying SGE Queues ulimit problem with infiniband Job stuck in the queue problem MPI startup
qstat qacct command qsub -- Submitting Jobs To Queue Instances qhost qmod qping SGE cheat sheet
Installation of SCE on a small set of multicore servers Usage of NFS Installation of the Master Host Installation of the Execution Hosts Perl Admin Tools and Scripts Humor Etc

Unreachable or error in node problem. Not a permissions problem? Well, maybe the nodes or the queues are unreachable. Check with:

qstat -f

or, for even more detail:

qstat -F

If the "state" column in qstat -f  has a big E, that host or queue is in an error state due to... well, something. Sometimes an error just occurs and marks the whole queue as "bad", which blocks all jobs from running in that queue, even though there is nothing otherwise wrong with it.

Use qmod -c <queue list>  to clear the error state for a queue.

[0]root@mysite17: # qmod -c m12a.q m32a.q
root@mysite17 changed state of "m12a.q@mysite52" (no error)
Queue instance "m32a.q@mysite16" is already in the specified state: no error
---sge server: /sge/default/spool/qmaster

 

[1] root@mysite17: # qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
all.q@mysite16                 BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite17                 BIP   0/0/12         0.06     lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite52                 BIP   0/0/12         12.02    lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite53                 BIP   0/0/80         39.72    lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite54                 BIP   0/0/80         0.02     lx24-amd64
---------------------------------------------------------------------------------
all.q@wx3481-ustc              BIP   0/0/8          -NA-     lx24-amd64    au
---------------------------------------------------------------------------------
c12.q@mysite52                 BIP   0/0/12         12.02    lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite16                 BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite53                 BIP   0/0/32         39.72    lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite54                 BIP   0/0/32         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
c40.q@mysite53                 BIP   0/0/40         39.72    lx24-amd64
---------------------------------------------------------------------------------
c40.q@mysite54                 BIP   0/0/40         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
m12a.q@mysite52                BIP   0/12/12        12.02    lx24-amd64
---------------------------------------------------------------------------------
m32a.q@mysite16                BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
m40a.q@mysite54                BIP   0/0/40         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
m40b.q@mysite53                BIP   0/40/40        39.72    lx24-amd64
 
# qmod -c c40.q c32.q m40a.q
root@mysite17 changed state of "c40.q@mysite54" (no error)
Queue instance "c40.q@mysite53" is already in the specified state: no error
Queue instance "c32.q@mysite16" is already in the specified state: no error
root@mysite17 changed state of "c32.q@mysite54" (no error)
Queue instance "c32.q@mysite53" is already in the specified state: no error
root@mysite17 changed state of "m40a.q@mysite54" (no error)
		

Maybe that's not the problem, though. Maybe there is some network problem preventing the SGE master from communicating with the exec hosts, such as routing problems or a firewall misconfiguration.

You can troubleshoot these things with qping, which will test whether the SGE processes on the master node and the exec nodes can communicate.

N.B.: remember, the execd  process on the exec node is responsible for establishing a TCP/IP connection to the qmaster  process on the master node, not the other way around. The execd  processes basically "phone home". So you have to run qping  from the exec nodes, not the master node!

Syntax example (I am running this on a exec node, and sheridan is the SGE master):

qping sheridan 536 qmaster 1

where 536  is the port that qmaster  is listening on, and 1  simply means that I am trying to reach a daemon. Can't reach it? Make sure your firewall has a hole on that port, that the routing is correct, that you can ping using the good old ping  command, that the qmaster  process is actually up, and so on.

Of course, you could ping the exec nodes from the master node, too, e.g. I can see if I can reach exec node kosh like this:

 $ qping kosh 537 execd 1

but why would you do such a crazy thing? execd  is responsible for reaching qmaster, not the other way around.

If the above checks out, check the messages log in /var/log/sge_messages on the submit and/or master node (on our Babylon Cluster, they're both the node sheridan ):

 $ tail /var/log/sge_messages

Personally, I like running:

 $ tail -f /var/log/sge_messages

before I submit the job, and then submit a job in a different window. The -f  option will update the tail of the file as it grows, so you can see the message log change "live" as your job executes and see what's happening as things take place.

(Note that the above is actually a symbolic link I put in to the messages log in the qmaster  spool directory, i.e. /opt/sge/default/spool/qmaster/messages .)

One thing that commonly goes wrong is permissions. Make sure that the user that submitted the job using qsub  actually has the permissions to write error, output, and other files to the paths you specified.

For even more precise troubleshooting... maybe the problem is unique only to some nodes(s) or some queue(s)? To pin it down, try to run the job only on some specific node or queue:

 $ qsub -l hostname=<node/host name> <other job params>  
 $ qsub -l qname=<queue name> <other job params> 

Maybe you should also try to SSH into the problem nodes directly and run the job locally from there, as your own user, and see if you can get any more detail on why it fails.

If all else fails...

Sometimes, the SGE master host will become so FUBARed that we have to resort to brute, traumatizing force to fix it. The following solution is equivalent to fixing a wristwatch with a bulldozer, but seems to cause more good than harm (although I can't guarantee that it doesn't cause long-term harm in favor of a short-term solution).

Basically, you wipe the database that keeps track of SGE jobs on the master host, taking any problem "stuck" jobs with it. (At least that's what I think  this does...)

I've found this useful when:

The solution:

ssh sheridan
su -
service sgemaster stop
cd /opt/sge/default/
mv spooldb spooldb.fubared
mkdir spooldb
cp spooldb.fubared/sge spooldb/
chown -R sgeadmin:sgeadmin spooldb
service sgemaster start

Wipe spooldb.fubared when you are confident that you won't need its contents again.

 

Job or Queue Reported in Error State E (N1 Grid Engine 6 User's Guide)

N1 Grid Engine 6 User's Guide

Job or queue errors are indicated by an uppercase E in the qstat output.

A job enters the error state when the grid engine system tries to run a job but fails for a reason that is specific to the job.

A queue enters the error state when the grid engine system tries to run a job but fails for a reason that is specific to the queue.

The grid engine system offers a set of possibilities for users and administrators to gather diagnosis information in case of job execution errors. Both the queue and the job error states result from a failed job execution. Therefore the diagnosis possibilities are applicable to both types of error states.



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: October 11, 2015