ulimit problem with infiniband in SGE

In SGE sge_conf defines the global and local Sun Grid Engine configurations and can be shown/modified by qconf(1) using the -sconf/-mconf options.

qconf -mconf

One important option in this respect is  execd_params  using which you can fix pretty nasty problem with SGE behavior in case Intfiniband is present

Note: Univa grid engine still has a bug and does not accept the work unlimited.

execd_params H_MEMORYLOCKED=32M  should be used. See Prentice Bisbal email

Problem

One of the issues with OpenMPI and SGE is that the maximum locked memory (on linux) is set way too low for Infiniband, and it can't lock enough memory. You can "fix" this with settings in /etc/security/limits.conf, simply add these two lines to the file

* soft memlock unlimited
* hard memlock unlimited 

However, it appears that this works for running OpenMPI over Infiniband apps by hand, but not through SGE.

qrsh ulimit -l

would always return 32 (kilobytes), and the Infiniband based job wouldn't run.

Solution

Subject:Setting memlock limit with SGE 6.2; was: Re: [GE users] worth a wiki entry for SGE with OpenMPI and Infiniband

From: Andy Schwierskott (andy...@sun.com)

Date: Jul 21, 2008 1:21:32 am

List: net.sunsource.gridengine.users

Hi,

just on a side node regarding the 'memlock' resource limit issue which are reported here sometimes: For SGE 6.2 (it's not part of the Beta and Beta refresh however) we added as a last minute feature the ability to configure the memlock limit and a few others on the execd level, i.e. via the 'execd_params' cluster config setting.

Full background: in the SGE queue config you can configure most but not all Unix resource limits like CPU time, max. virtual memory and so on. There are a few others like the maximum file descriptor limit which exist on virtually all OS'es and some which just exit on one or a few OS'es (like the "memlock" limit). It was too late to extend the queue configuration and we found the workaround to configure these limits indirectly by hacking the SGE execd startup scripts (this is the chain how the job inherits such limits if they are not set) to implicit and error prone, therefore we decided to enable an admin to set such limits via the execd_params setting.

It's not a 100% perfect solution: it's a execd setting valid for all jobs running in all queues on that host and it does not provide a solution to use the configured system wide limits set e.g. in /etc/security/limits.conf on Linux. Nevertheless it's much better than requiring to edit the job scripts or the execd startup scripts which could get overwritten with an update and would not work if for testing purposes the execd is started directly e.g. in debug mode.

For the interested reader here's an excerpt from the SGE 6.2 sge_conf(5) man page which describes the syntax and semantic of these settings:

S_DESCRIPTORS, H_DESCRIPTORS, S_MAXPROC, H_MAXPROC, S_MEMORYLOCKED, H_MEMORYLOCKED, S_LOCKS, H_LOCKS Specifies soft and hard resource limits as implemented by the setrlimit(2) system call. See this manual page on your system for more information. These parameters complete the list of limits set by the RESOURCE LIMITS parameter of the queue configuration as described in queue_conf(5). Unlike the resource limits in the queue configuration, these resource limits are set for every job on this execution host. If a value is not speci- fied, the resource limit is inherited from the execu- tion daemon process. Because this would lead to unpredicted results, if only one limit of a resource is set (soft or hard), the corresponding other limit is set to the same value. S_DESCRIPTORS and H_DESCRIPTORS specify a value one greater than the maximum file descriptor number that can be opened by any process of a job. S_MAXPROC and H_MAXPROC specify the maximum number of processes that can be created by the job user on this execution host S_MEMORYLOCKED and H_MEMORYLOCKED specify the maximum number of bytes of virtual memory that may be locked into RAM. S_LOCKS and H_LOCKS specify the maximum number of file locks any process of a job may establish. All of these values can be specified using the multi- plier letters k, K, m, M, g and G, see sge_types(1) for details.

So you would simply set

execd_params H_MEMORYLOCKED=unlimited
										

to set the soft and hard Linux "memlock" limit to unlimited.

Another workaround in my opinion is to set those limits in the execd startup script. At least this eliminates a different behavior if the execd is started at system boot time or later by an interactively logged in root user. As stated above care has to be taken when the execd startup script is changed, a new execd is installed or the execd is started directly without using the startup script.

Notes:

Use the command

qconf -mconf

In Sun Grid Engine and Oracl Grid Engine you would simply set  line 27 to

execd_params H_MEMORYLOCKED=unlimited

  1 execd_spool_dir              /sge/default/spool
  2 mailer                       /bin/mail
  3 xterm                        /usr/bin/X11/xterm
  4 load_sensor                  none
  5 prolog                       none
  6 epilog                       none
  7 shell_start_mode             posix_compliant
  8 login_shells                 sh,ksh,csh,tcsh
  9 min_uid                      0
 10 min_gid                      0
 11 user_lists                   none
 12 xuser_lists                  none
 13 projects                     none
 14 xprojects                    none
 15 enforce_project              false
 16 enforce_user                 auto
 17 load_report_time             00:00:40
 18 max_unheard                  00:05:00
 19 reschedule_unknown           00:00:00
 20 loglevel                     log_warning
 21 administrator_mail           none
 22 set_token_cmd                none
 23 pag_cmd                      none
 24 token_extend_time            none
 25 shepherd_cmd                 none
 26 qmaster_params               none
 27 execd_params                 H_MEMORYLOCKED=unlimited
 28 reporting_params             accounting=true reporting=false \
 29                              flush_time=00:00:15 joblog=false sharelog=00:00:00
 30 finished_jobs                100
 31 gid_range                    20000-20100
 32 qlogin_command               builtin
 33 qlogin_daemon                builtin
 34 rlogin_command               builtin
 35 rlogin_daemon                builtin
 36 rsh_command                  builtin
 37 rsh_daemon                   builtin
 38 max_aj_instances             2000
 39 max_aj_tasks                 75000
 40 max_u_jobs                   0
 41 max_jobs                     0
 42 max_advance_reservations     0
 43 auto_user_oticket            0

[Dec 3, 2008] Re [GE users] OpenMPI RLIMIT_MEMLOCK problem - Prentice Bisbal - net.sunsource.gridengine.users - MarkMail

5 messages in net.sunsource.gridengine.usersRe: [GE users] OpenMPI RLIMIT_MEMLOCK...

From	Sent On	Attachments
Prentice Bisbal	Dec 3, 2008 12:54 pm
Prentice Bisbal	Dec 3, 2008 2:06 pm
Prentice Bisbal	Dec 3, 2008 2:49 pm
pollinger	Dec 4, 2008 1:57 am
Prentice Bisbal	Dec 4, 2008 6:34 am

Subject:	Re: [GE users] OpenMPI RLIMIT_MEMLOCK problem
From:	Prentice Bisbal (pren...@ias.edu)
Date:	Dec 3, 2008 2:49:30 pm
List:	net.sunsource.gridengine.users

I think I found a bug in 6.2. When I set

execd_params H_MEMORYLOCKED=unlimited

and restart sge_execd on all my nodes, I get this:

$ qrsh ulimit -l 4

When I set

execd_params H_MEMORYLOCKED=UNLIMITED,

I get this

$ qrsh ulimit -l 5

If set H_MEMORYLOCKED to a numerical value, like this

execd_params H_MEMORYLOCKED=32M

I get the correct result:

qrsh ulimit -l 32768

Looks like the unlimited keyword is not defined properly in the code somewhere. Also, I have to restart sge_execd to get these changes to take effect. Is that the proper behavior? The documentation I linked to in my original post didn't explicitly say anything about restarting sge_execd after tinkering with these settings.

-- Prentice

Prentice Bisbal wrote:

I posted this earlier, but it never showed up on the list. Probably b/c I unsubscribed before SC08 and forget to resubscribe. If if the first attempt shows up, I apologize for posting twice.

I'm using SGE 6.2 with OpenMPI 1.2.8.

I just setup OpenMPI tight integration following the instructions here:

http://www.open-mpi.​org/faq/?category=ru​nning#run-n1ge-or-sg​e

I then defined my exec_params to set H_MEMLOCK=unlimited as described here:

http://gridengine.su​nsource.net/ds/viewM​essage.do?dsForumId=​38​ &dsMessageId=72405

Unfortunately, when I submit an MPI job, I still get MEMLOCK errors:

libibverbs: Warning: RLIMIT_MEMLOCK is 11162 bytes. This will severely limit memory registrations. libibverbs: Warning: RLIMIT_MEMLOCK is 11199 bytes. This will severely limit memory registrations. libibverbs: Warning: RLIMIT_MEMLOCK is 11181 bytes. This will severely limit memory registrations.

To make sure my parameters were took effect, I even stopped and restarted sge_execd on every compute node with no luck.

Any ideas? Relevant configuration information is below. Please note that in qconf I've tried

1. execd_params H_MEMORYLOCKED=ulimited 2. execd_params S_MEMORYLOCKED=ulimited 3. execd_params S_MEMORYLOCKED=ulimited H_MEMORYLOCKED=ulimited

All 3 had the same result.

Let me know if additional/complete config information would be helpful. I don't want to flood the list with unnecessary config information.

# qconf -sp orte pe_name orte slots 512 user_lists NONE xuser_lists NONE start_proc_args /bin/true stop_proc_args /bin/true allocation_rule $fill_up control_slaves TRUE job_is_first_task FALSE urgency_slots min accounting_summary FALSE

# qconf -sq all.q | grep pe_list pe_list make orte

# qconf -sconf | grep execd_params execd_params H_MEMORYLOCKED=unlimited S_MEMORYLOCKED=unlimited

$ more xhpl.sh #$ -S /bin/bash #$ -N xhpl #$ -pe orte 512 #$ -cwd #$ -V

MPI=/usr/local/openm​pi/gcc-4.1.2/x86_64/​ PATH=${MPI}/bin:$i{PATH} LD_LIBRARY_PATH=${MPI}/lib

mpirun ./xhpl

-- Prentice

Recommended Links

Setting memlock limit with SGE 6.2; was: Re: [GE users] worth a wiki entry for SGE with OpenMPI and Infiniband

Re [GE users] OpenMPI RLIMIT_MEMLOCK problem - Prentice Bisbal - net.sunsource.gridengine.users - MarkMail

MPI startup(): RLIMIT_MEMLOCK too small problem with SGE and Infiniband

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019