Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Bigger doesn't imply better. Bigger often is a sign of obesity, of lost control, of overcomplexity, of cancerous cells

Maui Scheduler

News High Performance Computing (HPC) Recommended Links Typical set of features Open Source Implementations Price-based classification Critique of Gartner Job Schedulers review
OpenPBS Maui Scheduler OAR Perl Admin Tools and Scripts   Humor Etc

Maui Cluster Scheduler, the precursor to Moab HPC Suite, is an open source job scheduler for clusters and supercomputers. It is an optimized, configurable tool capable of supporting an array of scheduling policies, dynamic priorities, extensive reservations, and fairshare capabilities. It is currently in use at hundreds of government, academic, and commercial sites throughout the world. All of the capabilities found in Maui are also found in Moab, while Moab has added features including basic trigger support, graphical administration tools, and a Web-based user portal among many others. To learn more about the features available on Maui and, see the Maui and Moab comparison brief for all the additional capabilities and value Moab offers.

Maui Scheduler is an batch system for cluster systems. It allows site administrators extensive control over which jobs are considered eligible for for scheduling, how the jobs are prioritized, and where these jobs are run. Maui supports advance reservations, QoS levels, backfill, and allocation management.

Its scheduling scheme is based on advanced wall-time reservations with backfill. The main difference from other common batch queue schedulers (e.g. NQS, DQS) is that Maui allows jobs to overtake a job with higher priority only if it does not delay the start of the prioritized job (i.e. backfill).

Opportunistic Scheduling versus Advance Reservation

In theory, unless there is a way of reserving resources in advance, any job that need more than a single piece of allocatable resource, runs the risk of starvation[1,2]. In practice, due to limited workload this is a danger only for jobs that require extensive resources. It is often intervened by periodically putting small, less demanding jobs on hold until the large, starving job have started. This is a rather intrusive operation since it also affects jobs that is not involved in the starvation.

An advance reservation scheme, as in the Maui scheduler, makes it possible to allocate resources in the future. (Compare scheduling a meeting with several participants. It is almost impossible unless there is a calendar available.)

The starvation can also be resolved by using a preemptive job scheduler. Unfortunately not all computer systems can handle this. Moreover, if jobs can be preempted it is more difficult to predict when the job will finish. 

Queues versus Quality of Service

When an idle job becomes eligible to run, it is assigned a priority. This priority is used to sort the jobs before the scheduler selects a job to start.

Many batch systems use queues to divide and classify the workload. Each queue is then assigned a priority and sometimes each job is assigned a second priority to sort themselves within the queue. This classifying scheme is often too coarse. To take into account all parameters that set a batch job policy, you may end up with more queues than jobs.

In Maui, "queues" have lost their importance in classification and priority calculations. Instead a Quality-of-Service (QoS) attribute can be used to classify the jobs. However, QoS is not a hierarchical scheme. It is merely a method of setting the parameters of a job when it enters the scheduler. All jobs eligible to run remain in one common idle-queue and their priorities are compared with all others.

Job State

Jobs in Maui can be in one of three major states:

Running
A jobs that have been alloted its required resources and have started its computation is considered running until it finish.
Queued (idle)
Jobs that are eligible to run. The priority is calculated here and the jobs are sorted according to calculated priority. Advance reservations are made starting with the job up front.
Non-queued
Jobs that, for some reason, are not allowed to start. Jobs in this state does not gain any queue-time priority.

There is a limit on the number of jobs a group/user can have in the Queued state. This prohibit users from acquiring longer queue-time than deserved by submitting large number of jobs.

Job Priority

Maui present numerous factors in the expression used to calculate the job priority to achieve a site's goals of fairness and utilization. Each factor is weighted according to its importance and the sum is used as the total priority of the job. The most important factors are described below together with the importance they have in the current configuration of Maui on Ingvar

Resource

The resource factor consist of several terms that describes the required resource to run the job; number of processors, amount of memory, size of empty disk space, and swap size. Depending on what type of jobs is favored, jobs can be pushed the front of the queue. Experience shows that favoring large jobs often improves system utilization.

Ingvar: Fairly low rating. A high utilization is desired but the fairness between users should not be affected.

Queue Time

This factor is based on the time the job has been eligible to run. This factor often has a very low weight in the priority calculation. Instead, more important is the expansion factor

Ingvar: Low rating. A fall-back.

Expansion

The expansion factor or XFactor is calculated using the equation:

  XFactor = (Queue_Time + Job_Time_Limit) / Job_Time_Limit
This relates the job time limit the user request to the total queueing and expected run time. A job with low time limit will increase its priority more quickly than a long job, pushing it to the front of the queue.

Ingvar: The most important factor after QoS. It verbalizes the general job scheduling policy.

Target

If the expansion factor is not enough to meet the scheduling goals, there is a Target factor that is increased exponentially as the actual queue time approach the target queue time.

Ingvar: Not used ...yet.

Fair Share

The fair share value is based on historical usage. It is divided into the user, group, and account associated with the job. Fair Share is a provocative factor. Although the intention is good, the effect of this factor is not easy to understand and rate to achieve fairness[3].

Ingvar: Excluded from any priority calculation.

Quality of Service

The QoS factor is a fixed number used to offset jobs with high quality-of-service.

Ingvar: Three different QoS exist; Normal, Bonus, and Disabled. Normal has a ten times higher QoS-factor than Bonus, always pushing bonus jobs to the back of the queue. Another feature in Maui prohibit Disabled jobs to make any reservations.

Reservation

There are two types of reservations in Maui:

Job Reservation

Every scheduling cycle, after the job priority have been calculated, Maui examines the jobs in the queued state and schedules advance reservations.

User Reservation
If a number of nodes is needed for a certain purpose on a certain time, it is possible, for the Maui administrator, to make an user reservation. This reservation is permanent and kept between scheduling cycles.

Also, there is standing reservations. This is user reservations which are scheduled automatically and repeatedly.

Links

http://supercluster.org
http://www.openpbs.org
User Commands
The following commands are available for users:
showq
Shows information about 1) running, 2) idle, and 3) non-queued jobs. Non-queued jobs are jobs that have been submitted to the batch queue system but are not considered eligible to run. These jobs does not gain any queued time priority.
showbf
Shows what resources are available for immediate use.
showstart <jobid>
Shows the earliest time the queued job <jobid> is eligible to start.
checkjob <jobid>
Shows various details about a submitted job. Users are only permitted to see details of jobs they own.


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

Maui Cluster Scheduler

Maui is an advanced job scheduler for use on clusters and supercomputers. It is a highly optimized and configurable tool capable of supporting a large array of scheduling policies, dynamic priorities, extensive reservations, and fairshare. It is currently in use at hundreds of leading government, academic, and commercial sites throughout the world. It improves the manageability and efficiency of machines ranging from clusters of a few processors to multi-teraflop supercomputers.

Maui is a community project* and may be downloaded, modified, and distributed. It has been made possible by the support of Cluster Resources, Inc and the contributions of many individuals and sites including the U.S. Department of Energy, PNNL, the Center for High Performance Computing at the University of Utah (CHPC), Ohio Supercomputing Center (OSC), University of Southern California (USC), SDSC, MHPCC, BYU, NCSA, and many others.

Features:

Maui extends the capabilities of base resource management systems by adding the following features: Maui interfaces with numerous resource management systems supporting any of the following scheduling API's

PBS Scheduling API - TORQUE, OpenPBS and PBSPro
Loadleveler Scheduling API - Loadleveler (IBM)
SGE Scheduling API - Sun Grid Engine (Sun)*
BProc Scheduling API - BProc (Scyld)**
SSS XML Scheduling API*
LSF Scheduling API - LSF (Platform)
Wiki FlatText Scheduling API (Wiki)
*partial support or under development
**supported under Clubmask

Maui is currently supported on all known variants of Linux, AIX, OSF/Tru-64, Solaris, HP-UX, IRIX, FreeBSD, and other UNIX platforms.

The Maui scheduler is mature, fully documented, and supported. It continues to be agressively developed and possesses a very active and growing user community. Its legacy of pushing the scheduling envelope continues as we promise to deliver the best possible scheduler supporting systems software will allow.

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

The Maui Job Scheduler

Maui Cluster Scheduler - Sourceforge page

Maui



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2018 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: September, 12, 2017